spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacob Maloney <jmalo...@conversantmedia.com>
Subject Issue with java spark broadcast
Date Fri, 10 Oct 2014 21:15:47 GMT
I'm trying to broadcast an accumulator I generated earlier in my app. However I get a nullpointer
exception whenever I reference the value.

		// The start of my accumulator generation
		LookupKeyToIntMap keyToIntMapper = new LookupKeyToIntMap();
		keyToIntMapper.setNumPartitions(intermediatePair.splits().size());
		keyToIntMapper.setMapAccumulator(keyToIntMap);
		JavaRDD<Tuple2<Integer,Iterable<Long>>> intermediateIntsTuple = intermediatePair.mapPartitionsWithIndex(keyToIntMapper,false);
		
		JavaPairRDD<Integer,Iterable<Long>> intermediatePairInts = JavaPairRDD.fromJavaRDD(intermediateIntsTuple);

		JavaPairRDD<Integer,Tuple2<Integer,Integer>> sims = intermediatePairInts.mapValues(new
SelfSim());
		
		// I force the RDD to evaluate so to avoid laziness issues
		Map<Integer,Tuple2<Integer,Integer>> simsMap = sims.collectAsMap();

		// Broadcast the map
		// If I include a print statement here on the accumulator I can print the map out succesfully
		broadcastVar = ctx.broadcast(keyToIntMap.value());
		
		//  Here I try to access the broadcasted map
		JavaPairRDD<Integer,Long> indidIntKeyPair = indidKeyPairFiltered.mapToPair(new PairFunction<Tuple2<String,Long>,
Integer, Long>(){		
			@Override
			public Tuple2<Integer,Long> call(Tuple2<String,Long> keyVal) throws Exception{
				Integer outInt = broadcastVar.value().inverse().get(keyVal._1);
				return new Tuple2<Integer,Long>(outInt,keyVal._2);
			}
		});

This works when I run it locally just fine but when I move it to a cluster environment it
throws nullpointerexceptions. My questions is why can't I access this map? And what do I have
to do to make it accessible.

Thanks,

Jacob

-----Original Message-----
From: user-help@spark.apache.org [mailto:user-help@spark.apache.org] 
Sent: Friday, October 10, 2014 4:02 PM
To: Jacob Maloney
Subject: FAQ for user@spark.apache.org

Hi! This is the ezmlm program. I'm managing the user@spark.apache.org mailing list.

FAQ - Frequently asked questions of the user@spark.apache.org list.

None available yet.


--- Administrative commands for the user list ---

I can handle administrative requests automatically. Please do not send them to the list address!
Instead, send your message to the correct command address:

To subscribe to the list, send a message to:
   <user-subscribe@spark.apache.org>

To remove your address from the list, send a message to:
   <user-unsubscribe@spark.apache.org>

Send mail to the following for info and FAQ for this list:
   <user-info@spark.apache.org>
   <user-faq@spark.apache.org>

Similar addresses exist for the digest list:
   <user-digest-subscribe@spark.apache.org>
   <user-digest-unsubscribe@spark.apache.org>

To get messages 123 through 145 (a maximum of 100 per request), mail:
   <user-get.123_145@spark.apache.org>

To get an index with subject and author for messages 123-456 , mail:
   <user-index.123_456@spark.apache.org>

They are always returned as sets of 100, max 2000 per request, so you'll actually get 100-499.

To receive all messages with the same subject as message 12345, send a short message to:
   <user-thread.12345@spark.apache.org>

The messages should contain one line or word of text to avoid being treated as sp@m, but I
will ignore their content.
Only the ADDRESS you send to is important.

You can start a subscription for an alternate address, for example "john@host.domain", just
add a hyphen and your address (with '=' instead of '@') after the command word:
<user-subscribe-john=host.domain@spark.apache.org>

To stop subscription for this address, mail:
<user-unsubscribe-john=host.domain@spark.apache.org>

In both cases, I'll send a confirmation message to that address. When you receive it, simply
reply to it to complete your subscription.

If despite following these instructions, you do not get the desired results, please contact
my owner at user-owner@spark.apache.org. Please be patient, my owner is a lot slower than
I am ;-)

--- Enclosed is a copy of the request I received.

Return-Path: <jmaloney@conversantmedia.com>
Received: (qmail 26980 invoked by uid 99); 10 Oct 2014 21:02:15 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Oct 2014 21:02:15 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,SPF_PASS,T_FILL_THIS_FORM_SHORT
X-Spam-Check-By: apache.org
Received-SPF: pass (nike.apache.org: domain of jmaloney@conversantmedia.com designates 69.8.121.83
as permitted sender)
Received: from [69.8.121.83] (HELO ord-smtp.vclk.net) (69.8.121.83)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 10 Oct 2014 21:01:49 +0000
Received: from mip2.netscaler3-4.sj2.vclk.net (HELO ORD-EXCA101.corp.valueclick.com) ([10.28.194.7])
  by ord-smtp101.corp.valueclick.com with ESMTP/TLS/AES128-SHA; 10 Oct 2014 13:01:47 -0800
Received: from ORD-EXDB102.corp.valueclick.com ([fe80::d5d8:1744:abdd:f44e])  by ORD-EXCA101.corp.valueclick.com
([10.110.1.62]) with mapi id  14.03.0195.001; Fri, 10 Oct 2014 16:01:46 -0500
From: Jacob Maloney <jmaloney@conversantmedia.com>
To: "user-faq@spark.apache.org" <user-faq@spark.apache.org>
Subject: faq
Thread-Topic: faq
Thread-Index: Ac/kzWUqxZW+z7tLRiqBUkKxHuckrg==
Date: Fri, 10 Oct 2014 21:01:45 +0000
Message-ID: <DD3D4E408B31A2409854BEF64318FD4133404C7D@ORD-EXDB102.corp.valueclick.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator: 
x-originating-ip: [192.168.176.48]
Content-Type: multipart/related;
	boundary="_004_DD3D4E408B31A2409854BEF64318FD4133404C7DORDEXDB102corpv_";
	type="multipart/alternative"
MIME-Version: 1.0
X-Virus-Checked: Checked by ClamAV on apache.org





This email and any files included with it may contain privileged,
proprietary and/or confidential information that is for the sole use
of the intended recipient(s).  Any disclosure, copying, distribution,
posting, or use of the information contained in or attached to this
email is prohibited unless permitted by the sender.  If you have
received this email in error, please immediately notify the sender
via return email, telephone, or fax and destroy this original transmission
and its included files without reading or saving it in any manner.
Thank you.


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message