spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kartheek.R" <kartheek.m...@gmail.com>
Subject Re: How to convert a non-rdd data to rdd.
Date Sun, 12 Oct 2014 08:15:11 GMT
Its a variable in spark-1.0.0/*/storagre/BlockManagerMaster.scala  class.
The return data of AskDriverWithReply() method for the getPeers() method.

Basically, it is a Seq[ArrayBuffer]:

ArraySeq(ArrayBuffer(BlockManagerId(1, s1, 47006, 0), BlockManagerId(0, s1,
34625, 0)), ArrayBuffer(BlockManagerId(1, s1, 47006, 0), BlockManagerId(0,
s2, 34625, 0)), ArrayBuffer(BlockManagerId(1, s1, 47006, 0),
BlockManagerId(0, s2, 34625, 0)), ArrayBuffer(BlockManagerId(1, s1, 47006,
0), BlockManagerId(0, s2, 34625, 0)), ArrayBuffer(BlockManagerId(<driver>,
karthik, 51051, 0), BlockManagerId(1, s1, 47006, 0)))


On Sun, Oct 12, 2014 at 12:59 PM, @Sanjiv Singh [via Apache Spark User
List] <ml-node+s1001560n16231h87@n3.nabble.com> wrote:

> Hi Karthik,
>
> Can you provide us more detail of dataset "data" that you wanted to
> parallelize with
>
> SparkContext.parallelize(data);
>
>
>
>
> Regards,
> Sanjiv Singh
>
>
> Regards
> Sanjiv Singh
> Mob :  +091 9990-447-339
>
> On Sun, Oct 12, 2014 at 11:45 AM, rapelly kartheek <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=16231&i=0>> wrote:
>
>> Hi,
>>
>> I am trying to write a String that is not an rdd to HDFS. This data is a
>> variable in Spark Scheduler code. None of the spark File operations are
>> working because my data is not rdd.
>>
>> So, I tried using SparkContext.parallelize(data). But it throws error:
>>
>> [error]
>> /home/karthik/spark-1.0.0/core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala:265:
>> not found: value SparkContext
>> [error]      SparkContext.parallelize(result)
>> [error]      ^
>> [error] one error found
>>
>> I realized that this data is part of the Scheduler. So, the Sparkcontext
>> would not have got created yet.
>>
>> Any help in "writing scheduler variable data to HDFS" is appreciated!!
>>
>> -Karthik
>>
>
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-convert-a-non-rdd-data-to-rdd-tp16230p16231.html
>  To unsubscribe from How to convert a non-rdd data to rdd., click here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=16230&code=a2FydGhlZWsubWJtc0BnbWFpbC5jb218MTYyMzB8LTE1NjA1NDM4NDM=>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-convert-a-non-rdd-data-to-rdd-tp16230p16232.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Mime
View raw message