spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Biplob Biswas <revolutioni...@gmail.com>
Subject Re: Running JavaBased Implementationof StreamingKmeans
Date Tue, 21 Jun 2016 08:47:24 GMT
Hi,

Can someone please look into this and tell me whats wrong?and why am I not
getting any output?

Thanks & Regards
Biplob Biswas

On Sun, Jun 19, 2016 at 1:29 PM, Biplob Biswas <revolutionisme@gmail.com>
wrote:

> Hi,
>
> Thanks for that input, I tried doing that but apparently thats not working
> as well. I thought i am having problems with my spark installation so I ran
> simple word count and that works, so I am not really sure what the problem
> is now.
>
> Is my translation of the scala code correct? I don't understand the syntax
> of scala very well thus wrote my own implementation of streaming kmeans in
> java, so i am hoping thats correct.
>
> Thanks & Regards
> Biplob Biswas
>
> On Sun, Jun 19, 2016 at 3:23 AM, Akhil Das <akhld@hacked.work> wrote:
>
>> SparkStreaming does not pick up old files by default, so you need to
>> start your job with master=local[2] (It needs 2 or more working threads, 1
>> to read the files and the other to do your computation) and once the job
>> start to run, place your input files in the input directories and you can
>> see them being picked up by sparkstreaming.
>>
>> On Sun, Jun 19, 2016 at 12:37 AM, Biplob Biswas <revolutionisme@gmail.com
>> > wrote:
>>
>>> Hi,
>>>
>>> I tried local[*] and local[2] and the result is the same. I don't really
>>> understand the problem here.
>>> How can I confirm that the files are read properly?
>>>
>>> Thanks & Regards
>>> Biplob Biswas
>>>
>>> On Sat, Jun 18, 2016 at 5:59 PM, Akhil Das <akhld@hacked.work> wrote:
>>>
>>>> Looks like you need to set your master to local[2] or local[*]
>>>>
>>>> On Sat, Jun 18, 2016 at 4:54 PM, Biplob Biswas <
>>>> revolutionisme@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I implemented the streamingKmeans example provided in the spark
>>>>> website but
>>>>> in Java.
>>>>> The full implementation is here,
>>>>>
>>>>> http://pastebin.com/CJQfWNvk
>>>>>
>>>>> But i am not getting anything in the output except occasional
>>>>> timestamps
>>>>> like one below:
>>>>>
>>>>> -------------------------------------------
>>>>> Time: 1466176935000 ms
>>>>> -------------------------------------------
>>>>>
>>>>> Also, i have 2 directories:
>>>>> "D:\spark\streaming example\Data Sets\training"
>>>>> "D:\spark\streaming example\Data Sets\test"
>>>>>
>>>>> and inside these directories i have 1 file each
>>>>> "samplegpsdata_train.txt"
>>>>> and "samplegpsdata_test.txt" with training data having 500 datapoints
>>>>> and
>>>>> test data with 60 datapoints.
>>>>>
>>>>> I am very new to the spark systems and any help is highly appreciated.
>>>>>
>>>>> Thank you so much
>>>>> Biplob Biswas
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Running-JavaBased-Implementationof-StreamingKmeans-tp27192.html
>>>>> Sent from the Apache Spark User List mailing list archive at
>>>>> Nabble.com.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Cheers!
>>>>
>>>>
>>>
>>
>>
>> --
>> Cheers!
>>
>>
>

Mime
View raw message