spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: Kafka Consumer in Spark Streaming
Date Wed, 05 Nov 2014 07:12:58 GMT
Your code doesn't trigger any action. How about the following?

JavaStreamingContext ssc = new JavaStreamingContext(sparkConf, new
Duration(60 * 1 * 1000));

        JavaPairReceiverInputDStream tweets = KafkaUtils.createStream(ssc,
"<machine>:2181", "1", map);

        JavaDStream<String> statuses = tweets.map(
                new Function<String, String>() {
                    public String call(String status) {
                        System.out.println(status);
                        return status;
                    }
                }
        );


statuses​.print()
​;​
 ​

​​

Or you could use foreachRDD instead of map() if your intention is just
printing.​​

Thanks
Best Regards

On Wed, Nov 5, 2014 at 12:35 PM, Something Something <
mailinglists19@gmail.com> wrote:

> It's not local.  My spark url is something like this:
>
>         String sparkUrl = "spark://<host name>:7077";
>
>
> On Tue, Nov 4, 2014 at 11:03 PM, Jain Rahul <jainr@ivycomptech.com> wrote:
>
>>
>>  I think you are running it locally.
>> Do you have local[1] here for master url? If yes change it to local[2] or
>> more number of threads.
>> It may be due to topic name mismatch also.
>>
>>      sparkConf.setMaster(“local[1]");
>>
>>  Regards,
>> Rahul
>>
>>   From: Something Something <mailinglists19@gmail.com>
>> Date: Wednesday, November 5, 2014 at 12:23 PM
>> To: "Shao, Saisai" <saisai.shao@intel.com>
>> Cc: "user@spark.apache.org" <user@spark.apache.org>
>>
>> Subject: Re: Kafka Consumer in Spark Streaming
>>
>>   Added foreach as follows.  Still don't see any output on my console.
>> Would this go to the worker logs as Jerry indicated?
>>
>>         JavaPairReceiverInputDStream tweets =
>> KafkaUtils.createStream(ssc, "<mymachine>:2181", "1", map);
>>         JavaDStream<String> statuses = tweets.map(
>>                 new Function<String, String>() {
>>                     public String call(String status) {
>>                         return status;
>>                     }
>>                 }
>>         );
>>
>>         statuses.foreach(new Function<JavaRDD<String>, Void>() {
>>             @Override
>>             public Void call(JavaRDD<String> stringJavaRDD) throws
>> Exception {
>>                 for (String str: stringJavaRDD.take(10)) {
>>                     System.out.println("Message: " + str);
>>                 }
>>                 return null;
>>             }
>>         });
>>
>>
>> On Tue, Nov 4, 2014 at 10:32 PM, Shao, Saisai <saisai.shao@intel.com>
>> wrote:
>>
>>>  If you’re running on a standalone mode, the log is under
>>> <SPAR_HOME>/work/ directory. I’m not sure for yarn or mesos, you can
check
>>> the document of Spark to see the details.
>>>
>>>
>>>
>>> Thanks
>>>
>>> Jerry
>>>
>>>
>>>
>>> *From:* Something Something [mailto:mailinglists19@gmail.com]
>>> *Sent:* Wednesday, November 05, 2014 2:28 PM
>>> *To:* Shao, Saisai
>>> *Cc:* user@spark.apache.org
>>> *Subject:* Re: Kafka Consumer in Spark Streaming
>>>
>>>
>>>
>>> The Kafka broker definitely has messages coming in.  But your #2 point
>>> is valid.  Needless to say I am a newbie to Spark.  I can't figure out
>>> where the 'executor' logs would be.  How would I find them?
>>>
>>> All I see printed on my screen is this:
>>>
>>> 14/11/04 22:21:23 INFO Slf4jLogger: Slf4jLogger started
>>> 14/11/04 22:21:23 INFO Remoting: Starting remoting
>>> 14/11/04 22:21:24 INFO Remoting: Remoting started; listening on
>>> addresses :[akka.tcp://spark@mymachie:60743]
>>> 14/11/04 22:21:24 INFO Remoting: Remoting now listens on addresses:
>>> [akka.tcp://spark@mymachine:60743]
>>> 14/11/04 22:21:24 WARN NativeCodeLoader: Unable to load native-hadoop
>>> library for your platform... using builtin-java classes where applicable
>>> 14/11/04 22:21:24 INFO JniBasedUnixGroupsMappingWithFallback: Falling
>>> back to shell based
>>> -------------------------------------------
>>> Time: 1415168520000 ms
>>> -------------------------------------------
>>> -------------------------------------------
>>> Time: 1415168520000 ms
>>> -------------------------------------------
>>>
>>> Keeps repeating this...
>>>
>>>
>>>
>>> On Tue, Nov 4, 2014 at 10:14 PM, Shao, Saisai <saisai.shao@intel.com>
>>> wrote:
>>>
>>>  Hi, would you mind describing your problem a little more specific.
>>>
>>>
>>>
>>> 1.      Is the Kafka broker currently has no data feed in?
>>>
>>> 2.      This code will print the lines, but not in the driver side, the
>>> code is running in the executor side, so you can check the log in worker
>>> dir to see if there’s any printing logs under this folder.
>>>
>>> 3.      Did you see any exceptions when running the app, this will help
>>> to define the problem.
>>>
>>>
>>>
>>> Thanks
>>>
>>> Jerry
>>>
>>>
>>>
>>> *From:* Something Something [mailto:mailinglists19@gmail.com]
>>> *Sent:* Wednesday, November 05, 2014 1:57 PM
>>> *To:* user@spark.apache.org
>>> *Subject:* Kafka Consumer in Spark Streaming
>>>
>>>
>>>
>>> I've following code in my program.  I don't get any error, but it's not
>>> consuming the messages either.  Shouldn't the following code print the line
>>> in the 'call' method?  What am I missing?
>>>
>>> Please help.  Thanks.
>>>
>>>
>>>
>>>         JavaStreamingContext ssc = new JavaStreamingContext(sparkConf,
>>> new Duration(60 * 1 * 1000));
>>>
>>>         JavaPairReceiverInputDStream tweets =
>>> KafkaUtils.createStream(ssc, "<machine>:2181", "1", map);
>>>
>>>         JavaDStream<String> statuses = tweets.map(
>>>                 new Function<String, String>() {
>>>                     public String call(String status) {
>>>                         System.out.println(status);
>>>                         return status;
>>>                     }
>>>                 }
>>>         );
>>>
>>>
>>>
>>
>>   This email and any attachments are confidential, and may be legally
>> privileged and protected by copyright. If you are not the intended
>> recipient dissemination or copying of this email is prohibited. If you have
>> received this in error, please notify the sender by replying by email and
>> then delete the email completely from your system. Any views or opinions
>> are solely those of the sender. This communication is not intended to form
>> a binding contract unless expressly indicated to the contrary and properly
>> authorised. Any actions taken on the basis of this email are at the
>> recipient's own risk.
>>
>
>

Mime
View raw message