spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gerard Maas <gerard.m...@gmail.com>
Subject Re: Spark-Streaming: output to cassandra
Date Thu, 04 Dec 2014 16:52:08 GMT
I guess he's already doing so, given the 'saveToCassandra' usage.
What I don't understand is the question "how do I specify a batch". That
doesn't make much sense to me. Could you explain further?

-kr, Gerard.

On Thu, Dec 4, 2014 at 5:36 PM, Akhil Das <akhil@sigmoidanalytics.com>
wrote:

> You can use the datastax's Cassandra connector.
> <https://github.com/datastax/spark-cassandra-connector/blob/master/doc/5_saving.md>
>
> Thanks
> Best Regards
>
> On Thu, Dec 4, 2014 at 8:21 PM, <m.sarosh@accenture.com> wrote:
>
>>  Hi,
>>
>>
>>  I have written the code below which is streaming data from kafka, and
>> printing to the console.
>>
>> I want to extend this, and want my data to go into Cassandra table
>> instead.
>>
>>
>>  JavaStreamingContext jssc = new JavaStreamingContext("local[4]",
>> "SparkStream", new Duration(1000));
>> JavaPairReceiverInputDStream<String, String> messages =
>> KafkaUtils.createStream(jssc, args[0], args[1], topicMap );
>>
>>  System.out.println("Connection done!");
>> JavaDStream<String> data = messages.map(new Function<Tuple2<String,
>> String>, String>()
>> {
>> public String call(Tuple2<String, String> message)
>> {
>> return message._2();
>> }
>> }
>> );
>> //data.print();   --> output to console
>>  data.foreachRDD(saveToCassandra("mykeyspace","mytable"));
>>  jssc.start();
>> jssc.awaitTermination();
>>
>>
>>
>>  How should I implement the line:
>>
>> data.foreachRDD(saveToCassandra("mykeyspace","mytable"));‚Äč
>>
>> so that data goes into Cassandra, in each batch.  And how do I specify a
>> batch, because if i do Ctrl+C on the console of streaming-job-jar, nothing
>> will be entered into cassandra for sure since it is getting killed.
>>
>>
>>  Please help.
>>
>>
>>    Thanks and Regards,
>>
>> *Md. Aiman Sarosh.*
>> Accenture Services Pvt. Ltd.
>> Mob #:  (+91) - 9836112841.
>>
>> ------------------------------
>>
>> This message is for the designated recipient only and may contain
>> privileged, proprietary, or otherwise confidential information. If you have
>> received it in error, please notify the sender immediately and delete the
>> original. Any other use of the e-mail by you is prohibited. Where allowed
>> by local law, electronic communications with Accenture and its affiliates,
>> including e-mail and instant messaging (including content), may be scanned
>> by our systems for the purposes of information security and assessment of
>> internal compliance with Accenture policy.
>>
>> ______________________________________________________________________________________
>>
>> www.accenture.com
>>
>
>

Mime
View raw message