spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: Saving to Cassandra from Spark Streaming
Date Wed, 29 Oct 2014 07:06:41 GMT
You need to set the following jar (cassandra-connector
<http://central.maven.org/maven2/com/datastax/spark/spark-cassandra-connector_2.10/1.1.0-alpha3/spark-cassandra-connector_2.10-1.1.0-alpha3.jar>)
in the classpath like:

ssc.sparkContext.addJar("/path/to/spark-cassandra-connector_2.10-1.1.0-alpha3.jar")

Thanks
Best Regards

On Tue, Oct 28, 2014 at 10:09 PM, Gerard Maas <gerard.maas@gmail.com> wrote:

> Looks like you're having some classpath issues.
> Are you providing your spark-cassandra-driver classes to your job?
>  sparkConf.setJars(Seq(jars...)) ?
>
> On Tue, Oct 28, 2014 at 5:34 PM, Harold Nguyen <harold@nexgate.com> wrote:
>
>> Hi all,
>>
>> I'm having trouble troubleshooting this particular block of code for
>> Spark Streaming and saving to Cassandra:
>>
>>     val lines = ssc.socketTextStream(args(0), args(1).toInt,
>> StorageLevel.MEMORY_AND_DISK_SER)
>>     val words = lines.flatMap(_.split(" "))
>>     val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
>>
>>     //-- Writing it to Cassandra
>>     wordCounts.saveToCassandra("test", "kv", SomeColumns("key", "value"),
>> 1)
>>
>> Could you tell me where I'm going wrong ? Can I not call
>> wordCounts.saveToCassandra ?
>>
>> Here's the error:
>>
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> com/datastax/spark/connector/mapper/ColumnMapper
>>
>> Thanks,
>>
>> Harold
>>
>
>

Mime
View raw message