spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kelly, Jonathan" <jonat...@amazon.com>
Subject Re: kinesis multiple records adding into stream
Date Fri, 16 Jan 2015 18:05:39 GMT
Are you referring to the PutRecords method, which was added in 1.9.9?  (See http://aws.amazon.com/releasenotes/1369906126177804)
 If so, can't you just depend upon this later version of the SDK in your app even though spark-streaming-kinesis-asl
is depending upon this earlier 1.9.3 version that does not yet have the PutRecords call?

Also, could you please explain your use case more fully?  spark-streaming-kinesis-asl is for
*reading* data from Kinesis in Spark, not for writing data, so I would expect that the part
of your code that would be writing to Kinesis would be a totally separate app anyway (unless
you are reading from Kinesis using spark-streaming-kinesis-asl, transforming it somehow, then
writing it back out to Kinesis).

~ Jonathan

From: Aniket Bhatnagar <aniket.bhatnagar@gmail.com<mailto:aniket.bhatnagar@gmail.com>>
Date: Friday, January 16, 2015 at 9:13 AM
To: Hafiz Mujadid <hafizmujadid00@gmail.com<mailto:hafizmujadid00@gmail.com>>,
"user@spark.apache.org<mailto:user@spark.apache.org>" <user@spark.apache.org<mailto:user@spark.apache.org>>
Subject: Re: kinesis multiple records adding into stream


Sorry. I couldn't understand the issue. Are you trying to send data to kinesis from a spark
batch/real time job?

- Aniket

On Fri, Jan 16, 2015, 9:40 PM Hafiz Mujadid <hafizmujadid00@gmail.com<mailto:hafizmujadid00@gmail.com>>
wrote:
Hi Experts!

I am using kinesis dependency as follow
groupId = org.apache.spark
 artifactId = spark-streaming-kinesis-asl_2.10
 version = 1.2.0

in this aws sdk version 1.8.3 is being used. in this sdk multiple records
can not be put in a single request. is it possible to put multiple records
in a single request ?


thanks



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/kinesis-multiple-records-adding-into-stream-tp21191.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<mailto:user-unsubscribe@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<mailto:user-help@spark.apache.org>


Mime
View raw message