spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lemieud <david.lemi...@radialpoint.com>
Subject Spark Streaming using Flume body size limitation
Date Tue, 20 May 2014 19:39:19 GMT
Hi,

I am trying to send events to spark streaming via flume.
It's working fine up to a certain point.
I have problems when the size of the body is over 1020 characters.

Basically, up to 1020 it works
1021 through 1024, the event will be accepted and there is no exception, but the channel seems
to be corrupted. Or at least no more events can make it through.
1025 and up, I am seeing some exceptions : java.io.StreamCorruptedException: invalid stream
header: 00000000

I created a test using the Embedded Agent and a local spark to expose the problem.
I am using the cdh5 distribution.
   <cdh-flume.version>1.4.0-cdh5.0.0</cdh-flume.version>
   <cdh-spark.version>0.9.0-cdh5.0.0</cdh-spark.version>
   <cdh-avro.version>1.7.5-cdh5.0.0</cdh-avro.version>

File is attached.

Has anybody ever seen this?
Any suggestions?

/D

*********************************************************************** This e-mail and attachments
are confidential, legally privileged, may be subject to copyright and sent solely for the
attention of the addressee(s). Any unauthorized use or disclosure is prohibited. Statements
and opinions expressed in this e-mail may not represent those of Radialpoint. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Le contenu du présent courriel est confidentiel, privilégié et peut être soumis à des
droits d'auteur. Il est envoyé à l'intention exclusive de son ou de ses destinataires. Il
est interdit de l'utiliser ou de le divulguer sans autorisation. Les opinions exprimées dans
le présent courriel peuvent diverger de celles de Radialpoint.


SparkFlumeStreamTest.java (11K) <http://apache-spark-user-list.1001560.n3.nabble.com/attachment/6127/0/SparkFlumeStreamTest.java>




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-using-Flume-body-size-limitation-tp6127.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Mime
View raw message