spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <t...@databricks.com>
Subject Re: Writing to a single file from multiple executors
Date Wed, 11 Mar 2015 22:39:27 GMT
Why do you have to write a single file?



On Wed, Mar 11, 2015 at 1:00 PM, SamyaMaiti <samya.maiti2012@gmail.com>
wrote:

> Hi Experts,
>
> I have a scenario, where in I want to write to a avro file from a streaming
> job that reads data from kafka.
>
> But the issue is, as there are multiple executors and when all try to write
> to a given file I get a concurrent exception.
>
> I way to mitigate the issue is to repartition & have a single writer task,
> but as my data is huge that is not a feasible option.
>
> Any suggestions welcomed.
>
> Regards,
> Sam
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Writing-to-a-single-file-from-multiple-executors-tp22003.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message