spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <jornfra...@gmail.com>
Subject Re: Ensuring an Avro File is NOT Splitable
Date Thu, 20 Oct 2016 12:17:29 GMT
What is the use case of this? You will reduce performance significantly.
Nevertheless, the way you propose is the way to go, but I do not recommend it.

> On 20 Oct 2016, at 14:00, Ashan Taha <ataha@currenex.com> wrote:
> 
> Hi
>  
> What’s the best way to make sure an Avro file is NOT Splitable when read in Spark?
> Would you override the AvroKeyInputFormat.issplitable (to return false) and then call
this using newAPIHadoopRDD? Or is there a better way using the sqlContext.read?
>  
> Thanks in advance

Mime
View raw message