spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <>
Subject Re: Ensuring an Avro File is NOT Splitable
Date Thu, 20 Oct 2016 12:17:29 GMT
What is the use case of this? You will reduce performance significantly.
Nevertheless, the way you propose is the way to go, but I do not recommend it.

> On 20 Oct 2016, at 14:00, Ashan Taha <> wrote:
> Hi
> What’s the best way to make sure an Avro file is NOT Splitable when read in Spark?
> Would you override the AvroKeyInputFormat.issplitable (to return false) and then call
this using newAPIHadoopRDD? Or is there a better way using the
> Thanks in advance

View raw message