spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Fine control with sc.sequenceFile
Date Mon, 29 Jun 2015 04:32:08 GMT
sequenceFile() calls hadoopFile() where:
    val confBroadcast = broadcast(new
SerializableConfiguration(hadoopConfiguration))

You can set the parameter in sc.hadoopConfiguration before calling
sc.sequenceFile().

Cheers

On Sun, Jun 28, 2015 at 9:23 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepujain@gmail.com> wrote:

> I can do this
>
>     val hadoopConf = new Configuration(sc.hadoopConfiguration)
>
> *    hadoopConf.set("mapreduce.input.fileinputformat.split.maxsize",
> "67108864")*
>
>     sc.newAPIHadoopFile(
>
>       path + "/*.avro",
>
>       classOf[AvroKeyInputFormat[GenericRecord]],
>
>       classOf[AvroKey[GenericRecord]],
>
>       classOf[NullWritable],
>
>       hadoopConf)
>
>
> But i cant do the same with
>
> sc.sequenceFile("path", classOf[Text], classOf[Text])
> How can i achieve the same with sequenceFile
> --
> Deepak
>
>

Mime
View raw message