spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ÐΞ€ρ@Ҝ (๏̯͡๏) <deepuj...@gmail.com>
Subject Re: Fine control with sc.sequenceFile
Date Mon, 29 Jun 2015 04:34:01 GMT
    val hadoopConf = new Configuration(sc.hadoopConfiguration)

    hadoopConf.set("mapreduce.input.fileinputformat.split.maxsize",
"67108864")


    sc.hadoopConfiguration(hadoopConf)

or

    sc.hadoopConfiguration = hadoopConf

threw error.

On Sun, Jun 28, 2015 at 9:32 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> sequenceFile() calls hadoopFile() where:
>     val confBroadcast = broadcast(new
> SerializableConfiguration(hadoopConfiguration))
>
> You can set the parameter in sc.hadoopConfiguration before calling
> sc.sequenceFile().
>
> Cheers
>
> On Sun, Jun 28, 2015 at 9:23 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <deepujain@gmail.com>
> wrote:
>
>> I can do this
>>
>>     val hadoopConf = new Configuration(sc.hadoopConfiguration)
>>
>> *    hadoopConf.set("mapreduce.input.fileinputformat.split.maxsize",
>> "67108864")*
>>
>>     sc.newAPIHadoopFile(
>>
>>       path + "/*.avro",
>>
>>       classOf[AvroKeyInputFormat[GenericRecord]],
>>
>>       classOf[AvroKey[GenericRecord]],
>>
>>       classOf[NullWritable],
>>
>>       hadoopConf)
>>
>>
>> But i cant do the same with
>>
>> sc.sequenceFile("path", classOf[Text], classOf[Text])
>> How can i achieve the same with sequenceFile
>> --
>> Deepak
>>
>>
>


-- 
Deepak

Mime
View raw message