spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Retrieve hadoop conf object from Python API
Date Wed, 01 Jul 2015 21:22:28 GMT
To close the loop.

This should work:
sc._jsc.hadoopConfiguration

See this method in JavaSparkContext :

  def hadoopConfiguration(): Configuration = {
    sc.hadoopConfiguration

On Tue, Jun 30, 2015 at 5:52 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> Minor correction:
> It should be sc._jsc
>
> Cheers
>
> On Tue, Jun 30, 2015 at 4:23 PM, ayan guha <guha.ayan@gmail.com> wrote:
>
>> There is a sc._jsc_ which you can access to get/set hadoop conf.
>>
>> On Wed, Jul 1, 2015 at 7:41 AM, Richard Ding <piguser@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I noticed that,  in Scala API, one can call hadoopConfiguration on
>>> SparkContext to retrieve the hadoop configuration object which is very
>>> handy in modifying certain hadoop properties at runtime. But there is no
>>> corresponding method in Python API.
>>>
>>> Will this method be added to Python API in a later release? In the mean
>>> time, what will be the work around (of setting hadoop properties at
>>> runtime) using Python API?
>>>
>>> Thanks,
>>> Richard
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Ayan Guha
>>
>
>

Mime
View raw message