spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ayan guha <guha.a...@gmail.com>
Subject Re: Livy with Spark package
Date Thu, 24 Aug 2017 03:36:21 GMT
Thanks and agreed :)

On Thu, Aug 24, 2017 at 12:01 PM, Saisai Shao <sai.sai.shao@gmail.com>
wrote:

> You could set "spark.jars.packages" in `conf` field of session post API (
> https://github.com/apache/incubator-livy/blob/master/
> docs/rest-api.md#post-sessions). This is equal to --package in
> spark-submit.
>
> BTW you'd better ask livy question in user@livy.incubator.apache.org.
>
> Thanks
> Jerry
>
> On Thu, Aug 24, 2017 at 8:11 AM, ayan guha <guha.ayan@gmail.com> wrote:
>
>> Hi
>>
>> I have a python program which I invoke as
>>
>>  spark-submit --packages com.databricks:spark-avro_2.11:3.2.0
>> somefile.py  "2017-08-23 02:00:00"  and it works....
>>
>> Now I want to submit this file using Livy. I could work out most of the
>> stuff (like putting files to HDFS etc) but not able to understand how/where
>> to configure the "packages" switch.......Any help?
>> --
>> Best Regards,
>> Ayan Guha
>>
>
>


-- 
Best Regards,
Ayan Guha

Mime
View raw message