spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gourav Sengupta <gourav.sengu...@gmail.com>
Subject Re: Boto3 library send to pyspark
Date Wed, 17 Apr 2019 20:42:48 GMT
Hi,

not sure about different environment but it is something that you can look
into as well. Why would you need different environments for your job?

Regards,
Gourav

On Wed, Apr 17, 2019 at 11:43 AM Gorka Bravo Martinez <
gorka.bravo.martinez@cern.ch> wrote:

> Hi Gourav,
>
> you mean by seting a different python environment while running pyspark?
>
> Cheers, Gorka.
> ________________________________________
> From: Gourav Sengupta [gourav.sengupta@gmail.com]
> Sent: 17 April 2019 10:06
> To: Gorka Bravo Martinez
> Cc: user@spark.apache.org
> Subject: Re: Boto3 library send to pyspark
>
> Hi,
>
> there is addPyFile, and then there is python environment, try to search
> for using python package managers like canopy and conda.
>
> Regards,
> Gourav
>
> On Wed, Apr 17, 2019 at 8:50 AM Gorka Bravo Martinez <
> gorka.bravo.martinez@cern.ch<mailto:gorka.bravo.martinez@cern.ch>> wrote:
> Hi all,
>
> I would like to send a boto/boto3 library while running pyspark with yarn
> client mode, how is it possible?
> I am aware sc.addFile(<path>) can add a .py file, is it the same for a
> library?
>
> Cheers, Gorka.
>

Mime
View raw message