spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kartik Ohri <kartikohr...@gmail.com>
Subject Re: Structuring a PySpark Application
Date Thu, 01 Jul 2021 06:44:18 GMT
Hi Gourav,

Thanks for the suggestion, I'll check it out.

Regards,
Kartik

On Thu, Jul 1, 2021 at 5:38 AM Gourav Sengupta <gourav.sengupta@gmail.com>
wrote:

> Hi,
>
> I think that reading Matei Zaharia's book "SPARK the definitive guide"
> will be a good and best starting point.
>
> Regards,
> Gourav Sengupta
>
> On Wed, Jun 30, 2021 at 3:47 PM Kartik Ohri <kartikohri13@gmail.com>
> wrote:
>
>> Hi all!
>>
>> I am working on a Pyspark application and would like suggestions on how
>> it should be structured.
>>
>> We have a number of possible jobs, organized in modules. There is also a "
>> RequestConsumer
>> <https://github.com/metabrainz/listenbrainz-server/blob/master/listenbrainz_spark/request_consumer/request_consumer.py>"
>> class which consumes from a messaging queue. Each message contains the name
>> of the job to invoke and the arguments to be passed to it. Messages are put
>> into the message queue by cronjobs, manually etc.
>>
>> We submit a zip file containing all python files to a Spark cluster
>> running on YARN and ask it to run the RequestConsumer. This
>> <https://github.com/metabrainz/listenbrainz-server/blob/master/docker/start-spark-request-consumer.sh#L23-L34>
>> is the exact spark-submit command for the interested. The results of the
>> jobs are collected
>> <https://github.com/metabrainz/listenbrainz-server/blob/master/listenbrainz_spark/request_consumer/request_consumer.py#L120-L122>
>> by the request consumer and pushed into another queue.
>>
>> My question is whether this type of structure makes sense. Should the
>> Request Consumer instead run independently of Spark and invoke spark-submit
>> scripts when it needs to trigger a job? Or is there another recommendation?
>>
>> Thank you all in advance for taking the time to read this email and
>> helping.
>>
>> Regards,
>> Kartik.
>>
>>
>>

Mime
View raw message