pig-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohini Palaniswamy <roh...@apache.org>
Subject Re: Submitting multiple Pig Scripts on the same Session
Date Tue, 22 Jan 2019 18:49:22 GMT
If you are using PigServer and submitting programmatically via same jvm, it
should automatically reuse the application if the requested AM resources
are same.

https://github.com/apache/pig/blob/trunk/src/org/apache/pig/backend/hadoop/executionengine/tez/TezSessionManager.java#L242-L245

On Fri, Jan 18, 2019 at 12:20 PM Diego Pereira <diego.ns.pereira@gmail.com>
wrote:

> Hi!
>
> We are developing an application that is looking for new files on a folder,
> running a few Pig Scripts to prepare those files and, finally, loading them
> into our database.
>
> The problem is that, for small files, the time that Pig / Tez / Yarn take
> to create a new application master and spawn new containers is way longer
> than the time it takes processing.
>
> Since Tez Sessions already allows a single Pig script to run multiple DAGs
> against the same application master, is there a way to reuse that
> application master and it´s containers for multiple Pig Scripts submissions
> ?
>
> Regards,
>
> Diego
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message