pig-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohini Palaniswamy <roh...@apache.org>
Subject Re: Submitting multiple Pig Scripts on the same Session
Date Tue, 22 Jan 2019 18:49:22 GMT
If you are using PigServer and submitting programmatically via same jvm, it
should automatically reuse the application if the requested AM resources
are same.


On Fri, Jan 18, 2019 at 12:20 PM Diego Pereira <diego.ns.pereira@gmail.com>

> Hi!
> We are developing an application that is looking for new files on a folder,
> running a few Pig Scripts to prepare those files and, finally, loading them
> into our database.
> The problem is that, for small files, the time that Pig / Tez / Yarn take
> to create a new application master and spawn new containers is way longer
> than the time it takes processing.
> Since Tez Sessions already allows a single Pig script to run multiple DAGs
> against the same application master, is there a way to reuse that
> application master and it´s containers for multiple Pig Scripts submissions
> ?
> Regards,
> Diego

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message