pig-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Diego Pereira <diego.ns.pere...@gmail.com>
Subject Submitting multiple Pig Scripts on the same Session
Date Fri, 18 Jan 2019 20:20:41 GMT

We are developing an application that is looking for new files on a folder,
running a few Pig Scripts to prepare those files and, finally, loading them
into our database.

The problem is that, for small files, the time that Pig / Tez / Yarn take
to create a new application master and spawn new containers is way longer
than the time it takes processing.

Since Tez Sessions already allows a single Pig script to run multiple DAGs
against the same application master, is there a way to reuse that
application master and it´s containers for multiple Pig Scripts submissions



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message