tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Eagles <jeag...@gmail.com>
Subject Re: Scaling VertexInitializedEvent
Date Mon, 04 Feb 2019 21:50:52 GMT
Eric, Could you post a stack trace of the error. We have had a few
bugs in the past that prevented messages over 64MB. This is only an
artificial limit based on the high-level reader and writer APIs that
are used. If a low-level api is used, messages of arbitrary length (or
at least up to 2GB IIRC) should be possible.

On Wed, Jan 30, 2019 at 2:47 PM Eric Goodman
<Eric.Goodman@microsoft.com.invalid> wrote:
> Hi Tez devs,
> The current design of VertexInitializedEvent<https://github.com/apache/tez/blob/3f2373e2b2ab3825ef50e9f19b8704265542a8b2/tez-dag/src/main/java/org/apache/tez/dag/history/events/VertexInitializedEvent.java>
contains all of the InputDataInformationEvents for a particular vertex. We've had trouble
scaling this for large vertices as Protobuf limits message sizes to 64 MB. I'm wondering if
anyone is working on a more scaleable solution, and if not, if you guys have any suggestions
for how to decompose this event into smaller events so that Protobuf's message size limit
is never an issue.
> Thanks,
> Eric

View raw message