tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Eagles <jeag...@gmail.com>
Subject Re: Scaling VertexInitializedEvent
Date Mon, 04 Feb 2019 21:59:02 GMT
Not sure if you received this email or not, Eric as I don't see you
subscribed to the dev apache list. Re-Replying to the original message
plus two extra jira links
https://issues.apache.org/jira/browse/TEZ-3914
https://issues.apache.org/jira/browse/TEZ-3784

Eric, Could you post a stack trace of the error. We have had a few
bugs in the past that prevented messages over 64MB. This is only an
artificial limit based on the high-level reader and writer APIs that
are used. If a low-level api is used, messages of arbitrary length (or
at least up to 2GB IIRC) should be possible.


On Mon, Feb 4, 2019 at 3:50 PM Jonathan Eagles <jeagles@gmail.com> wrote:
>
> Eric, Could you post a stack trace of the error. We have had a few
> bugs in the past that prevented messages over 64MB. This is only an
> artificial limit based on the high-level reader and writer APIs that
> are used. If a low-level api is used, messages of arbitrary length (or
> at least up to 2GB IIRC) should be possible.
>
> On Wed, Jan 30, 2019 at 2:47 PM Eric Goodman
> <Eric.Goodman@microsoft.com.invalid> wrote:
> >
> > Hi Tez devs,
> >
> > The current design of VertexInitializedEvent<https://github.com/apache/tez/blob/3f2373e2b2ab3825ef50e9f19b8704265542a8b2/tez-dag/src/main/java/org/apache/tez/dag/history/events/VertexInitializedEvent.java>
contains all of the InputDataInformationEvents for a particular vertex. We've had trouble
scaling this for large vertices as Protobuf limits message sizes to 64 MB. I'm wondering if
anyone is working on a more scaleable solution, and if not, if you guys have any suggestions
for how to decompose this event into smaller events so that Protobuf's message size limit
is never an issue.
> >
> > Thanks,
> > Eric

Mime
View raw message