tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Solal Pirelli <t-sop...@microsoft.com.INVALID>
Subject RE: Proposal: Simulator mode
Date Thu, 28 Sep 2017 21:05:24 GMT

Gentle ping. :)

It seems my current implementation is buggy when sending events from the fake processor (e.g.
to test how Tez handles load), is there documentation on Tez hearbeat requests/responses anywhere?
I don't know what the `preRoutedStartIndex` and `startIndex` values in `TezHeartbeatRequest`
are for, nor whether I should be doing anything more than one heartbeat with a "progress is
100%" and a "task is finished" event.


Solal Pirelli

-----Original Message-----
From: Solal Pirelli [mailto:t-sopire@microsoft.com.INVALID] 
Sent: Monday, September 25, 2017 4:41 PM
To: dev@tez.apache.org
Subject: Proposal: Simulator mode


I opened a JIRA issue and was redirected to the mailing list, so here I am. :)

is early work on a new feature proposal: a Tez "simulator" in which vertices are not actually
executed, but instead use a simplified "fake" processor (which by default does nothing) to
let a developer see how Tez will handle certain workloads.

The goal is to be relatively close to an actual Tez run (including support for e.g. blacklisting
nodes, to see what happens when simulating an operation with a high failure rate) , without
requiring an actual Hadoop cluster; the whole thing runs inside a single JVM.

The JIRA issue describes the current implementation, and some possible questions a simulator
could help answer.

What do you think about this proposal?
I'd appreciate any pointers regarding the implementation.


Solal Pirelli

View raw message