tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Solal Pirelli <t-sop...@microsoft.com.INVALID>
Subject Proposal: Simulator mode
Date Mon, 25 Sep 2017 23:40:37 GMT

I opened a JIRA issue and was redirected to the mailing list, so here I am. :)

https://issues.apache.org/jira/browse/TEZ-3841 is early work on a new feature proposal: a
Tez "simulator" in which vertices are not actually executed, but instead use a simplified
"fake" processor (which by default does nothing) to let a developer see how Tez will handle
certain workloads.

The goal is to be relatively close to an actual Tez run (including support for e.g. blacklisting
nodes, to see what happens when simulating an operation with a high failure rate) , without
requiring an actual Hadoop cluster; the whole thing runs inside a single JVM.

The JIRA issue describes the current implementation, and some possible questions a simulator
could help answer.

What do you think about this proposal?
I'd appreciate any pointers regarding the implementation.


Solal Pirelli

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message