spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aniket More <Aniket.M...@bitwiseglobal.com>
Subject Identifying DataFrames in executing tasks
Date Wed, 07 Dec 2016 09:27:59 GMT
Hi,
I am doing a POC in which I have implemented custom Spark Listener.
I have overridden methods such as onTaskEnd(taskEnd:
SparkListenerTaskEnd),onStageCompleted(stageCompleted:
SparkListenerStageCompleted),etc.
from which I get information such as
taskId,recordsWritten,stageId,recordsRead,etc.
But I am not able to identify the dataframe executing in a task. 
For eg: I need to identify the dataframe where input file is read or task in
which dataframes are joined.

Can someone suggest me some solution for above use case where I can get
dataframes information in executing tasks?
Thanks




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Identifying-DataFrames-in-executing-tasks-tp28176.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message