spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akshay Bhardwaj <akshay.bhardwaj1...@gmail.com>
Subject What is Spark context cleaner in structured streaming
Date Thu, 02 May 2019 11:51:04 GMT
Hi All,

I am using Spark structured streaming with spark 2.3 running on Yarn
cluster with hadoop 2.7.3 In the driver logs I see numerous lines as below.

2019-05-02 17:14:24.619 ContextCleaner Spark Context Cleaner [INFO] Cleaned
accumulator 81492577
2019-05-02 17:14:24.619 ContextCleaner Spark Context Cleaner [INFO] Cleaned
accumulator 81492514
2019-05-02 17:14:24.619 ContextCleaner Spark Context Cleaner [INFO] Cleaned
accumulator 81492531
2019-05-02 17:14:24.620 ContextCleaner Spark Context Cleaner [INFO] Cleaned
accumulator 81492524
2019-05-02 17:14:24.620 ContextCleaner Spark Context Cleaner [INFO] Cleaned
accumulator 81492617
2019-05-02 17:14:24.620 ContextCleaner Spark Context Cleaner [INFO] Cleaned
accumulator 81492475
2019-05-02 17:14:24.620 ContextCleaner Spark Context Cleaner [INFO] Cleaned
accumulator 81492467

Could anyone explain what do these logs indicate internally? I have
streaming interval of 500ms, reading data from Kafka topic with max batch
size as 1000.


Akshay Bhardwaj
+91-97111-33849

Mime
View raw message