spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tomasz Gawęda <>
Subject SQL Visualization for cached Dataset
Date Tue, 02 Jan 2018 18:44:16 GMT

Recently I had to optimize few Apache Spark SQL queries. Some of the 
Datasets were reused, so they were cached. However after caching I don't 
see SQL Visualization for the cached Dataset in Spark UI - I see only 
InMemoryRelation node. Explain result at the bottom of the page still 
has full plan.

Is this an expected behaviour? In such cases we have much less options 
to debug performance in Spark. My suggestion is to show full diagram on 
the first action after cache or to show separate SQL query for cache - 
second option however probably is not possible as cache does not trigger 
calculation, so we can't get metrics.

Workaround is to temporairly disable caching, but it consumes much time 
to do it, especially on large datasets

Pozdrawiam / Best regards,


View raw message