spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Beabes <mailinglist...@gmail.com>
Subject Debugging tools for Spark Structured Streaming
Date Fri, 30 Oct 2020 00:02:11 GMT
We're using Spark 2.4. We recently pushed to production a product that's
using Spark Structured Streaming. It's working well most of the time but
occasionally, when the load is high, we've noticed that there are only 10+
'Active Tasks' even though we've provided 128 cores. Would like to debug
this further. Why are all the Cores not getting used? How do we debug this?
Please help. Thanks.

Mime
View raw message