spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Sulimanov <>
Subject Question regarding cached partitions
Date Tue, 31 Oct 2017 03:10:11 GMT

I started Spark Streaming job with 96 executors which reads from 96 Kafka partitions and applies
mapWithState on the incoming DStream.

Why would it cache only 77 partitions? Do I have to allocate more memory? Currently each executor
gets 10 GB and it is not clear why it can't cache all 96 partitions.

Thank you!
View raw message