spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Sulimanov <asulima...@tremorvideodsp.com>
Subject Question regarding cached partitions
Date Tue, 31 Oct 2017 03:10:11 GMT
Hi,

I started Spark Streaming job with 96 executors which reads from 96 Kafka partitions and applies
mapWithState on the incoming DStream.

Why would it cache only 77 partitions? Do I have to allocate more memory? Currently each executor
gets 10 GB and it is not clear why it can't cache all 96 partitions.
[cid:5DD89010-AB5E-43B3-88F6-486FDB58CD75]

Thank you!
Mime
View raw message