spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Charles O. Bajomo" <>
Subject [Spark Streamiing] Streaming job failing consistently after 1h
Date Mon, 06 Mar 2017 02:37:29 GMT
Hello all, 

I have a strange behaviour I can't understand. I have a streaming job using a custom java
receiver that pull data from a jms queue that I process and then write to HDFS as parquet
and avro files. For some reason my job keeps failing after 1hr and 30 minutes. When It fails
I get an error saying the "container is running beyond physical memory limits. Current Usage
4.5GB of 4.5GB physical memory used. 6.4GB of 9.4GB virtual memory used. ". to be honest I
don;t understand the error, What are the memory limits shown in the error referring to? I
allocated 10 executors with 6 cores each and 4G of executor and driver memory. I set the overhead
memory to 2.8G, so the values don't add up. 

Anyone have any idea what the error is referring? I have increased the memory and i didn't
help, it appears it just bought me more time. 


View raw message