spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ruijing Li <liruijin...@gmail.com>
Subject Out of memory HDFS Multiple Cluster Write
Date Fri, 20 Dec 2019 08:34:08 GMT
Hi all,

I have encountered a strange executor OOM error. I have a data pipeline
using Spark 2.3 Scala 2.11.12. This pipeline writes the output to one HDFS
location as parquet then reads the files back in and writes to multiple
hadoop clusters (all co-located in the same datacenter).  It should be a
very simple task, but executors are being killed off exceeding container
thresholds. From logs, it is exceeding given memory (using Mesos as the
cluster manager).

The ETL process works perfectly fine with the given resources, doing joins
and adding columns. The output is written successfully the first time. *Only
when the pipeline at the end reads the output from HDFS and writes it to
different HDFS cluster paths does it fail.* (It does a
spark.read.parquet(source).write.parquet(dest))

This doesn't really make sense and I'm wondering what configurations I
should start looking at.

-- 
Cheers,
Ruijing Li
-- 
Cheers,
Ruijing Li

Mime
View raw message