spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "yujhe.li" <liyu...@gmail.com>
Subject Re: Repartition not working on a csv file
Date Sun, 01 Jul 2018 02:28:31 GMT
Abdeali Kothari wrote
> I am using Spark 2.3.0 and trying to read a CSV file which has 500
> records.
> When I try to read it, spark says that it has two stages: 10, 11 and then
> they join into stage 12.

What's your CSV size per file? I think Spark optimizer may put many files
into one task when reading small files.



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message