spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nicolas Paris <nipari...@gmail.com>
Subject Re: Dataframe from 1.5G json (non JSONL)
Date Tue, 05 Jun 2018 20:37:26 GMT
have you played with driver/executor memory configuration ?

Increasing them should avoid OOM

2018-06-05 22:30 GMT+02:00 raksja <shanmugkraja@gmail.com>:

> Agreed, gzip or non splittable, the question that i have and examples i
> have
> posted above all are referring to non compressed file. A single json file
> with Array of objects in a continuous line.
>
>
>
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Mime
View raw message