spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sanyal Arnab <arnabsanyal2...@gmail.com>
Subject Fwd: [ Write JSON ] - An error occurred while calling o545.save
Date Wed, 18 Mar 2020 12:16:18 GMT
Hi,

I am getting "Py4JJavaError: An error occurred while calling o545.save"
error while executing below code.

myDF = spark.read.format("csv")\
.option("header",True)\
.option("mode","FAILFAST")\
.schema(myManualSchema)\
.load("C:\\Arnab\\Spark\\data\\2015-summary.csv")

*myDF.write.format("json").mode("overwrite").save("C:\\Arnab\\Spark\\data\\tmp\\my_json_file")*

*Note:- "2015-summary.csv" is 8KB and I have 8GB RAM. No other application
is running while I run this code.*
Stack Trace -

---------------------------------------------------------------------------Py4JJavaError
                            Traceback (most recent call
last)<ipython-input-23-a7940ceaf77d> in <module>----> 1
myDF.write.format("json").mode("overwrite").save("C:\\Arnab\\Spark\\data\\tmp\\my_json_file")
C:\Arnab\Spark\spark-3.0.0-preview2-bin-hadoop3.2\python\pyspark\sql\readwriter.py
in save(self, path, format, mode, partitionBy, **options)    767
      self._jwrite.save()    768         else:--> 769
self._jwrite.save(path)    770     771     @since(1.4)
C:\Arnab\Spark\spark-3.0.0-preview2-bin-hadoop3.2\python\lib\py4j-0.10.8.1-src.zip\py4j\java_gateway.py
in __call__(self, *args)   1284         answer =
self.gateway_client.send_command(command)   1285         return_value
= get_return_value(-> 1286             answer, self.gateway_client,
self.target_id, self.name)   1287    1288         for temp_arg in
temp_args:
C:\Arnab\Spark\spark-3.0.0-preview2-bin-hadoop3.2\python\pyspark\sql\utils.py
in deco(*a, **kw)     96     def deco(*a, **kw):     97
try:---> 98             return f(*a, **kw)     99         except
py4j.protocol.Py4JJavaError as e:    100             converted =
convert_exception(e.java_exception)
C:\Arnab\Spark\spark-3.0.0-preview2-bin-hadoop3.2\python\lib\py4j-0.10.8.1-src.zip\py4j\protocol.py
in get_return_value(answer, gateway_client, target_id, name)    326
             raise Py4JJavaError(    327                     "An error
occurred while calling {0}{1}{2}.\n".--> 328
format(target_id, ".", name), value)    329             else:    330
              raise Py4JError(
Py4JJavaError: An error occurred while calling o799.save.
: org.apache.spark.SparkException: Job aborted.

Full Stack Trace is attached.

Thanks in advance for your help.

Regards
Arnab

Mime
View raw message