spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fernando Paladini <>
Subject "Method json([class java.util.HashMap]) does not exist" when reading JSON
Date Tue, 29 Sep 2015 14:23:05 GMT
Hello guys,

I'm very new to Spark and I'm having some troubles when reading a JSON to
dataframe on PySpark.

I'm getting a JSON object from an API response and I would like to store it
in Spark as a DataFrame (I've read that DataFrame is better than RDD,
that's accurate?). For what I've read
on documentation, I just need to call the method in
order to do what I want.

*Following is the code from my test application:*
json_object = json.loads(response.text)
sc = SparkContext("local", appName="JSON to RDD")
sqlContext = SQLContext(sc)
dataframe =

*The problem is that when I run **"spark-submit" I got the
following error:*
15/09/29 01:18:54 INFO BlockManagerMasterEndpoint: Registering block
manager localhost:48634 with 530.0 MB RAM, BlockManagerId(driver,
localhost, 48634)
15/09/29 01:18:54 INFO BlockManagerMaster: Registered BlockManager
Traceback (most recent call last):
  File "/home/paladini/ufxc/lisha/learning/spark-api-kairos/", line
35, in <module>
    dataframe =
  File "/opt/spark/python/lib/", line
144, in json
  File "/opt/spark/python/lib/",
line 538, in __call__
  File "/opt/spark/python/lib/", line 36,
in deco
  File "/opt/spark/python/lib/", line
304, in get_return_value
py4j.protocol.Py4JError: An error occurred while calling o21.json. Trace:
py4j.Py4JException: Method json([class java.util.HashMap]) does not exist
    at py4j.reflection.ReflectionEngine.getMethod(
    at py4j.reflection.ReflectionEngine.getMethod(
    at py4j.Gateway.invoke(
    at py4j.commands.AbstractCommand.invokeMethod(
    at py4j.commands.CallCommand.execute(

*What I'm doing wrong? *
Check out this gist <>
to see the JSON I'm trying to load.


View raw message