spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zenglong chen <czlong.kel...@gmail.com>
Subject Spark ImportError: No module named XXX
Date Fri, 19 Jul 2019 10:57:15 GMT
Hi,all:
    aused by: org.apache.spark.api.python.PythonException: Traceback (most
recent call last):
  File "/home/ubuntu/spark-2.4.3/python/lib/pyspark.zip/pyspark/worker.py",
line 364, in main
    func, profiler, deserializer, serializer = read_command(pickleSer,
infile)
  File "/home/ubuntu/spark-2.4.3/python/lib/pyspark.zip/pyspark/worker.py",
line 69, in read_command
    command = serializer._read_with_length(file)
  File
"/home/ubuntu/spark-2.4.3/python/lib/pyspark.zip/pyspark/serializers.py",
line 172, in _read_with_length
    return self.loads(obj)
  File
"/home/ubuntu/spark-2.4.3/python/lib/pyspark.zip/pyspark/serializers.py",
line 583, in loads
    return pickle.loads(obj)
ImportError: No module named
risk_platform.mutli_process_thread.decision_tree

Is the problem caused by comunication between spark master and slave in
pickle way?
Python cannot pickle instancemethod and code object,So that is why load
error?
Thanks for answer!

Mime
View raw message