spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hxngillani <f2017279...@umt.edu.pk>
Subject How to load Python Pickle File in Spark Data frame
Date Mon, 26 Aug 2019 05:23:34 GMT
Hello  Dear Members 
i want to train model using Bigdl, i have data set of Medical images in the
form of pickle object files (,pck).that pickle file is a 3D image(3D array) 

i have tried 
pickleRdd = sc.pickleFilehome/student/BigDL- 
 trainings/elephantscale/data/volumetric_data/329637-8.pck
 sqlContext = SQLContext(sc)
 df = sqlContext.createDataFrame(pickleRdd)

this code throwing and error 
Caused by: java.io.IOException:
file:/home/student/BigDL-trainings/elephantscale/data/volumetric_data/329637-8.pck
not a SequenceFile


the things i came to know is that 
The function
sc.pickleFile
loads a pickle file that is created by 
rdd.saveAsPickleFile

I am loading a pickle file that is created by Python's "pickle" module  
My Question is that  is there any way to load that file in spark data frame 



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message