spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zakaria Hili <zakah...@gmail.com>
Subject convert spark dataframe to numpy (ndarray)
Date Fri, 28 Oct 2016 08:26:31 GMT
Hi,

Is there any way to convert a spark dataframe into numpy ndarray without
using toPandas operation ?

Example:

C1   C2  C3   C4 0.7 3.0 1000 109540.9 4.2 1200 12345

I want to get this output:

[(0.7, 3.0, 1000L, 10954),(0.9, 4.2, 1200L, 12345)],
    dtype=[('C1', '<f8'), ('C2', '<f8'), ('C3', '<i8'), ('C4', '<f8'),(2,))])

NB:I'm working with pyspark 1.6.2


Thank you
ᐧ

Mime
View raw message