spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zhengruifeng (JIRA)" <>
Subject [jira] [Created] (SPARK-22320) ORC should support VectorUDF/MatrixUDT
Date Fri, 20 Oct 2017 10:56:00 GMT
zhengruifeng created SPARK-22320:

             Summary: ORC should support VectorUDF/MatrixUDT
                 Key: SPARK-22320
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.3.0
            Reporter: zhengruifeng

I save dataframe containing ((ml.Vector}}s in ORC format, when I read it back, the format
is changed.
scala> import

scala> val data = Seq((1,Vectors.dense(1.0,2.0)), (2,Vectors.sparse(8, Array(4), Array(1.0))))
data: Seq[(Int,] = List((1,[1.0,2.0]), (2,(8,[4],[1.0])))

scala> val df = data.toDF("i", "vec")
df: org.apache.spark.sql.DataFrame = [i: int, vec: vector]

scala> df.schema
res0: org.apache.spark.sql.types.StructType = StructType(StructField(i,IntegerType,false),

scala> df.write.orc("/tmp/123")

scala> val df2 ="/tmp/123")
df2: org.apache.spark.sql.DataFrame = [i: int, vec: struct<type: tinyint, size: int ...
2 more fields>]

scala> df2.schema
res3: org.apache.spark.sql.types.StructType = StructType(StructField(i,IntegerType,true),
StructField(vec,StructType(StructField(type,ByteType,true), StructField(size,IntegerType,true),
StructField(indices,ArrayType(IntegerType,true),true), StructField(values,ArrayType(DoubleType,true),true)),true))

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message