phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dawid Wysakowicz (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-2469) Problem with ARRAYs in Phoenix-spark
Date Tue, 01 Dec 2015 18:19:11 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-2469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15034258#comment-15034258
] 

Dawid Wysakowicz commented on PHOENIX-2469:
-------------------------------------------

Ok. I've applied you suggestion regarding the BINARY, VARBINARY, BINARY[] and added tests
for those cases. I've also had to add also cases for WrappedArray as it changed in spark 1.5.
I've left the ArrayBuffer for backward compatibility. 

Hope this time it will be ok.

> Problem with ARRAYs in Phoenix-spark
> ------------------------------------
>
>                 Key: PHOENIX-2469
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-2469
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: Dawid Wysakowicz
>         Attachments: phoenix-spark.patch
>
>
> I've recently found some behaviour that I found buggy when working with phoenix-spark
and arrays.
> Take a look at those unit tests:
> {code:java}
>   test("Can save arrays from custom dataframes back to phoenix") {
>     val dataSet = List(Row(2L, Array("String1", "String2", "String3")))
>     val sqlContext = new SQLContext(sc)
>     val schema = StructType(
>         Seq(StructField("ID", LongType, nullable = false),
>             StructField("VCARRAY", ArrayType(StringType))))
>     val rowRDD = sc.parallelize(dataSet)
>     // Apply the schema to the RDD.
>     val df = sqlContext.createDataFrame(rowRDD, schema)
>     df.write
>       .format("org.apache.phoenix.spark")
>       .options(Map("table" -> "ARRAY_TEST_TABLE", "zkUrl" -> quorumAddress))
>       .mode(SaveMode.Overwrite)
>       .save()
>   }
> {code}
> {code:java}
>   test("Can save arrays of AnyVal type back to phoenix") {
>     val dataSet = List((2L, Array(1, 2, 3), Array(1L, 2L, 3L)))
>     sc
>       .parallelize(dataSet)
>       .saveToPhoenix(
>         "ARRAY_ANYVAL_TEST_TABLE",
>         Seq("ID", "INTARRAY", "BIGINTARRAY"),
>         zkUrl = Some(quorumAddress)
>       )
>     // Load the results back
>     val stmt = conn.createStatement()
>     val rs = stmt.executeQuery("SELECT INTARRAY, BIGINTARRAY FROM ARRAY_ANYVAL_TEST_TABLE
WHERE ID = 2")
>     rs.next()
>     val intArray = rs.getArray(1).getArray().asInstanceOf[Array[Int]]
>     val longArray = rs.getArray(2).getArray().asInstanceOf[Array[Long]]
>     // Verify the arrays are equal
>     intArray shouldEqual dataSet(0)._2
>     longArray shouldEqual dataSet(0)._3
>   }
> {code}
> Both fail with some ClassCastExceptions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message