phoenix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PHOENIX-2469) Problem with ARRAYs in Phoenix-spark
Date Wed, 02 Dec 2015 15:50:10 GMT

    [ https://issues.apache.org/jira/browse/PHOENIX-2469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15035997#comment-15035997
] 

Hudson commented on PHOENIX-2469:
---------------------------------

SUCCESS: Integrated in Phoenix-master #997 (See [https://builds.apache.org/job/Phoenix-master/997/])
PHOENIX-2469 Problem with arrays in phoenix-spark (Dawid Wysakowicz) (jmahonin: rev b8faae52c6bee91393678e74de09ab8a215da856)
* phoenix-spark/src/main/scala/org/apache/phoenix/spark/PhoenixRecordWritable.scala
* phoenix-spark/src/it/resources/setup.sql
* phoenix-spark/pom.xml
* phoenix-spark/src/it/scala/org/apache/phoenix/spark/PhoenixSparkIT.scala


> Problem with ARRAYs in Phoenix-spark
> ------------------------------------
>
>                 Key: PHOENIX-2469
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-2469
>             Project: Phoenix
>          Issue Type: Bug
>            Reporter: Dawid Wysakowicz
>             Fix For: 4.7.0
>
>         Attachments: phoenix-spark.patch
>
>
> I've recently found some behaviour that I found buggy when working with phoenix-spark
and arrays.
> Take a look at those unit tests:
> {code:java}
>   test("Can save arrays from custom dataframes back to phoenix") {
>     val dataSet = List(Row(2L, Array("String1", "String2", "String3")))
>     val sqlContext = new SQLContext(sc)
>     val schema = StructType(
>         Seq(StructField("ID", LongType, nullable = false),
>             StructField("VCARRAY", ArrayType(StringType))))
>     val rowRDD = sc.parallelize(dataSet)
>     // Apply the schema to the RDD.
>     val df = sqlContext.createDataFrame(rowRDD, schema)
>     df.write
>       .format("org.apache.phoenix.spark")
>       .options(Map("table" -> "ARRAY_TEST_TABLE", "zkUrl" -> quorumAddress))
>       .mode(SaveMode.Overwrite)
>       .save()
>   }
> {code}
> {code:java}
>   test("Can save arrays of AnyVal type back to phoenix") {
>     val dataSet = List((2L, Array(1, 2, 3), Array(1L, 2L, 3L)))
>     sc
>       .parallelize(dataSet)
>       .saveToPhoenix(
>         "ARRAY_ANYVAL_TEST_TABLE",
>         Seq("ID", "INTARRAY", "BIGINTARRAY"),
>         zkUrl = Some(quorumAddress)
>       )
>     // Load the results back
>     val stmt = conn.createStatement()
>     val rs = stmt.executeQuery("SELECT INTARRAY, BIGINTARRAY FROM ARRAY_ANYVAL_TEST_TABLE
WHERE ID = 2")
>     rs.next()
>     val intArray = rs.getArray(1).getArray().asInstanceOf[Array[Int]]
>     val longArray = rs.getArray(2).getArray().asInstanceOf[Array[Long]]
>     // Verify the arrays are equal
>     intArray shouldEqual dataSet(0)._2
>     longArray shouldEqual dataSet(0)._3
>   }
> {code}
> Both fail with some ClassCastExceptions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message