spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyukjin Kwon (Jira)" <j...@apache.org>
Subject [jira] [Created] (SPARK-29339) Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)
Date Thu, 03 Oct 2019 03:00:02 GMT
Hyukjin Kwon created SPARK-29339:
------------------------------------

             Summary: Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor
build)
                 Key: SPARK-29339
                 URL: https://issues.apache.org/jira/browse/SPARK-29339
             Project: Spark
          Issue Type: Improvement
          Components: SparkR
    Affects Versions: 3.0.0
            Reporter: Hyukjin Kwon


dapply and gapply with Arrow optimization and Arrow 0.14 seems failing:

{code}
> collect(dapply(df, function(rdf) { data.frame(rdf$gear + 1) }, structType("gear double")))
Error in readBin(con, raw(), as.integer(dataLen), endian = "big") :
  invalid 'n' argument
{code}

We should fix it and also test it in AppVeyor



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message