spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyukjin Kwon (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-12484) DataFrame withColumn() does not work in Java
Date Tue, 10 Jan 2017 06:07:58 GMT

     [ https://issues.apache.org/jira/browse/SPARK-12484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Hyukjin Kwon resolved SPARK-12484.
----------------------------------
    Resolution: Invalid

The API works as expected and they are being tested. I don't think just saying {{withColumn}}
in Java does not work makes sense at all.

Also, according to the email thread you suggested, it seems not really issue in Spark.

I think it'd make sense open another issue or reopen this after fixing the title and description
if it is really an issue.

Per description/title, it does not make sense and therefore I am resolving this as {{Invalid}}.

> DataFrame withColumn() does not work in Java
> --------------------------------------------
>
>                 Key: SPARK-12484
>                 URL: https://issues.apache.org/jira/browse/SPARK-12484
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.5.2
>         Environment: mac El Cap. 10.11.2
> Java 8
>            Reporter: Andrew Davidson
>         Attachments: UDFTest.java
>
>
> DataFrame transformerdDF = df.withColumn(fieldName, newCol); raises
>  org.apache.spark.sql.AnalysisException: resolved attribute(s) _c0#2 missing from id#0,labelStr#1
in operator !Project [id#0,labelStr#1,_c0#2 AS transformedByUDF#3];
>         at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:37)
>         at org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:44)
>         at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:154)
>         at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:49)
>         at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:103)
>         at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:49)
>         at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:44)
>         at org.apache.spark.sql.SQLContext$QueryExecution.assertAnalyzed(SQLContext.scala:914)
>         at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:132)
>         at org.apache.spark.sql.DataFrame.org$apache$spark$sql$DataFrame$$logicalPlanToDataFrame(DataFrame.scala:154)
>         at org.apache.spark.sql.DataFrame.select(DataFrame.scala:691)
>         at org.apache.spark.sql.DataFrame.withColumn(DataFrame.scala:1150)
>         at com.pws.fantasySport.ml.UDFTest.test(UDFTest.java:75)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message