spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Owen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-26757) GraphX EdgeRDDImpl and VertexRDDImpl `count` method cannot handle empty RDDs
Date Wed, 06 Feb 2019 16:50:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-26757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sean Owen updated SPARK-26757:
------------------------------
    Fix Version/s:     (was: 2.3.4)
                   2.3.3

> GraphX EdgeRDDImpl and VertexRDDImpl `count` method cannot handle empty RDDs
> ----------------------------------------------------------------------------
>
>                 Key: SPARK-26757
>                 URL: https://issues.apache.org/jira/browse/SPARK-26757
>             Project: Spark
>          Issue Type: Bug
>          Components: GraphX
>    Affects Versions: 2.3.1, 2.3.2, 2.4.0
>            Reporter: Huon Wilson
>            Assignee: Huon Wilson
>            Priority: Minor
>             Fix For: 2.3.3, 2.4.1, 3.0.0
>
>
> The {{EdgeRDDImpl}} and {{VertexRDDImpl}} types provided by {{GraphX}} throw an {{java.lang.UnsupportedOperationException:
empty collection}} exception if {{count}} is called on an empty instance, when they should
return 0.
> {code:scala}
> import org.apache.spark.graphx.{Graph, Edge}
> val graph = Graph.fromEdges(sc.emptyRDD[Edge[Unit]], 0)
> graph.vertices.count
> graph.edges.count
> {code}
> Running that code in a spark-shell:
> {code:none}
> scala> import org.apache.spark.graphx.{Graph, Edge}
> import org.apache.spark.graphx.{Graph, Edge}
> scala> val graph = Graph.fromEdges(sc.emptyRDD[Edge[Unit]], 0)
> graph: org.apache.spark.graphx.Graph[Int,Unit] = org.apache.spark.graphx.impl.GraphImpl@6879e983
> scala> graph.vertices.count
> java.lang.UnsupportedOperationException: empty collection
>   at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$apply$36.apply(RDD.scala:1031)
>   at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$apply$36.apply(RDD.scala:1031)
>   at scala.Option.getOrElse(Option.scala:121)
>   at org.apache.spark.rdd.RDD$$anonfun$reduce$1.apply(RDD.scala:1031)
>   at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>   at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
>   at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
>   at org.apache.spark.rdd.RDD.reduce(RDD.scala:1011)
>   at org.apache.spark.graphx.impl.VertexRDDImpl.count(VertexRDDImpl.scala:90)
>   ... 49 elided
> scala> graph.edges.count
> java.lang.UnsupportedOperationException: empty collection
>   at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$apply$36.apply(RDD.scala:1031)
>   at org.apache.spark.rdd.RDD$$anonfun$reduce$1$$anonfun$apply$36.apply(RDD.scala:1031)
>   at scala.Option.getOrElse(Option.scala:121)
>   at org.apache.spark.rdd.RDD$$anonfun$reduce$1.apply(RDD.scala:1031)
>   at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
>   at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
>   at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
>   at org.apache.spark.rdd.RDD.reduce(RDD.scala:1011)
>   at org.apache.spark.graphx.impl.EdgeRDDImpl.count(EdgeRDDImpl.scala:90)
>   ... 49 elided
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message