spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guillaume Martres (Jira)" <>
Subject [jira] [Commented] (SPARK-29292) Fix internal usages of mutable collection for Seq in 2.13
Date Sun, 03 May 2020 14:49:00 GMT


Guillaume Martres commented on SPARK-29292:

Right, so ideally you'd change Seq to collection.Seq for parameter types but not for result
type, but that might be tricky to manage and it looks like that was already considered and
dismissed in the discussion you linked to.

Going back to the issue with copying when using toSeq everywhere, one way to reduce that would
be to use builders when possible: instead of creating an ArrayBuffer, filling it, then copying
using toSeq, one can create a `Seq.newBuilder`, fill it, then calll `result` on it to get
back an immutable Seq without copying. (This might be worse than using ArrayBuffer in 2.12,
because the default builder will construct a List, ideally one would use immutable.ArraySeq.newBuilder
to get something backed by a plain Array but that one doesn't exist in 2.12). If most of the
collections being copied are small then this might not make a significant difference any way.

> Fix internal usages of mutable collection for Seq in 2.13
> ---------------------------------------------------------
>                 Key: SPARK-29292
>                 URL:
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Spark Core
>    Affects Versions: 3.0.0
>            Reporter: Sean R. Owen
>            Assignee: Sean R. Owen
>            Priority: Minor
> Kind of related to, but a simpler subset.

> In 2.13, a mutable collection can't be returned as a {{scala.collection.Seq}}. It's easy
enough to call .toSeq on these as that still works on 2.12.
> {code}
> [ERROR] [Error] /Users/seanowen/Documents/spark_2.13/core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala:467:
type mismatch;
>  found   : Seq[String] (in scala.collection) 
>  required: Seq[String] (in scala.collection.immutable) 
> {code}

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message