spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Ash (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-534) Make SparkContext thread-safe
Date Sat, 18 Oct 2014 00:27:34 GMT

     [ https://issues.apache.org/jira/browse/SPARK-534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Ash updated SPARK-534:
-----------------------------
    Description: 
SparkEnv (used by SparkContext) is not thread-safe and it causes issues with scala's Futures
and parrallel collections.
For example, this will not work:

{code}
val f = Futures.future({
  sc.textFile("hdfs://....")
})
f.apply()
{code}

Workaround for now:

{code}
val f = Futures.future({
  SparkEnv.set(sc.env)
  sc.textFile("hdfs://....")
})
f.apply()
{code}

  was:
SparkEnv (used by SparkContext) is not thread-safe and it causes issues with scala's Futures
and parrallel collections.
For example, this will not work:

val f = Futures.future({
  sc.textFile("hdfs://....")
})
f.apply()

Workaround for now:

val f = Futures.future({
  SparkEnv.set(sc.env)
  sc.textFile("hdfs://....")
})
f.apply()


> Make SparkContext thread-safe
> -----------------------------
>
>                 Key: SPARK-534
>                 URL: https://issues.apache.org/jira/browse/SPARK-534
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 0.5.0, 0.5.1, 0.6.0, 0.6.1, 0.7.0, 0.6.2, 0.5.2, 0.7.1, 0.7.2, 0.7.3
>            Reporter: tjhunter
>            Priority: Blocker
>
> SparkEnv (used by SparkContext) is not thread-safe and it causes issues with scala's
Futures and parrallel collections.
> For example, this will not work:
> {code}
> val f = Futures.future({
>   sc.textFile("hdfs://....")
> })
> f.apply()
> {code}
> Workaround for now:
> {code}
> val f = Futures.future({
>   SparkEnv.set(sc.env)
>   sc.textFile("hdfs://....")
> })
> f.apply()
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message