spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From DB Tsai <dbt...@stanford.edu>
Subject Is spark context in local mode thread-safe?
Date Mon, 09 Jun 2014 22:50:46 GMT
Hi guys,

We would like to use spark hadoop api to get the first couple hundred
lines in design time to quickly show users the file-structure/meta
data, and the values in those lines without launching the full spark
job in cluster.

Since we're web-based application, there will be multiple users using
the spark hadoop api, for exmaple, sc.textFile(filePath). I wonder if
those APIs are thread-safe in local mode (each user will have its own
SparkContext object).

Secondly, it seems that even in local mode, the jetty UI tracker will
be lunched. For this kind of cheap operation, having jetty UI tracker
for each operation will be very expensive. Is there a way to disable
this behavior?

Thanks.

Sincerely,

DB Tsai
-------------------------------------------------------
My Blog: https://www.dbtsai.com
LinkedIn: https://www.linkedin.com/in/dbtsai

Mime
View raw message