spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Holmberg <>
Subject Re: Stress testing hdfs with Spark
Date Tue, 05 Apr 2016 19:56:44 GMT
I'm trying to get rough estimate how much data I can write within certain time period (GB/sec).

On 05 Apr 2016, at 22:49, Mich Talebzadeh <<>>

Hi Jan,

What is the definition of stress test in here? What are the matrices? Throughput of data,
latency, velocity, volume?


Dr Mich Talebzadeh


On 5 April 2016 at 20:42, Jan Holmberg <<>>
I'm trying to figure out how to write lots of data from each worker. I tried rdd.saveAsTextFile
but got OOM when generating 1024MB string for a worker. Increasing worker memory would mean
that I should drop the number of workers.
Soo, any idea how to write ex. 1gb file from each worker?

To unsubscribe, e-mail:<>
For additional commands, e-mail:<>

View raw message