spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saisai Shao <sai.sai.s...@gmail.com>
Subject Re: Eliminating shuffle write and spill disk IO reads/writes in Spark
Date Fri, 01 Apr 2016 23:44:09 GMT
So I think ramdisk is simple way to try.

Besides I think Reynold's suggestion is quite valid, with such high-end
machine, putting everything in memory might not improve the performance a
lot as assumed. Since bottleneck will be shifted, like memory bandwidth,
NUMA, CPU efficiency (serialization-deserialization, data processing...).
Also code design should well consider such usage scenario, to use resource
more efficiently.

Thanks
Saisai

On Sat, Apr 2, 2016 at 7:27 AM, Michael Slavitch <slavitch@gmail.com> wrote:

> Yes we see it on final write.  Our preference is to eliminate this.
>
>
> On Fri, Apr 1, 2016, 7:25 PM Saisai Shao <sai.sai.shao@gmail.com> wrote:
>
>> Hi Michael, shuffle data (mapper output) have to be materialized into
>> disk finally, no matter how large memory you have, it is the design purpose
>> of Spark. In you scenario, since you have a big memory, shuffle spill
>> should not happen frequently, most of the disk IO you see might be final
>> shuffle file write.
>>
>> So if you want to avoid this disk IO, you could use ramdisk as Reynold
>> suggested. If you want to avoid FS overhead of ramdisk, you could try to
>> hack a new shuffle implementation, since shuffle framework is pluggable.
>>
>>
>> On Sat, Apr 2, 2016 at 6:48 AM, Michael Slavitch <slavitch@gmail.com>
>> wrote:
>>
>>> As I mentioned earlier this flag is now ignored.
>>>
>>>
>>> On Fri, Apr 1, 2016, 6:39 PM Michael Slavitch <slavitch@gmail.com>
>>> wrote:
>>>
>>>> Shuffling a 1tb set of keys and values (aka sort by key)  results in
>>>> about 500gb of io to disk if compression is enabled. Is there any way to
>>>> eliminate shuffling causing io?
>>>>
>>>> On Fri, Apr 1, 2016, 6:32 PM Reynold Xin <rxin@databricks.com> wrote:
>>>>
>>>>> Michael - I'm not sure if you actually read my email, but spill has
>>>>> nothing to do with the shuffle files on disk. It was for the partitioning
>>>>> (i.e. sorting) process. If that flag is off, Spark will just run out
of
>>>>> memory when data doesn't fit in memory.
>>>>>
>>>>>
>>>>> On Fri, Apr 1, 2016 at 3:28 PM, Michael Slavitch <slavitch@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> RAMdisk is a fine interim step but there is a lot of layers
>>>>>> eliminated by keeping things in memory unless there is need for spillover.
>>>>>>   At one time there was support for turning off spilling.  That was
>>>>>> eliminated.  Why?
>>>>>>
>>>>>>
>>>>>> On Fri, Apr 1, 2016, 6:05 PM Mridul Muralidharan <mridul@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I think Reynold's suggestion of using ram disk would be a good
way to
>>>>>>> test if these are the bottlenecks or something else is.
>>>>>>> For most practical purposes, pointing local dir to ramdisk should
>>>>>>> effectively give you 'similar' performance as shuffling from
memory.
>>>>>>>
>>>>>>> Are there concerns with taking that approach to test ? (I dont
see
>>>>>>> any, but I am not sure if I missed something).
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> Mridul
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Apr 1, 2016 at 2:10 PM, Michael Slavitch <slavitch@gmail.com>
>>>>>>> wrote:
>>>>>>> > I totally disagree that it’s not a problem.
>>>>>>> >
>>>>>>> > - Network fetch throughput on 40G Ethernet exceeds the throughput
>>>>>>> of NVME
>>>>>>> > drives.
>>>>>>> > - What Spark is depending on is Linux’s IO cache as an
effective
>>>>>>> buffer pool
>>>>>>> > This is fine for small jobs but not for jobs with datasets
in the
>>>>>>> TB/node
>>>>>>> > range.
>>>>>>> > - On larger jobs flushing the cache causes Linux to block.
>>>>>>> > - On a modern 56-hyperthread 2-socket host the latency caused
by
>>>>>>> multiple
>>>>>>> > executors writing out to disk increases greatly.
>>>>>>> >
>>>>>>> > I thought the whole point of Spark was in-memory computing?
 It’s
>>>>>>> in fact
>>>>>>> > in-memory for some things but  use spark.local.dir as a
buffer
>>>>>>> pool of
>>>>>>> > others.
>>>>>>> >
>>>>>>> > Hence, the performance of  Spark is gated by the performance
of
>>>>>>> > spark.local.dir, even on large memory systems.
>>>>>>> >
>>>>>>> > "Currently it is not possible to not write shuffle files
to disk.”
>>>>>>> >
>>>>>>> > What changes >would< make it possible?
>>>>>>> >
>>>>>>> > The only one that seems possible is to clone the shuffle
service
>>>>>>> and make it
>>>>>>> > in-memory.
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > On Apr 1, 2016, at 4:57 PM, Reynold Xin <rxin@databricks.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > spark.shuffle.spill actually has nothing to do with whether
we
>>>>>>> write shuffle
>>>>>>> > files to disk. Currently it is not possible to not write
shuffle
>>>>>>> files to
>>>>>>> > disk, and typically it is not a problem because the network
fetch
>>>>>>> throughput
>>>>>>> > is lower than what disks can sustain. In most cases, especially
>>>>>>> with SSDs,
>>>>>>> > there is little difference between putting all of those
in memory
>>>>>>> and on
>>>>>>> > disk.
>>>>>>> >
>>>>>>> > However, it is becoming more common to run Spark on a few
number
>>>>>>> of beefy
>>>>>>> > nodes (e.g. 2 nodes each with 1TB of RAM). We do want to
look into
>>>>>>> improving
>>>>>>> > performance for those. Meantime, you can setup local ramdisks
on
>>>>>>> each node
>>>>>>> > for shuffle writes.
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > On Fri, Apr 1, 2016 at 11:32 AM, Michael Slavitch <
>>>>>>> slavitch@gmail.com>
>>>>>>> > wrote:
>>>>>>> >>
>>>>>>> >> Hello;
>>>>>>> >>
>>>>>>> >> I’m working on spark with very large memory systems
(2TB+) and
>>>>>>> notice that
>>>>>>> >> Spark spills to disk in shuffle.  Is there a way to
force spark
>>>>>>> to stay in
>>>>>>> >> memory when doing shuffle operations?   The goal is
to keep the
>>>>>>> shuffle data
>>>>>>> >> either in the heap or in off-heap memory (in 1.6.x)
and never
>>>>>>> touch the IO
>>>>>>> >> subsystem.  I am willing to have the job fail if it
runs out of
>>>>>>> RAM.
>>>>>>> >>
>>>>>>> >> spark.shuffle.spill true  is deprecated in 1.6 and does
not work
>>>>>>> in
>>>>>>> >> Tungsten sort in 1.5.x
>>>>>>> >>
>>>>>>> >> "WARN UnsafeShuffleManager: spark.shuffle.spill was
set to false,
>>>>>>> but this
>>>>>>> >> is ignored by the tungsten-sort shuffle manager; its
optimized
>>>>>>> shuffles will
>>>>>>> >> continue to spill to disk when necessary.”
>>>>>>> >>
>>>>>>> >> If this is impossible via configuration changes what
code changes
>>>>>>> would be
>>>>>>> >> needed to accomplish this?
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> >> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>>>>> >> For additional commands, e-mail: user-help@spark.apache.org
>>>>>>> >>
>>>>>>> >
>>>>>>> >
>>>>>>>
>>>>>> --
>>>>>> Michael Slavitch
>>>>>> 62 Renfrew Ave.
>>>>>> Ottawa Ontario
>>>>>> K1S 1Z5
>>>>>>
>>>>>
>>>>> --
>>>> Michael Slavitch
>>>> 62 Renfrew Ave.
>>>> Ottawa Ontario
>>>> K1S 1Z5
>>>>
>>> --
>>> Michael Slavitch
>>> 62 Renfrew Ave.
>>> Ottawa Ontario
>>> K1S 1Z5
>>>
>>
>> --
> Michael Slavitch
> 62 Renfrew Ave.
> Ottawa Ontario
> K1S 1Z5
>

Mime
View raw message