spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Slavitch <slavi...@gmail.com>
Subject Re: Eliminating shuffle write and spill disk IO reads/writes in Spark
Date Fri, 01 Apr 2016 23:27:29 GMT
Yes we see it on final write.  Our preference is to eliminate this.

On Fri, Apr 1, 2016, 7:25 PM Saisai Shao <sai.sai.shao@gmail.com> wrote:

> Hi Michael, shuffle data (mapper output) have to be materialized into disk
> finally, no matter how large memory you have, it is the design purpose of
> Spark. In you scenario, since you have a big memory, shuffle spill should
> not happen frequently, most of the disk IO you see might be final shuffle
> file write.
>
> So if you want to avoid this disk IO, you could use ramdisk as Reynold
> suggested. If you want to avoid FS overhead of ramdisk, you could try to
> hack a new shuffle implementation, since shuffle framework is pluggable.
>
>
> On Sat, Apr 2, 2016 at 6:48 AM, Michael Slavitch <slavitch@gmail.com>
> wrote:
>
>> As I mentioned earlier this flag is now ignored.
>>
>>
>> On Fri, Apr 1, 2016, 6:39 PM Michael Slavitch <slavitch@gmail.com> wrote:
>>
>>> Shuffling a 1tb set of keys and values (aka sort by key)  results in
>>> about 500gb of io to disk if compression is enabled. Is there any way to
>>> eliminate shuffling causing io?
>>>
>>> On Fri, Apr 1, 2016, 6:32 PM Reynold Xin <rxin@databricks.com> wrote:
>>>
>>>> Michael - I'm not sure if you actually read my email, but spill has
>>>> nothing to do with the shuffle files on disk. It was for the partitioning
>>>> (i.e. sorting) process. If that flag is off, Spark will just run out of
>>>> memory when data doesn't fit in memory.
>>>>
>>>>
>>>> On Fri, Apr 1, 2016 at 3:28 PM, Michael Slavitch <slavitch@gmail.com>
>>>> wrote:
>>>>
>>>>> RAMdisk is a fine interim step but there is a lot of layers eliminated
>>>>> by keeping things in memory unless there is need for spillover.   At
one
>>>>> time there was support for turning off spilling.  That was eliminated.
>>>>> Why?
>>>>>
>>>>>
>>>>> On Fri, Apr 1, 2016, 6:05 PM Mridul Muralidharan <mridul@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I think Reynold's suggestion of using ram disk would be a good way
to
>>>>>> test if these are the bottlenecks or something else is.
>>>>>> For most practical purposes, pointing local dir to ramdisk should
>>>>>> effectively give you 'similar' performance as shuffling from memory.
>>>>>>
>>>>>> Are there concerns with taking that approach to test ? (I dont see
>>>>>> any, but I am not sure if I missed something).
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Mridul
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Apr 1, 2016 at 2:10 PM, Michael Slavitch <slavitch@gmail.com>
>>>>>> wrote:
>>>>>> > I totally disagree that it’s not a problem.
>>>>>> >
>>>>>> > - Network fetch throughput on 40G Ethernet exceeds the throughput
>>>>>> of NVME
>>>>>> > drives.
>>>>>> > - What Spark is depending on is Linux’s IO cache as an effective
>>>>>> buffer pool
>>>>>> > This is fine for small jobs but not for jobs with datasets in
the
>>>>>> TB/node
>>>>>> > range.
>>>>>> > - On larger jobs flushing the cache causes Linux to block.
>>>>>> > - On a modern 56-hyperthread 2-socket host the latency caused
by
>>>>>> multiple
>>>>>> > executors writing out to disk increases greatly.
>>>>>> >
>>>>>> > I thought the whole point of Spark was in-memory computing?
 It’s
>>>>>> in fact
>>>>>> > in-memory for some things but  use spark.local.dir as a buffer
pool
>>>>>> of
>>>>>> > others.
>>>>>> >
>>>>>> > Hence, the performance of  Spark is gated by the performance
of
>>>>>> > spark.local.dir, even on large memory systems.
>>>>>> >
>>>>>> > "Currently it is not possible to not write shuffle files to
disk.”
>>>>>> >
>>>>>> > What changes >would< make it possible?
>>>>>> >
>>>>>> > The only one that seems possible is to clone the shuffle service
>>>>>> and make it
>>>>>> > in-memory.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On Apr 1, 2016, at 4:57 PM, Reynold Xin <rxin@databricks.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > spark.shuffle.spill actually has nothing to do with whether
we
>>>>>> write shuffle
>>>>>> > files to disk. Currently it is not possible to not write shuffle
>>>>>> files to
>>>>>> > disk, and typically it is not a problem because the network
fetch
>>>>>> throughput
>>>>>> > is lower than what disks can sustain. In most cases, especially
>>>>>> with SSDs,
>>>>>> > there is little difference between putting all of those in memory
>>>>>> and on
>>>>>> > disk.
>>>>>> >
>>>>>> > However, it is becoming more common to run Spark on a few number
of
>>>>>> beefy
>>>>>> > nodes (e.g. 2 nodes each with 1TB of RAM). We do want to look
into
>>>>>> improving
>>>>>> > performance for those. Meantime, you can setup local ramdisks
on
>>>>>> each node
>>>>>> > for shuffle writes.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > On Fri, Apr 1, 2016 at 11:32 AM, Michael Slavitch <
>>>>>> slavitch@gmail.com>
>>>>>> > wrote:
>>>>>> >>
>>>>>> >> Hello;
>>>>>> >>
>>>>>> >> I’m working on spark with very large memory systems (2TB+)
and
>>>>>> notice that
>>>>>> >> Spark spills to disk in shuffle.  Is there a way to force
spark to
>>>>>> stay in
>>>>>> >> memory when doing shuffle operations?   The goal is to keep
the
>>>>>> shuffle data
>>>>>> >> either in the heap or in off-heap memory (in 1.6.x) and
never
>>>>>> touch the IO
>>>>>> >> subsystem.  I am willing to have the job fail if it runs
out of
>>>>>> RAM.
>>>>>> >>
>>>>>> >> spark.shuffle.spill true  is deprecated in 1.6 and does
not work in
>>>>>> >> Tungsten sort in 1.5.x
>>>>>> >>
>>>>>> >> "WARN UnsafeShuffleManager: spark.shuffle.spill was set
to false,
>>>>>> but this
>>>>>> >> is ignored by the tungsten-sort shuffle manager; its optimized
>>>>>> shuffles will
>>>>>> >> continue to spill to disk when necessary.”
>>>>>> >>
>>>>>> >> If this is impossible via configuration changes what code
changes
>>>>>> would be
>>>>>> >> needed to accomplish this?
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> ---------------------------------------------------------------------
>>>>>> >> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>>>> >> For additional commands, e-mail: user-help@spark.apache.org
>>>>>> >>
>>>>>> >
>>>>>> >
>>>>>>
>>>>> --
>>>>> Michael Slavitch
>>>>> 62 Renfrew Ave.
>>>>> Ottawa Ontario
>>>>> K1S 1Z5
>>>>>
>>>>
>>>> --
>>> Michael Slavitch
>>> 62 Renfrew Ave.
>>> Ottawa Ontario
>>> K1S 1Z5
>>>
>> --
>> Michael Slavitch
>> 62 Renfrew Ave.
>> Ottawa Ontario
>> K1S 1Z5
>>
>
> --
Michael Slavitch
62 Renfrew Ave.
Ottawa Ontario
K1S 1Z5

Mime
View raw message