spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anders Arpteg <arp...@spotify.com>
Subject Re: Large number of conf broadcasts
Date Mon, 26 Oct 2015 08:55:02 GMT
Nice Koert, lets hope it gets merged soon.

/Anders

On Fri, Oct 23, 2015 at 6:32 PM Koert Kuipers <koert@tresata.com> wrote:

> https://github.com/databricks/spark-avro/pull/95
>
> On Fri, Oct 23, 2015 at 5:01 AM, Koert Kuipers <koert@tresata.com> wrote:
>
>> oh no wonder... it undoes the glob (i was reading from /some/path/*),
>> creates a hadoopRdd for every path, and then creates a union of them using
>> UnionRDD.
>>
>> thats not what i want... no need to do union. AvroInpuFormat already has
>> the ability to handle globs (or multiple paths comma separated) very
>> efficiently. AvroRelation should just pass the paths (comma separated).
>>
>>
>>
>>
>> On Thu, Oct 22, 2015 at 1:37 PM, Anders Arpteg <arpteg@spotify.com>
>> wrote:
>>
>>> Yes, seems unnecessary. I actually tried patching the
>>> com.databricks.spark.avro reader to only broadcast once per dataset,
>>> instead of every single file/partition. It seems to work just as fine, and
>>> there are significantly less broadcasts and not seeing out of memory issues
>>> any more. Strange that more people does not react to this, since the
>>> broadcasting seems completely unnecessary...
>>>
>>> Best,
>>> Anders
>>>
>>>
>>> On Thu, Oct 22, 2015 at 7:03 PM Koert Kuipers <koert@tresata.com> wrote:
>>>
>>>> i am seeing the same thing. its gona completely crazy creating
>>>> broadcasts for the last 15 mins or so. killing it...
>>>>
>>>> On Thu, Sep 24, 2015 at 1:24 PM, Anders Arpteg <arpteg@spotify.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Running spark 1.5.0 in yarn-client mode, and am curios in why there
>>>>> are so many broadcast being done when loading datasets with large number
of
>>>>> partitions/files. Have datasets with thousands of partitions, i.e. hdfs
>>>>> files in the avro folder, and sometime loading hundreds of these large
>>>>> datasets. Believe I have located the broadcast to line
>>>>> SparkContext.scala:1006. It seems to just broadcast the hadoop
>>>>> configuration, and I don't see why it should be necessary to broadcast
that
>>>>> for EVERY file? Wouldn't it be possible to reuse the same broadcast
>>>>> configuration? It hardly the case the the configuration would be different
>>>>> between each file in a single dataset. Seems to be wasting lots of memory
>>>>> and needs to persist unnecessarily to disk (see below again).
>>>>>
>>>>> Thanks,
>>>>> Anders
>>>>>
>>>>> 15/09/24 17:11:11 INFO BlockManager: Writing block
>>>>> broadcast_1871_piece0 to disk
>>>>>  [19/49086]15/09/24 17:11:11 INFO BlockManagerInfo: Added
>>>>> broadcast_1871_piece0 on disk on 10.254.35.24:49428 (size: 23.1 KB)
>>>>> 15/09/24 17:11:11 INFO MemoryStore: Block broadcast_4803_piece0 stored
>>>>> as bytes in memory (estimated size 23.1 KB, free 2.4 KB)
>>>>> 15/09/24 17:11:11 INFO BlockManagerInfo: Added broadcast_4803_piece0
>>>>> in memory on 10.254.35.24:49428 (size: 23.1 KB, free: 464.0 MB)
>>>>> 15/09/24 17:11:11 INFO SpotifySparkContext: Created broadcast 4803
>>>>> from hadoopFile at AvroRelation.scala:121
>>>>> 15/09/24 17:11:11 WARN MemoryStore: Failed to reserve initial memory
>>>>> threshold of 1024.0 KB for computing block broadcast_4804 in memory
>>>>> .
>>>>> 15/09/24 17:11:11 WARN MemoryStore: Not enough space to cache
>>>>> broadcast_4804 in memory! (computed 496.0 B so far)
>>>>> 15/09/24 17:11:11 INFO MemoryStore: Memory use = 530.3 MB (blocks) +
>>>>> 0.0 B (scratch space shared across 0 tasks(s)) = 530.3 MB. Storage
>>>>> limit = 530.3 MB.
>>>>> 15/09/24 17:11:11 WARN MemoryStore: Persisting block broadcast_4804 to
>>>>> disk instead.
>>>>> 15/09/24 17:11:11 INFO MemoryStore: ensureFreeSpace(23703) called with
>>>>> curMem=556036460, maxMem=556038881
>>>>> 15/09/24 17:11:11 INFO MemoryStore: 1 blocks selected for dropping
>>>>> 15/09/24 17:11:11 INFO BlockManager: Dropping block
>>>>> broadcast_1872_piece0 from memory
>>>>> 15/09/24 17:11:11 INFO BlockManager: Writing block
>>>>> broadcast_1872_piece0 to disk
>>>>>
>>>>>
>>>>
>>>>
>>
>

Mime
View raw message