spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <mich...@databricks.com>
Subject Re: [SS] Any way to optimize memory consumption of SS?
Date Tue, 12 Sep 2017 18:22:43 GMT
Can you show the full query you are running?

On Tue, Sep 12, 2017 at 10:11 AM, 张万新 <kevinzwx1992@gmail.com> wrote:

> Hi,
>
> I'm using structured streaming to count unique visits of our website. I
> use spark on yarn mode with 4 executor instances and from 2 cores * 5g
> memory to 4 cores * 10g memory for each executor, but there are frequent
> full gc, and once the count raises to about more than 4.5 millions the
> application will be blocked and finally crash in OOM. It's kind of
> unreasonable. So is there any suggestion to optimize the memory consumption
> of SS? Thanks.
>

Mime
View raw message