spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <sro...@gmail.com>
Subject Re: Lock issue with SQLConf.getConf
Date Sat, 11 Sep 2021 13:33:49 GMT
Looks like this was improved in
https://issues.apache.org/jira/browse/SPARK-35701 for 3.2.0

On Fri, Sep 10, 2021 at 10:21 PM Kohki Nishio <taroplus@gmail.com> wrote:

> Hello,
> I'm running spark in local mode and seeing multiple threads showing like
> below, anybody knows why it's not using a concurrent hash map  ?
>
> ---
> "request-handler-dispatcher-19785" #107532 prio=5 os_prio=0
> tid=0x00007fbd78036000 nid=0x4ebf runnable [0x00007fc6e83af000]
>    java.lang.Thread.State: RUNNABLE
> at java.util.Collections$SynchronizedMap.get(Collections.java:2586)
> - locked <0x00007fc901c7d9f8> (a java.util.Collections$SynchronizedMap)
> at org.apache.spark.sql.internal.SQLConf.getConf(SQLConf.scala:3750)
> at
> org.apache.spark.sql.internal.SQLConf.constraintPropagationEnabled(SQLConf.scala:3346)
>
> "Executor task launch worker for task 23.0 in stage 279783.0 (TID 449746)"
> #107929 daemon prio=5 os_prio=0 tid=0x00007fbe2c005000 nid=0x55f6 waiting
> for monitor entry [0x00007fc6e4037000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at java.util.Collections$SynchronizedMap.get(Collections.java:2586)
> - waiting to lock <0x00007fc901c7d9f8> (a
> java.util.Collections$SynchronizedMap)
> at org.apache.spark.sql.internal.SQLConf.getConf(SQLConf.scala:3750)
> at
> org.apache.spark.sql.internal.SQLConf.methodSplitThreshold(SQLConf.scala:3330)
>
> "Executor task launch worker for task 16.0 in stage 279883.0 (TID 449728)"
> #107690 daemon prio=5 os_prio=0 tid=0x00007fbc60056800 nid=0x53c6 waiting
> for monitor entry [0x00007fc65a3a8000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
> at java.util.Collections$SynchronizedMap.get(Collections.java:2586)
> - waiting to lock <0x00007fc901c7d9f8> (a
> java.util.Collections$SynchronizedMap)
> at org.apache.spark.sql.internal.SQLConf.getConf(SQLConf.scala:3750)
> at
> org.apache.spark.sql.internal.SQLConf.planChangeLogLevel(SQLConf.scala:3160)
> at
> org.apache.spark.sql.catalyst.rules.PlanChangeLogger.<init>(RuleExecutor.scala:49)
>
> ---
>
>
> --
> Kohki Nishio
>

Mime
View raw message