spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kohki Nishio <tarop...@gmail.com>
Subject Re: Lock issue with SQLConf.getConf
Date Sat, 11 Sep 2021 15:15:01 GMT
Awesome, thanks!

On Sat, Sep 11, 2021 at 6:34 AM Sean Owen <srowen@gmail.com> wrote:

> Looks like this was improved in
> https://issues.apache.org/jira/browse/SPARK-35701 for 3.2.0
>
> On Fri, Sep 10, 2021 at 10:21 PM Kohki Nishio <taroplus@gmail.com> wrote:
>
>> Hello,
>> I'm running spark in local mode and seeing multiple threads showing like
>> below, anybody knows why it's not using a concurrent hash map  ?
>>
>> ---
>> "request-handler-dispatcher-19785" #107532 prio=5 os_prio=0
>> tid=0x00007fbd78036000 nid=0x4ebf runnable [0x00007fc6e83af000]
>>    java.lang.Thread.State: RUNNABLE
>> at java.util.Collections$SynchronizedMap.get(Collections.java:2586)
>> - locked <0x00007fc901c7d9f8> (a java.util.Collections$SynchronizedMap)
>> at org.apache.spark.sql.internal.SQLConf.getConf(SQLConf.scala:3750)
>> at
>> org.apache.spark.sql.internal.SQLConf.constraintPropagationEnabled(SQLConf.scala:3346)
>>
>> "Executor task launch worker for task 23.0 in stage 279783.0 (TID
>> 449746)" #107929 daemon prio=5 os_prio=0 tid=0x00007fbe2c005000 nid=0x55f6
>> waiting for monitor entry [0x00007fc6e4037000]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>> at java.util.Collections$SynchronizedMap.get(Collections.java:2586)
>> - waiting to lock <0x00007fc901c7d9f8> (a
>> java.util.Collections$SynchronizedMap)
>> at org.apache.spark.sql.internal.SQLConf.getConf(SQLConf.scala:3750)
>> at
>> org.apache.spark.sql.internal.SQLConf.methodSplitThreshold(SQLConf.scala:3330)
>>
>> "Executor task launch worker for task 16.0 in stage 279883.0 (TID
>> 449728)" #107690 daemon prio=5 os_prio=0 tid=0x00007fbc60056800 nid=0x53c6
>> waiting for monitor entry [0x00007fc65a3a8000]
>>    java.lang.Thread.State: BLOCKED (on object monitor)
>> at java.util.Collections$SynchronizedMap.get(Collections.java:2586)
>> - waiting to lock <0x00007fc901c7d9f8> (a
>> java.util.Collections$SynchronizedMap)
>> at org.apache.spark.sql.internal.SQLConf.getConf(SQLConf.scala:3750)
>> at
>> org.apache.spark.sql.internal.SQLConf.planChangeLogLevel(SQLConf.scala:3160)
>> at
>> org.apache.spark.sql.catalyst.rules.PlanChangeLogger.<init>(RuleExecutor.scala:49)
>>
>> ---
>>
>>
>> --
>> Kohki Nishio
>>
>

-- 
Kohki Nishio

Mime
View raw message