spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jungtaek Lim <kabhwan.opensou...@gmail.com>
Subject Re: Spark hangs while reading from jdbc - does nothing Removing Guess work from trouble shooting
Date Thu, 16 Apr 2020 07:36:42 GMT
Do thread dump continuously, per specific period (like 1s) and see the
change of stack / lock for each thread. (This is not easy to be done in UI
so maybe doing manually would be the only option. Not sure Spark UI will
provide the same, haven't used at all.)

It will tell which thread is being blocked (even it's shown as running) and
which point to look at.

On Thu, Apr 16, 2020 at 4:29 PM Ruijing Li <liruijing09@gmail.com> wrote:

> Once I do. thread dump, what should I be looking for to tell where it is
> hanging? Seeing a lot of timed_waiting and waiting on driver. Driver is
> also being blocked by spark UI. If there are no tasks, is there a point to
> do thread dump of executors?
>
> On Tue, Apr 14, 2020 at 4:49 AM Gabor Somogyi <gabor.g.somogyi@gmail.com>
> wrote:
>
>> The simplest way is to do thread dump which doesn't require any fancy
>> tool (it's available on Spark UI).
>> Without thread dump it's hard to say anything...
>>
>>
>> On Tue, Apr 14, 2020 at 11:32 AM jane thorpe <janethorpe1@aol.com.invalid>
>> wrote:
>>
>>> Here a is another tool I use Logic Analyser  7:55
>>> https://youtu.be/LnzuMJLZRdU
>>>
>>> you could take some suggestions for improving performance  queries.
>>> https://dzone.com/articles/why-you-should-not-use-select-in-sql-query-1
>>>
>>>
>>> Jane thorpe
>>> janethorpe1@aol.com
>>>
>>>
>>> -----Original Message-----
>>> From: jane thorpe <janethorpe1@aol.com.INVALID>
>>> To: janethorpe1 <janethorpe1@aol.com>; mich.talebzadeh <
>>> mich.talebzadeh@gmail.com>; liruijing09 <liruijing09@gmail.com>; user
<
>>> user@spark.apache.org>
>>> Sent: Mon, 13 Apr 2020 8:32
>>> Subject: Re: Spark hangs while reading from jdbc - does nothing Removing
>>> Guess work from trouble shooting
>>>
>>>
>>>
>>> This tool may be useful for you to trouble shoot your problems away.
>>>
>>>
>>> https://www.javacodegeeks.com/2020/04/simplifying-apm-remove-the-guesswork-from-troubleshooting.html
>>>
>>>
>>> "APM tools typically use a waterfall-type view to show the blocking
>>> time of different components cascading through the control flow within an
>>> application.
>>> These types of visualizations are useful, and AppOptics has them, but
>>> they can be difficult to understand for those of us without a PhD."
>>>
>>> Especially  helpful if you want to understand through visualisation and
>>> you do not have a phD.
>>>
>>>
>>> Jane thorpe
>>> janethorpe1@aol.com
>>>
>>>
>>> -----Original Message-----
>>> From: jane thorpe <janethorpe1@aol.com.INVALID>
>>> To: mich.talebzadeh <mich.talebzadeh@gmail.com>; liruijing09 <
>>> liruijing09@gmail.com>; user <user@spark.apache.org>
>>> CC: user <user@spark.apache.org>
>>> Sent: Sun, 12 Apr 2020 4:35
>>> Subject: Re: Spark hangs while reading from jdbc - does nothing
>>>
>>> You seem to be implying the error is intermittent.
>>> You seem to be implying data is being ingested  via JDBC. So the
>>> connection has proven itself to be working unless no data is arriving from
>>> the  JDBC channel at all.  If no data is arriving then one could say it
>>> could be  the JDBC.
>>> If the error is intermittent  then it is likely a resource involved in
>>> processing is filling to capacity.
>>> Try reducing the data ingestion volume and see if that completes, then
>>> increase the data ingested  incrementally.
>>> I assume you have  run the job on small amount of data so you have
>>> completed your prototype stage successfully.
>>>
>>> ------------------------------
>>> On Saturday, 11 April 2020 Mich Talebzadeh <mich.talebzadeh@gmail.com>
>>> wrote:
>>> Hi,
>>>
>>> Have you checked your JDBC connections from Spark to Oracle. What is
>>> Oracle saying? Is it doing anything or hanging?
>>>
>>> set pagesize 9999
>>> set linesize 140
>>> set heading off
>>> select SUBSTR(name,1,8) || ' sessions as on '||TO_CHAR(CURRENT_DATE,
>>> 'MON DD YYYY HH:MI AM') from v$database;
>>> set heading on
>>> column spid heading "OS PID" format a6
>>> column process format a13 heading "Client ProcID"
>>> column username  format a15
>>> column sid       format 999
>>> column serial#   format 99999
>>> column STATUS    format a3 HEADING 'ACT'
>>> column last      format 9,999.99
>>> column TotGets   format 999,999,999,999 HEADING 'Logical I/O'
>>> column phyRds    format 999,999,999 HEADING 'Physical I/O'
>>> column total_memory format 999,999,999 HEADING 'MEM/KB'
>>> --
>>> SELECT
>>>           substr(a.username,1,15) "LOGIN"
>>>         , substr(a.sid,1,5) || ','||substr(a.serial#,1,5) AS
>>> "SID/serial#"
>>>         , TO_CHAR(a.logon_time, 'DD/MM HH:MI') "LOGGED IN SINCE"
>>>         , substr(a.machine,1,10) HOST
>>>         , substr(p.username,1,8)||'/'||substr(p.spid,1,5) "OS PID"
>>>         , substr(a.osuser,1,8)||'/'||substr(a.process,1,5) "Client PID"
>>>         , substr(a.program,1,15) PROGRAM
>>>         --,ROUND((CURRENT_DATE-a.logon_time)*24) AS "Logged/Hours"
>>>         , (
>>>                 select round(sum(ss.value)/1024) from v$sesstat ss,
>>> v$statname sn
>>>                 where ss.sid = a.sid and
>>>                         sn.statistic# = ss.statistic# and
>>>                         -- sn.name in ('session pga memory')
>>>                         sn.name in ('session pga memory','session uga
>>> memory')
>>>           ) AS total_memory
>>>         , (b.block_gets + b.consistent_gets) TotGets
>>>         , b.physical_reads phyRds
>>>         , decode(a.status, 'ACTIVE', 'Y','INACTIVE', 'N') STATUS
>>>         , CASE WHEN a.sid in (select sid from v$mystat where rownum = 1)
>>> THEN '<-- YOU' ELSE ' ' END "INFO"
>>> FROM
>>>          v$process p
>>>         ,v$session a
>>>         ,v$sess_io b
>>> WHERE
>>> a.paddr = p.addr
>>> AND p.background IS NULL
>>> --AND  a.sid NOT IN (select sid from v$mystat where rownum = 1)
>>> AND a.sid = b.sid
>>> AND a.username is not null
>>> --AND (a.last_call_et < 3600 or a.status = 'ACTIVE')
>>> --AND CURRENT_DATE - logon_time > 0
>>> --AND a.sid NOT IN ( select sid from v$mystat where rownum=1)  --
>>> exclude me
>>> --AND (b.block_gets + b.consistent_gets) > 0
>>> ORDER BY a.username;
>>> exit
>>>
>>> HTH
>>>
>>> Dr Mich Talebzadeh
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>> On Fri, 10 Apr 2020 at 17:37, Ruijing Li <liruijing09@gmail.com> wrote:
>>>
>>> Hi all,
>>>
>>> I am on spark 2.4.4 and using scala 2.11.12, and running cluster mode on
>>> mesos. I am ingesting from an oracle database using spark.read.jdbc. I am
>>> seeing a strange issue where spark just hangs and does nothing, not
>>> starting any new tasks. Normally this job finishes in 30 stages but
>>> sometimes it stops at 29 completed stages and doesn’t start the last stage.
>>> The spark job is idling and there is no pending or active task. What could
>>> be the problem? Thanks.
>>> --
>>> Cheers,
>>> Ruijing Li
>>>
>>> --
> Cheers,
> Ruijing Li
>

Mime
View raw message