spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yan Fang <>
Subject How do you debug with the logs ?
Date Wed, 03 Sep 2014 21:05:59 GMT
Hi guys,

curious how you deal with the logs. I feel difficulty in debugging with the
logs: run spark-streaming in our yarn cluster using client-mode. So have
two logs: yarn log and local log ( for client ). Whenever I have problem,
 the log is too big to read with "gedit" and "grep". (e.g. after running 10
hours, the local log is 1GB ). Do you use any tools to analyze/monitor/read
the logs? such as logstash?


Fang, Yan
+1 (206) 849-4108

View raw message