spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hyukjin Kwon <>
Subject Re: no logging in pyspark code?
Date Thu, 06 Sep 2018 05:04:48 GMT
FYI, we do have a basic logging by warnings module.

2018년 8월 28일 (화) 오전 2:05, Imran Rashid <>님이

> ah, great, thanks!  sorry I missed that, I'll watch that jira.
> On Mon, Aug 27, 2018 at 12:41 PM Ilan Filonenko <> wrote:
>> A JIRA has been opened up on this exact topic: SPARK-25236
>> <>, a few days ago,
>> after seeing another case of print(_, file=sys.stderr) in a most recent
>> review. I agree that we should include logging for PySpark workers.
>> On Mon, Aug 27, 2018 at 1:29 PM, Imran Rashid <
>>> wrote:
>>> Another question on pyspark code -- how come there is no logging at
>>> all?  does python logging have an unreasonable overhead, or its impossible
>>> to configure or something?
>>> I'm really surprised nobody has ever wanted to me able to turn on some
>>> debug or trace logging in pyspark by just configuring a logging level.
>>> For me, I wanted this during debugging while developing -- I'd work on
>>> some part of the code and drop in a bunch of print statements.  Then I'd
>>> rip those out when I think I'm ready to submit a patch.  But then I realize
>>> I forgot some case, then more debugging -- oh gotta add those print
>>> statements in again ...
>>> does somebody jsut need to setup the configuration properly, or is there
>>> a bigger reason to avoid logging in python?
>>> thanks,
>>> Imran

View raw message