spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eike von Seggern <eike.segg...@sevenval.com>
Subject Re: Show function name in Logs for PythonUDFRunner
Date Thu, 22 Nov 2018 12:24:03 GMT
Hi,

Abdeali Kothari <abdealikothari@gmail.com> schrieb am Do., 22. Nov. 2018 um
10:04 Uhr:

> When I run Python UDFs with pyspark, I get multiple logs where it says:
>
> 18/11/22 01:51:59 INFO python.PythonUDFRunner: Times: total = 44, boot = -25, init =
67, finish = 2
>
>
> I am wondering if in these logs I can identify easily which of my
> PythonUDFs this timing information is for (I have about a hundred) so it's
> quite difficult for em to identify this easily ...
>

If the log is created using Python's logging module, it should be possible.
This supports `funcName` (
https://docs.python.org/3.6/library/logging.html#logrecord-attributes). But
I do not know how to configure the log-format for pyspark.

 HTH

Eike

Mime
View raw message