spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Darabos <daniel.dara...@lynxanalytics.com>
Subject Re: Better line number hints for logging?
Date Thu, 05 Jun 2014 12:47:07 GMT
On Wed, Jun 4, 2014 at 10:39 PM, Matei Zaharia <matei.zaharia@gmail.com>
wrote:

> That’s a good idea too, maybe we can change CallSiteInfo to do that.
>

I've filed an issue: https://issues.apache.org/jira/browse/SPARK-2035

Matei
>
> On Jun 4, 2014, at 8:44 AM, Daniel Darabos <
> daniel.darabos@lynxanalytics.com> wrote:
>
> Oh, this would be super useful for us too!
>
> Actually wouldn't it be best if you could see the whole call stack on the
> UI, rather than just one line? (Of course you would have to click to expand
> it.)
>
>
> On Wed, Jun 4, 2014 at 2:38 AM, John Salvatier <jsalvatier@gmail.com>
> wrote:
>
>> Ok, I will probably open a Jira.
>>
>>
>> On Tue, Jun 3, 2014 at 5:29 PM, Matei Zaharia <matei.zaharia@gmail.com>
>> wrote:
>>
>>> You can use RDD.setName to give it a name. There’s also a creationSite
>>> field that is private[spark] — we may want to add a public setter for that
>>> later. If the name isn’t enough and you’d like this, please open a JIRA
>>> issue for it.
>>>
>>> Matei
>>>
>>> On Jun 3, 2014, at 5:22 PM, John Salvatier <jsalvatier@gmail.com> wrote:
>>>
>>> I have created some extension methods for RDDs in RichRecordRDD and
>>> these are working exceptionally well for me.
>>>
>>> However, when looking at the logs, its impossible to tell what's going
>>> on because all the line number hints point to RichRecordRDD.scala rather
>>> than the code that uses it. For example:
>>>
>>>> INFO scheduler.DAGScheduler: Submitting Stage 122 (MappedRDD[1223] at
>>>> map at RichRecordRDD.scala:633), which is now runnable
>>>
>>> Is there any way set up my extension methods class so that the logs will
>>> print a more useful line number?
>>>
>>>
>>>
>>
>
>

Mime
View raw message