spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reynold Xin <>
Subject Re: Monitoring system extensibility
Date Mon, 10 Oct 2016 08:00:51 GMT
I just took a quick look and set a target version on the JIRA. But Pete I
think the primary problem with the JIRA and pull request is that it really
just argues (or implements) opening up a private API, which is a valid
point but there are a lot more that needs to be done before making some
private API public.

At the very least, we need to answer the following:

1. Is the existing API maintainable? E.g. Is it OK to just expose coda hale
metrics in the API? Do we need to worry about dependency conflicts? Should
we wrap it?

2. Is the existing API sufficiently general (to cover use cases)? What
about security related setup?

On Fri, Oct 7, 2016 at 2:03 AM, Pete Robbins <> wrote:

> Which has happened. The last comment being in August with someone saying
> it was important to them. They PR has been around since March and despite a
> request to be reviewed has not got any committer's attention. Without that,
> it is going nowhere. The historic Jiras requesting other sinks such as
> Kafka, OpenTSBD etc have also been ignored.
> So for now we continue creating classes in o.a.s package.
> On Fri, 7 Oct 2016 at 09:50 Reynold Xin <> wrote:
>> So to be constructive and in order to actually open up these APIs, it
>> would be useful for users to comment on the JIRA ticket on their use cases
>> (rather than "I want this to be public"), and then we can design an API
>> that would address those use cases. In some cases the solution is to just
>> make the existing internal API public. But turning some internal API public
>> without thinking about whether those APIs are sufficiently expressive and
>> maintainable is not a great way to design APIs in general.
>> On Friday, October 7, 2016, Pete Robbins <> wrote:
>>> I brought this up last year and there was a Jira raised:
>>> For now I just have my SInk and Source in an o.a.s package name which is
>>> not ideal but the only way round this.
>>> On Fri, 7 Oct 2016 at 08:30 Reynold Xin <> wrote:
>>>> They have always been private, haven't they?
>>>> src/main/scala/org/apache/spark/metrics/source/Source.scala
>>>> On Thu, Oct 6, 2016 at 7:38 AM, Alexander Oleynikov <
>>>>> wrote:
>>>>> Hi.
>>>>> As of v2.0.1, the traits `org.apache.spark.metrics.source.Source` and
>>>>> `org.apache.spark.metrics.sink.Sink` are defined as private to
>>>>> ‘spark’ package, so it becomes troublesome to create a new implementation
>>>>> in the user’s code (but still possible in a hacky way).
>>>>> This seems to be the only missing piece to allow extension of the
>>>>> metrics system and I wonder whether is was conscious design decision
>>>>> limit the visibility. Is it possible to broaden the visibility scope
>>>>> these traits in the future versions?
>>>>> Thanks,
>>>>> Alexander
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail:

View raw message