commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Herbert <alex.d.herb...@gmail.com>
Subject Re: [statistics][descriptive] Classes or static methods for common descriptive statistics?
Date Wed, 29 May 2019 13:58:56 GMT

On 29/05/2019 12:50, Gilles Sadowski wrote:
> Hello.
>
> Le mar. 28 mai 2019 à 20:36, Alex Herbert <alex.d.herbert@gmail.com> a écrit
:
>>
>>
>>> On 28 May 2019, at 18:09, Eric Barnhill <ericbarnhill@gmail.com> wrote:
>>>
>>> The previous commons-math interface for descriptive statistics used a
>>> paradigm of constructing classes for various statistical functions and
>>> calling evaluate(). Example
>>>
>>> Mean mean = new Mean();
>>> double mn = mean.evaluate(double[])
>>>
>>> I wrote this type of code all through grad school and always found it
>>> unnecessarily bulky.  To me these summary statistics are classic use cases
>>> for static methods:
>>>
>>> double mean .= Mean.evaluate(double[])
>>>
>>> I don't have any particular problem with the evaluate() syntax.
>>>
>>> I looked over the old Math 4 API to see if there were any benefits to the
>>> previous class-oriented approach that we might not want to lose. But I
>>> don't think there were, the functionality outside of evaluate() is minimal.
>> A quick check shows that evaluate comes from UnivariateStatistic. This has some more
methods that add little to an instance view of the computation:
>>
>> double evaluate(double[] values) throws MathIllegalArgumentException;
>> double evaluate(double[] values, int begin, int length) throws MathIllegalArgumentException;
>> UnivariateStatistic copy();
>>
>> However it is extended by StorelessUnivariateStatistic which adds methods to update
the statistic:
>>
>> void increment(double d);
>> void incrementAll(double[] values) throws MathIllegalArgumentException;
>> void incrementAll(double[] values, int start, int length) throws MathIllegalArgumentException;
>> double getResult();
>> long getN();
>> void clear();
>> StorelessUnivariateStatistic copy();
>>
>> This type of functionality would be lost by static methods.
>>
>> If you are moving to a functional interface type pattern for each statistic then
you will lose the other functionality possible with an instance state, namely updating with
more values or combining instances.
>>
>> So this is a question of whether updating a statistic is required after the first
computation.
>>
>> Will there be an alternative in the library for a map-reduce type operation using
instances that can be combined using Stream.collect:
>>
>>      <R> R collect(Supplier<R> supplier,
>>                    ObjDoubleConsumer<R> accumulator,
>>                    BiConsumer<R, R> combiner);
>>
>> Here <R> would be Mean:
>>
>> double mean = Arrays.stream(new double[1000]).collect(Mean::new, Mean::add, Mean::add).getMean()
with:
>>
>> void add(double);
>> void add(Mean);
>> double getMean();
>>
>> (Untested code)
>>
>>> Finally we should consider whether we really need a separate class for each
>>> statistic at all. Do we want to call:
>>>
>>> Mean.evaluate()
>>>
>>> or
>>>
>>> SummaryStats.mean()
>>>
>>> or maybe
>>>
>>> Stats.mean() ?
>>>
>>> The last being nice and compact.
>>>
>>> Let's make a decision so our esteemed mentee Virendra knows in what
>>> direction to take his work this summer. :)
> I'm not sure I understand the implicit conclusions of this conversation
> and the other one there:
>      https://markmail.org/message/7dmyhzuy6lublyb5
>
> Do we agree that the core issue is *not* how to compute a mean, or a
> median, or a fourth moment, but how any and all of those can be
> computed seamlessly through a functional API (stream)?
>
> As Alex pointed out, a useful functionality is the ability to "combine"
> instances, e.g. if data are collected by several threads.
> A potential use-case is the retrieval of the current value of (any)
> statistical quantities while the data continues to be collected.
>
> An initial idea would be:
> public interface StatQuantity {
>      public double value(double[]); // For "basic" usage.
>      public double value(DoubleStream); // For "advanced" usage.
> }
>
> public class StatCollection {
>      /** Specify which quantities this collection will hold/compute. */
>      public StatCollection(Map<String, StatQuantity> stats) { /*... */ }
>
>      /**
>       * Start a worker thread.
>       * @param data Values for which the stat quantities must be computed.
>       */
>      public void startCollector(DoubleStream data) { /* ... */ }
>
>      /** Combine current state of workers. */
>      public void collect() { /* ... */ }
>
>      /** @return the current (combined) value of a named quantity. */
>      public double get(String name) { /* ... */ }
>
>      private StatCollector implements Callable {
>          StatCollector(DoubleStream data) { /* ... */ }
>      }
> }
>
> This is all totally untested, very partial, and probably wrong-headed but
> I thought that we were looking at this kind of refactoring.
>
> Regards,
> Gilles

I don't think you can pass in a Stream to be worked on. The Stream API 
requires that you pass something into the stream and the stream contents 
are changed (intermediate operation) or consumed (terminating 
operation). Only when a terminating operation is invoked is the stream 
pipeline activated.

So the new classes have to be useable in intermediate and terminating 
operations.

If the idea of the refactoring was to move all the old API to a new API 
that can be used with streams then each Statistic should be based on 
ideas presented in:

java.util.DoubleSummaryStatistics
java.util.IntSummaryStatistics
java.util.LongSummaryStatistics

Each of which implement respectively:

DoubleConsumer
IntConsumer
LongConsumer

Plus:

- a method for combining with themselves
- an empty constructor (to act as a Supplier of the object)

So this would require:

public interface DoubleStatQuantity extends DoubleConsumer {
   // inherits:
   // public void accept(double value)
   public void combine(DoubleStatQuantity other);
   public double value();
   public DoubleStatQuantity newInstance();
}

Note that the combine method would have to check the input is of the 
correct type. This can be fixed using Self-types with Java [1]:

public interface DoubleStatQuantity<B extends DoubleStatQuantity<B>> extends DoubleConsumer
{
   public void combine(B other);
   public double value();
}

public class Max implements DoubleStatQuantity<Max> {
   private double max = Double.NEGATIVE_INFINITY;

   @Override
   public void accept(double value) {
     max = Math.max(max, value);
   }

   @Override
   public void combine(Max other) {
     max = Math.max(max, other.max);
   }

   @Override
   public double value() {
     return max;
   }

   @Override
   public Max newInstance() {
     return new Max();
   }
}

It is a matter of opinion on whether this is readable. It is probably 
why the JDK implementations offer the functionality but do not declare 
it in a generic way.

The StatCollection would then be:

public class StatCollection implements DoubleConsumer {
   private Map<String, DoubleStatQuantity<?>> stats;

   /** Specify which quantities this collection will hold/compute. */
   public StatCollection(Map<String, DoubleStatQuantity<?>> stats) {
     this.stats = stats;
   }

   @Override
   public void accept(double value) {
     stats.values().forEach(stat -> stat.accept(value));
   }

   /** @return the current value of a named quantity. */
   public double get(String name) {
     return stats.get(name).value();
   }
}

A more performant implementation is required (based on a list and stats 
by index) but this is the idea.

Note that with the generic <?> it is not easily possible for 
StatCollection to implement DoubleStatQuantity<StatCollection> as the 
combine method ends up having to combine stats of type 
DoubleStatQuantity<?>. I've not tried but expect a runtime exception if 
the classes are different when combined.

I prefer the route of adding a set of classes that implement the current 
algorithms, support DoubleConsumer and have a combine method. These 
requirements could be specified by an interface:

public interface DoubleStatQuantity extends DoubleConsumer {
   // inherits:
   // public void accept(double value)
   public void combine(DoubleStatQuantity other);
   public double value();
}

The interface behaviour for the combine method if the other cannot be 
combined should do one of: throw; or ignore the input.

Note that the interface it is not strictly necessary other than to 
support a generic combined statistics collection. However this may be 
better served with a dedicated class to compute combined statistics (as 
per the JDK classes) in order to eliminate duplication in the algorithm.

Keeping the idea simple and based on the current JDK implementations 
would allow the port to begin. It can always be made more explicit with 
interfaces to specify operations later.


[1] https://www.sitepoint.com/self-types-with-javas-generics/

>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message