spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Rudenko <>
Subject Re: what is metadata in StructField ?
Date Wed, 15 Jul 2015 10:37:41 GMT
Hi Mathieu,
metadata is very usefull if you need to save some data about a column 
(e.g. count of null values, cardinality, domain, min/max/std, etc.). 
It's currently used in ml package in attributes:

Take a look how i'm using metadata to get summary statistics from h2o:

Let me know if you'll have questions.

Peter Rudenko

On 2015-07-15 12:48, matd wrote:
> I see in StructField that we can provide metadata.
> What is it meant for ?  How is it used by Spark later on ?
> Are there any rules on what we can/cannot do with it ?
> I'm building some DataFrame processing, and I need to maintain a set of
> (meta)data along with the DF. I was wondering if I can use
> StructField.metadata for this use, or if I should build my own structure.
> Mathieu
> --
> View this message in context:
> Sent from the Apache Spark User List mailing list archive at
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message