spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lohith Samaga M <Lohith.Sam...@mphasis.com>
Subject RE: Need to user univariate summary stats
Date Thu, 04 Feb 2016 09:42:29 GMT
Hi Arun,
                You can do df.agg(max(,,), min(..)).


Best regards / Mit freundlichen Grüßen / Sincères salutations
M. Lohith Samaga

From: Arunkumar Pillai [mailto:arunkumar1111@gmail.com]
Sent: Thursday, February 04, 2016 14.53
To: user@spark.apache.org
Subject: Need to user univariate summary stats

Hi

I'm currently using query

sqlContext.sql("SELECT MAX(variablesArray) FROM " + tableName)

to extract mean max min.
is there any better  optimized way ?


In the example i saw df.groupBy("key").agg(skewness("a"), kurtosis("a"))

But i don't have key anywhere in the data.

How to extract the univariate summary stats from df. please help

--
Thanks and Regards
        Arun
Information transmitted by this e-mail is proprietary to Mphasis, its associated companies
and/ or its customers and is intended 
for use only by the individual or entity to which it is addressed, and may contain information
that is privileged, confidential or 
exempt from disclosure under applicable law. If you are not the intended recipient or it appears
that this mail has been forwarded 
to you without proper authority, you are notified that any use or dissemination of this information
in any manner is strictly 
prohibited. In such cases, please notify us immediately at mailmaster@mphasis.com and delete
this mail from your records.
Mime
View raw message