spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From unk1102 <>
Subject Does DataFrame has something like set hive.groupby.skewindata=true;
Date Sat, 21 May 2016 20:48:08 GMT
Hi I am having DataFrame with huge skew data in terms of TB and I am doing
groupby on 8 fields which I cant avoid unfortunately. I am looking to
optimize this I have found hive has

set hive.groupby.skewindata=true;

I dont use Hive I have Spark DataFrame can we achieve above Spark? Please
guide. Thanks in advance.

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message