spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tiandiwoxin1234 <tiandiwo...@icloud.com>
Subject Re: Problem using limit clause in spark sql
Date Sat, 26 Dec 2015 08:46:30 GMT
As for 'rdd.zipwithIndex.partitionBy(YourCustomPartitioner)', can I just drop some records
using my custom partitioner, otherwise I still have to call rdd.take() to get exactly 10000
records.

And repartition is THE expensive operation that I want to walk around.

Actually, what I expect the limit clause would do is using some kind of coordinator to assign
each partition a number of records to reserve and the sum of which is exactly the limit(or
). But it seems this cannot be easily done.  

> 在 2015年12月25日,下午11:10,manasdebashiskar [via Apache Spark User List] <ml-node+s1001560n25797h10@n3.nabble.com>
写道:
> 
> It can be easily done using an RDD. 
> 
> rdd.zipwithIndex.partitionBy(YourCustomPartitioner) should give you your items. 
> Here YourCustomPartitioner will know how to pick sample items from each partition. 
> 
> If you want to stick to Dataframe you can always repartition the data after you apply
the limit. 
> 
> ..Manas 
> 
> If you reply to this email, your message will be added to the discussion below:
> http://apache-spark-user-list.1001560.n3.nabble.com/Problem-using-limit-clause-in-spark-sql-tp25789p25797.html
<http://apache-spark-user-list.1001560.n3.nabble.com/Problem-using-limit-clause-in-spark-sql-tp25789p25797.html>
> To unsubscribe from Problem using limit clause in spark sql, click here <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=25789&code=dGlhbmRpd294aW5AaWNsb3VkLmNvbXwyNTc4OXwtOTkyODc3MDI5>.
> NAML <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Problem-using-limit-clause-in-spark-sql-tp25789p25798.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Mime
View raw message