spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <em...@yeikel.com>
Subject Is it possible to rate limit an UDP?
Date Tue, 08 Jan 2019 23:21:27 GMT
I have a data frame for which I apply an UDF that calls a REST web service.
This web service is distributed in only a few nodes and it won't be able to
handle a massive load from Spark. 

 

Is it possible to rate limit this UDP? For example , something like 100
op/s. 

 

If not , what are the options? Is splitting the df an option? 

 

I've read a similar question in Stack overflow [1] and the solution suggests
Spark Streaming , but my application does not involve streaming. Do I need
to turn the operations into a streaming workflow to achieve something like
that? 

 

Current Workflow : Hive -> Spark ->  Service

 

Thank you

 

[1]
https://stackoverflow.com/questions/43953882/how-to-rate-limit-a-spark-map-o
peration


Mime
View raw message