spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <>
Subject Is it possible to rate limit an UDP?
Date Tue, 08 Jan 2019 23:21:27 GMT
I have a data frame for which I apply an UDF that calls a REST web service.
This web service is distributed in only a few nodes and it won't be able to
handle a massive load from Spark. 


Is it possible to rate limit this UDP? For example , something like 100


If not , what are the options? Is splitting the df an option? 


I've read a similar question in Stack overflow [1] and the solution suggests
Spark Streaming , but my application does not involve streaming. Do I need
to turn the operations into a streaming workflow to achieve something like


Current Workflow : Hive -> Spark ->  Service


Thank you



View raw message