spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xilang Yan <>
Subject Limit the block size of data received by spring streaming receiver
Date Mon, 15 Jan 2018 10:17:44 GMT

We use a customize receiver to receive data from our MQ. We used to use def 
store(dataItem: T) to store data however I found the block size can be very 
different from 0.5K to 5M size. So that data partition processing time is 
very different. Shuffle is an option, but I want to avoid it. 

I notice that def store(dataBuffer: ArrayBuffer[T]) can store the whole data 
into a block so I can control block size, however I also noticed that this 
method doesn't apply any rate limit on it, I have to do rate limit myself. 

So by now, I haven't have a good way to control block size, I am asking if 
spark can add rate limit on  store(dataBuffer: ArrayBuffer[T]) method or 
have a way to control block size generated by BlockGenerator 

Sent from:

To unsubscribe e-mail:

View raw message