spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From conner <>
Subject How to avoid long-running jobs blocking short-running jobs
Date Sat, 03 Nov 2018 09:04:01 GMT

I use spark cluster to run ETL jobs and analysis computation about the data
after elt stage.
The elt jobs can keep running for several hours, but analysis computation is
a short-running job which can finish in a few seconds.
The dilemma I entrapped is that my application runs in a single JVM and
can't be a cluster application, so just one spark context in my application
currently. But when the elt jobs are running,
the jobs will occupy all resource including worker executors too long to
block all my analysis computation jobs. 

My solution is to find a good way to divide the spark cluster resource into
two. One part for analysis computation jobs, another for
elt jobs. if the part for elt jobs is free, I can allocate analysis
computation jobs to it.
So I want to find a middleware that can support two spark context and it
must be embedded in my application. I do some research on the third party
project spark job server. It can divide spark resource by launching another
JVM to run spark context with a specific resource.
these operations are invisible to the upper layer, so it's a good solution
for me. But this project is running in a single JVM  and just support REST
API, I can't endure the data transfer by TCP again
which too slow to me. I want to get a result from spark cluster by TCP and
give this result to view layer to show.
Can anybody give me some good suggestion? I shall be so grateful.

Sent from:

To unsubscribe e-mail:

View raw message