spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From akshay naidu <akshaynaid...@gmail.com>
Subject Re: sqoop import job not working when spark thrift server is running.
Date Sat, 24 Feb 2018 12:47:11 GMT
>
> it sure is not able to get sufficient resources from YARN to start the
> containers.
>
that's right. I worked when I reduced executors from thrift but it also
reduced thrift's performance.

But it is not the solution i am looking forward to. my sqoop import job
runs just once a day, and thrift apps will running for 24/7 for
fetching-processing-displaying online reports on website. reducing
executors and keeping some in spare is helping in running more jobs other
than thrift parallely but it's wasting the core when other jobs are not
working.

is there something which can help in allocating resources dynamically?
which will automatically allocate maximum resources to thrift when there
are no other jobs running, and automatically share resources with jobs/apps
other than thrift.?

I've heard of property in yarn - dynamicAlloction , can this help?


Thanks.

On Sat, Feb 24, 2018 at 7:14 AM, vijay.bvp <bvpsarma@gmail.com> wrote:

> it sure is not able to get sufficient resources from YARN to start the
> containers.
> is it only with this import job or if you submit any other job its failing
> to start.
>
> As a test just try to run another spark job or a mapredue job  and see if
> the job can be started.
>
> Reduce the thrift server executors and see overall there is available
> cluster capacity for new jobs.
>
>
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Mime
View raw message