spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Rovner <alex.rov...@magnetic.com>
Subject Re: How to optimize group by query fired using hiveContext.sql?
Date Mon, 05 Oct 2015 02:30:09 GMT
Can you at least copy paste the error(s) you are seeing when the job fails?
Without the error message(s), it's hard to even suggest anything.

*Alex Rovner*
*Director, Data Engineering *
*o:* 646.759.0052

* <http://www.magnetic.com/>*

On Sat, Oct 3, 2015 at 9:50 AM, Umesh Kacha <umesh.kacha@gmail.com> wrote:

> Hi thanks I cant share yarn logs because of privacy in my company but I
> can tell you I have seen yarn logs there I have not found anything except
> YARN killing container because it is exceeds physical memory capacity.
>
> I am using the following command line script Above job launches around
> 1500 ExecutorService threads from a driver with a thread pool of 15 so at a
> time 15 jobs will be running as showing in UI.
>
> ./spark-submit --class com.xyz.abc.MySparkJob
>
> --conf "spark.executor.extraJavaOptions=-XX:MaxPermSize=512M" -
>
> -driver-java-options -XX:MaxPermSize=512m -
>
> -driver-memory 4g --master yarn-client
>
> --executor-memory 27G --executor-cores 2
>
> --num-executors 40
>
> --jars /path/to/others-jars
>
> /path/to/spark-job.jar
>
>
> On Sat, Oct 3, 2015 at 7:11 PM, Alex Rovner <alex.rovner@magnetic.com>
> wrote:
>
>> Can you send over your yarn logs along with the command you are using to
>> submit your job?
>>
>> *Alex Rovner*
>> *Director, Data Engineering *
>> *o:* 646.759.0052
>>
>> * <http://www.magnetic.com/>*
>>
>> On Sat, Oct 3, 2015 at 9:07 AM, Umesh Kacha <umesh.kacha@gmail.com>
>> wrote:
>>
>>> Hi Alex thanks much for the reply. Please read the following for more
>>> details about my problem.
>>>
>>>
>>> http://stackoverflow.com/questions/32317285/spark-executor-oom-issue-on-yarn
>>>
>>> My each container has 8 core and 30 GB max memory. So I am using
>>> yarn-client mode using 40 executors with 27GB/2 cores. If I use more cores
>>> then my job start loosing more executors. I tried to set
>>> spark.yarn.executor.memoryOverhead around 2 GB even 8 GB but it does
>>> not help I loose executors no matter what. The reason is my jobs shuffles
>>> lots of data even 20 GB of data in every job in UI I have seen it. Shuffle
>>> happens because of group by and I cant avoid it in my case.
>>>
>>>
>>>
>>> On Sat, Oct 3, 2015 at 6:27 PM, Alex Rovner <alex.rovner@magnetic.com>
>>> wrote:
>>>
>>>> This sounds like you need to increase YARN overhead settings with the "spark.yarn.executor.memoryOverhead"
>>>> parameter. See http://spark.apache.org/docs/latest/running-on-yarn.html
>>>> for more information on the setting.
>>>>
>>>> If that does not work for you, please provide the error messages and
>>>> the command line you are using to submit your jobs for further
>>>> troubleshooting.
>>>>
>>>>
>>>> *Alex Rovner*
>>>> *Director, Data Engineering *
>>>> *o:* 646.759.0052
>>>>
>>>> * <http://www.magnetic.com/>*
>>>>
>>>> On Sat, Oct 3, 2015 at 6:19 AM, unk1102 <umesh.kacha@gmail.com> wrote:
>>>>
>>>>> Hi I have couple of Spark jobs which uses group by query which is
>>>>> getting
>>>>> fired from hiveContext.sql() Now I know group by is evil but my use
>>>>> case I
>>>>> cant avoid group by I have around 7-8 fields on which I need to do
>>>>> group by.
>>>>> Also I am using df1.except(df2) which also seems heavy operation and
>>>>> does
>>>>> lots of shuffling please see my UI snap
>>>>> <
>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/file/n24914/IMG_20151003_151830218.jpg
>>>>> >
>>>>>
>>>>> I have tried almost all optimisation including Spark 1.5 but nothing
>>>>> seems
>>>>> to be working and my job fails hangs because of executor will reach
>>>>> physical
>>>>> memory limit and YARN will kill it. I have around 1TB of data to
>>>>> process and
>>>>> it is skewed. Please guide.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-optimize-group-by-query-fired-using-hiveContext-sql-tp24914.html
>>>>> Sent from the Apache Spark User List mailing list archive at
>>>>> Nabble.com.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message