spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prithish <prith...@gmail.com>
Subject Re: RDD blocks on Spark Driver
Date Wed, 01 Mar 2017 05:35:16 GMT
This is the command I am running:

spark-submit --deploy-mode cluster --master yarn --class com.myorg.myApp
s3://my-bucket/myapp-0.1.jar

On Wed, Mar 1, 2017 at 12:22 AM, Jonathan Kelly <jonathakamzn@gmail.com>
wrote:

> Prithish,
>
> It would be helpful for you to share the spark-submit command you are
> running.
>
> ~ Jonathan
>
> On Sun, Feb 26, 2017 at 8:29 AM Prithish <prithish@gmail.com> wrote:
>
>> Thanks for the responses, I am running this on Amazon EMR which runs the
>> Yarn cluster manager.
>>
>> On Sat, Feb 25, 2017 at 4:45 PM, liangyhgood@gmail.com <
>> liangyhgood@gmail.com> wrote:
>>
>> Hi,
>>  I think you are using the local model of Spark. There
>> are mainly four models, which are local, standalone,  yarn
>> and Mesos. Also, "blocks" is relative to hdfs, "partitions"
>>  is relative to spark.
>>
>> liangyihuai
>>
>> ---Original---
>> *From:* "Jacek Laskowski "<jacek@japila.pl>
>> *Date:* 2017/2/25 02:45:20
>> *To:* "prithish"<prithish@gmail.com>;
>> *Cc:* "user"<user@spark.apache.org>;
>> *Subject:* Re: RDD blocks on Spark Driver
>>
>> Hi,
>>
>> Guess you're use local mode which has only one executor called driver. Is
>> my guessing correct?
>>
>> Jacek
>>
>> On 23 Feb 2017 2:03 a.m., <prithish@gmail.com> wrote:
>>
>> Hello,
>>
>> Had a question. When I look at the executors tab in Spark UI, I notice
>> that some RDD blocks are assigned to the driver as well. Can someone please
>> tell me why?
>>
>> Thanks for the help.
>>
>>
>>

Mime
View raw message