spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shyla deshpande <deshpandesh...@gmail.com>
Subject Re: Spark streaming giving me a bunch of WARNINGS, please help meunderstand them
Date Mon, 10 Jul 2017 15:26:54 GMT
WARN  Use an existing SparkContext, some configuration may not take effect.
         I wanted to restart the spark streaming app, so stopped the
running and issued a new spark submit. Why and how it will use a existing
         SparkContext?
=> you are using checkpoint to restore the sparkcontext.
=> No, I am not using checkpoint for recovery. I need the checkpoint
because I am doing stateful streaming.

WARN  Spark is not running in local mode, therefore the checkpoint
directory must not be on the local filesystem. Directory
'file:/efs/checkpoint' appears to be on the local filesystem.
=>the CP path should be HDFSand so on. If you want to use local path, the
cluster model should be local.
=> I am using the AWS EFS mount for checkpoint because I am running in
standalone mode.

WARN  overriding enable.auto.commit to false for executor
=>stop the executor to commit the offset auto
=>No, I don't want the autocommit. I do the commit later after my output
operation using the commitAsync API

WARN  overriding auto.offset.reset to none for executor
=>it set the index where the executor read msg
=> I set this to none, because I want it to continue from where it left off

WARN  overriding executor group.id to spark-executor-mygroupid
=> set the groupid of consumer. If you do not set, it will set a default
and give a warning.
=> I am setting the groupid, mygroupid and it is adding spark-executor- as
a prefix...


On Mon, Jul 10, 2017 at 12:39 AM, 萝卜丝炒饭 <1427357147@qq.com> wrote:

> It seems you are usibg kafka 0.10.
> See my comments below.
>
> ---Original---
> *From:* "shyla deshpande"<deshpandeshyla@gmail.com>
> *Date:* 2017/7/10 08:17:10
> *To:* "user"<user@spark.apache.org>;
> *Subject:* Spark streaming giving me a bunch of WARNINGS, please help
> meunderstand them
>
> WARN  Use an existing SparkContext, some configuration may not take effect.
>          I wanted to restart the spark streaming app, so stopped the
> running and issued a new spark submit. Why and how it will use a existing
>          SparkContext?
> => you are using checkpoint to restore the sparkcontext.
> WARN  Spark is not running in local mode, therefore the checkpoint
> directory must not be on the local filesystem. Directory
> 'file:/efs/checkpoint' appears to be on the local filesystem.
> =>the CP path should be HDFSand so on.
> If you want to use local path, the cluster model should be local.
>
>
> WARN  overriding enable.auto.commit to false for executor
> =>stop the executor to commit the offset auto.
>
> WARN  overriding auto.offset.reset to none for executor
> =>it set the index where the executor read msg
> WARN  overriding executor group.id to spark-executor-mygroupid
> => set the groupid of consumer. If you do not set, it will set a default
> and give a warning.
>
> WARN  overriding receive.buffer.bytes to 65536 see KAFKA-3135
> WARN  overriding enable.auto.commit to false for executor
> WARN  overriding auto.offset.reset to none for executor
>

Mime
View raw message