spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <luohui20...@sina.com>
Subject 回复:Re: run spark apps in linux crontab
Date Thu, 21 Jul 2016 06:48:29 GMT

got it.
difference:> : all messages goes to the log file, leaving no messages in STDOUTtee: all
message goes to the log file and STDOUT at the same time.

--------------------------------

 

Thanks&amp;Best regards!
San.Luo

----- 原始邮件 -----
发件人:Chanh Le <giaosudau@gmail.com>
收件人:luohui20001@sina.com
抄送人:focus <focusheart@qq.com>, user <user@spark.apache.org>
主题:Re: run spark apps in linux crontab
日期:2016年07月21日 11点38分

you should you use command.sh | tee file.log
On Jul 21, 2016, at 10:36 AM, <luohui20001@sina.com> <luohui20001@sina.com> wrote:

thank you focus, and all.this problem solved by adding a line ". /etc/profile" in my shell.

--------------------------------

 

Thanks&amp;Best regards!
San.Luo

----- 原始邮件 -----
发件人:"focus" <focusheart@qq.com>
收件人:"luohui20001" <luohui20001@sina.com>, "user@spark.apache.org" <user@spark.apache.org>
主题:Re:run spark apps in linux crontab
日期:2016年07月20日 18点11分

Hi, I just meet this problem, too! The reason is crontab runtime doesn't have the variables
you defined, such as $SPARK_HOME.I defined the $SPARK_HOME and other variables in /etc/profile
like this:
export $MYSCRIPTS=/opt/myscriptsexport $SPARK_HOME=/opt/spark
then, in my crontab job script daily_job.sh
#!/bin/sh
. /etc/profile
$SPARK_HOME/bin/spark-submit $MYSCRIPTS/fix_fh_yesterday.py
then, in crontab -e
0 8 * * * /home/user/daily_job.sh
hope this helps~



------------------ Original ------------------From: "luohui20001"<luohui20001@sina.com>;
Date: 2016年7月20日(星期三) 晚上6:00To: "user@spark.apache.org"<user@spark.apache.org>;
Subject: run spark apps in linux crontab
hi guys:      I add a spark-submit job into my Linux crontab list by the means below ,however
none of them works. If I change it to a normal shell script, it is ok. I don't quite understand
why. I checked the 8080 web ui of my spark cluster, no job submitted, and there is not messages
in /home/hadoop/log.       Any idea is welcome.
[hadoop@master ~]$ crontab -e1.22 21 * * * sh /home/hadoop/shellscripts/run4.sh > /home/hadoop/log

and in run4.sh,it wrote:$SPARK_HOME/bin/spark-submit --class com.abc.myclass --total-executor-cores
10 --jars $SPARK_HOME/lib/MyDep.jar $SPARK_HOME/MyJar.jar  > /home/hadoop/log 
2.
22 21 * * * $SPARK_HOME/bin/spark-submit --class com.abc.myclass --total-executor-cores 10
--jars $SPARK_HOME/lib/MyDep.jar $SPARK_HOME/MyJar.jar  > /home/hadoop/log 
3.22 21 * * * /usr/lib/spark/bin/spark-submit --class com.abc.myclass --total-executor-cores
10 --jars /usr/lib/spark/lib/MyDep.jar /usr/lib/spark/MyJar.jar  > /home/hadoop/log 
4.22 21 * * * hadoop /usr/lib/spark/bin/spark-submit --class com.abc.myclass --total-executor-cores
10 --jars /usr/lib/spark/lib/MyDep.jar /usr/lib/spark/MyJar.jar  > /home/hadoop/log 
--------------------------------

 

Thanks&amp;Best regards!
San.Luo


Mime
View raw message