spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anubhav Agarwal <anubha...@gmail.com>
Subject Re: Starting sparkthrift server
Date Mon, 23 Mar 2015 21:00:19 GMT
When I start spark-shell (for example) it does not write to the
/tmp/spark-events folder. It remains empty. I have even tried it after
giving that folder rwx permission for user, group and others.

Neil's colleague,
Anu

On Mon, Mar 23, 2015 at 4:50 PM, Denny Lee <denny.g.lee@gmail.com> wrote:

> When you say the job has access, do you mean that when you run spark-submit
> or spark-shell (for example), it is able to write to the /tmp/spark-events
> folder?
>
>
> On Mon, Mar 23, 2015 at 1:02 PM Neil Dev <neilkdev@gmail.com> wrote:
>
> > we are running this right now as root user and the folder
> /tmp/spark-events
> > was manually created and the Job has access to this folder
> >
> > On Mon, Mar 23, 2015 at 3:38 PM, Denny Lee <denny.g.lee@gmail.com>
> wrote:
> >
> >> It appears that you are running the thrift-server using the spark-events
> >> account but the /tmp/spark-events folder doesn't exist or the user
> running
> >> thrift-server does not have access to it.  Have you been able to run
> Hive
> >> using the spark-events user so that way the /tmp/spark-events folder has
> >> been created.  If you need to reassign the scratch dir / log dir to
> another
> >> folder (instead of /tmp/spark-events), you could use  --hiveconf to
> assign
> >> those to another folder.
> >>
> >>
> >> On Mon, Mar 23, 2015 at 8:39 AM Neil Dev <neilkdev@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> I am having issues starting spark-thriftserver. I'm running spark 1.3.o
> >>> with Hadoop 2.4.0. I would like to be able to change its port too so, I
> >>> can
> >>> hive hive-thriftserver as well as spark-thriftserver running at the
> same
> >>> time.
> >>>
> >>> Starting sparkthrift server:-
> >>> sudo ./start-thriftserver.sh --master spark://ip-172-31-10-124:7077
> >>> --executor-memory 2G
> >>>
> >>> Error:-
> >>> I created the folder manually but still getting the following error----
> >>> Exception in thread "main" java.lang.IllegalArgumentException: Log
> >>> directory /tmp/spark-events does not exist.
> >>>
> >>>
> >>> I am getting the following error
> >>> 15/03/23 15:07:02 ERROR thrift.ThriftCLIService: Error:
> >>> org.apache.thrift.transport.TTransportException: Could not create
> >>> ServerSocket on address0.0.0.0/0.0.0.0:10000.
> >>>         at
> >>> org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:93)
> >>>         at
> >>> org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:79)
> >>>         at
> >>> org.apache.hive.service.auth.HiveAuthFactory.getServerSocket(
> >>> HiveAuthFactory.java:236)
> >>>         at
> >>> org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.
> >>> run(ThriftBinaryCLIService.java:69)
> >>>         at java.lang.Thread.run(Thread.java:745)
> >>>
> >>> Thanks
> >>> Neil
> >>>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message