spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denny Lee <denny.g....@gmail.com>
Subject Re: Starting sparkthrift server
Date Tue, 24 Mar 2015 00:41:18 GMT
In that case, can you use the configurations to specify the folders?  I'm
wondering if this actually Hive in play here and somehow the
/tmp/spark-events is being specified for the logs for Hive?

On Mon, Mar 23, 2015 at 2:00 PM Anubhav Agarwal <anubhav33@gmail.com> wrote:

> When I start spark-shell (for example) it does not write to the
> /tmp/spark-events folder. It remains empty. I have even tried it after
> giving that folder rwx permission for user, group and others.
>
> Neil's colleague,
> Anu
>
> On Mon, Mar 23, 2015 at 4:50 PM, Denny Lee <denny.g.lee@gmail.com> wrote:
>
>> When you say the job has access, do you mean that when you run
>> spark-submit
>> or spark-shell (for example), it is able to write to the /tmp/spark-events
>> folder?
>>
>>
>> On Mon, Mar 23, 2015 at 1:02 PM Neil Dev <neilkdev@gmail.com> wrote:
>>
>> > we are running this right now as root user and the folder
>> /tmp/spark-events
>> > was manually created and the Job has access to this folder
>> >
>> > On Mon, Mar 23, 2015 at 3:38 PM, Denny Lee <denny.g.lee@gmail.com>
>> wrote:
>> >
>> >> It appears that you are running the thrift-server using the
>> spark-events
>> >> account but the /tmp/spark-events folder doesn't exist or the user
>> running
>> >> thrift-server does not have access to it.  Have you been able to run
>> Hive
>> >> using the spark-events user so that way the /tmp/spark-events folder
>> has
>> >> been created.  If you need to reassign the scratch dir / log dir to
>> another
>> >> folder (instead of /tmp/spark-events), you could use  --hiveconf to
>> assign
>> >> those to another folder.
>> >>
>> >>
>> >> On Mon, Mar 23, 2015 at 8:39 AM Neil Dev <neilkdev@gmail.com> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> I am having issues starting spark-thriftserver. I'm running spark
>> 1.3.o
>> >>> with Hadoop 2.4.0. I would like to be able to change its port too so,
>> I
>> >>> can
>> >>> hive hive-thriftserver as well as spark-thriftserver running at the
>> same
>> >>> time.
>> >>>
>> >>> Starting sparkthrift server:-
>> >>> sudo ./start-thriftserver.sh --master spark://ip-172-31-10-124:7077
>> >>> --executor-memory 2G
>> >>>
>> >>> Error:-
>> >>> I created the folder manually but still getting the following
>> error----
>> >>> Exception in thread "main" java.lang.IllegalArgumentException: Log
>> >>> directory /tmp/spark-events does not exist.
>> >>>
>> >>>
>> >>> I am getting the following error
>> >>> 15/03/23 15:07:02 ERROR thrift.ThriftCLIService: Error:
>> >>> org.apache.thrift.transport.TTransportException: Could not create
>> >>> ServerSocket on address0.0.0.0/0.0.0.0:10000.
>> >>>         at
>> >>>
>> org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:93)
>> >>>         at
>> >>>
>> org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:79)
>> >>>         at
>> >>> org.apache.hive.service.auth.HiveAuthFactory.getServerSocket(
>> >>> HiveAuthFactory.java:236)
>> >>>         at
>> >>> org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.
>> >>> run(ThriftBinaryCLIService.java:69)
>> >>>         at java.lang.Thread.run(Thread.java:745)
>> >>>
>> >>> Thanks
>> >>> Neil
>> >>>
>> >>
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message