spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mich Talebzadeh <mich.talebza...@gmail.com>
Subject Re: Executors assigned to STS and number of workers in Stand Alone Mode
Date Tue, 26 Jul 2016 17:58:58 GMT
All my mistake. Sometimes one forgets simple things :)

The reason why it could not see Hive databases was that there was no
soft-link $SPARK_HOME/conf to hive-sitre.xml on this node

hive-site.xml -> /usr/lib/hive/conf/hive-site.xml

It worked fine after creating the soft-link


HTH





Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 26 July 2016 at 11:53, Mich Talebzadeh <mich.talebzadeh@gmail.com> wrote:

> Hi guys.
>
> I can start STS on local on the master node and it can see all hive
> databases. That looks good for now.
>
> On another node (meaning a slave node)
>
>
>    1. Started Hive Thrift Server HTS and that can see all the underlying
>    Hive databases. So I have two HTSs now
>    2. I started STS on this node local first. It runs OK but cannot see
>    the underlying Hive databases.
>    3. I started STS on this node and used --master spark://
>    50.140.197.217:7077 (where the master standalone is running) but still
>    cannot see Hive databases
>
> So my question how can I start STS on a non-master node such that it can
> see Hive databases? What is missing here?
>
> Thanks
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 26 July 2016 at 03:16, Chanh Le <giaosudau@gmail.com> wrote:
>
>> STS can work on Mesos as well.
>>
>>
>>
>> On Jul 26, 2016, at 6:16 AM, ayan guha <guha.ayan@gmail.com> wrote:
>>
>> STS works on YARN, as a yarn-client application.
>>
>> One issue: STS is not HA-supported, though there was some discussion to
>> make it HA similar to Hive Server. So what we did is to run sts on multiple
>> nodes and tie them to a load balancer. .
>>
>> On Tue, Jul 26, 2016 at 8:06 AM, Mich Talebzadeh <
>> mich.talebzadeh@gmail.com> wrote:
>>
>>> Correction.
>>>
>>> STS uses the same UI to display details about all processes running
>>> against it which is helpful but gets crowded
>>>
>>> :)
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>> On 25 July 2016 at 22:26, Mich Talebzadeh <mich.talebzadeh@gmail.com>
>>> wrote:
>>>
>>>> We also should remember that STS is a pretty useful tool. With JDBC you
>>>> can use beeline, Zeppelin, Squirrel and other tools against it.
>>>>
>>>> One thing I like to change is the UI port that the thrift server
>>>> listens and you can change it at startup using spark.ui.port. This is
>>>> fixed at thrift startup and can only display one sql query at a time which
>>>> is kind not useful.
>>>>
>>>> As one can run multiple clients against STS, it is a
>>>> limitation that one cannot change the UI port at runtime.
>>>>
>>>> Cheers
>>>>
>>>> Dr Mich Talebzadeh
>>>>
>>>>
>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>
>>>>
>>>> http://talebzadehmich.wordpress.com
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>> On 25 July 2016 at 22:04, Jacek Laskowski <jacek@japila.pl> wrote:
>>>>
>>>>> On Mon, Jul 25, 2016 at 10:57 PM, Mich Talebzadeh
>>>>> <mich.talebzadeh@gmail.com> wrote:
>>>>>
>>>>> > Yarn promises the best resource management I believe. Having said
>>>>> that I have not used Mesos myself.
>>>>>
>>>>> I'm glad you've mentioned it.
>>>>>
>>>>> I think Cloudera (and Hortonworks?) guys are doing a great job with
>>>>> bringing all the features of YARN to Spark and I think Spark on YARN
>>>>> shines features-wise.
>>>>>
>>>>> I'm not in a position to compare YARN vs Mesos for their resource
>>>>> management, but Spark on Mesos is certainly lagging behind Spark on
>>>>> YARN regarding the features Spark uses off the scheduler backends --
>>>>> security, data locality, queues, etc. (or I might be simply biased
>>>>> after having spent months with Spark on YARN mostly?).
>>>>>
>>>>> Jacek
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> Best Regards,
>> Ayan Guha
>>
>>
>>
>

Mime
View raw message