spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashish Soni <asoni.le...@gmail.com>
Subject Re: Spark-Submit / Spark-Shell Error Standalone cluster
Date Mon, 29 Jun 2015 12:57:24 GMT
yes ./ is doing something but it has to do something with in the
spark-submit and spark-shell file also as if i define the SPARK_HOME
explicitly it works ( see below )  from any where  , not sure why the
original code which ships with spark is not able to resolve the correct
path see below



[root@SparkDev bin]# *more spark-submit*

SPARK_HOME="$(cd "`dirname "$0"`"/..; pwd)"  - *This is not able to resolve
the correct path *
#SPARK_HOME=/usr/hdp/2.2.6.0-2800/spark - *When i define the path it works*
# disable randomized hash for string in Python 3.3+
export PYTHONHASHSEED=0

# Only define a usage function if an upstream script hasn't done so.
if ! type -t usage >/dev/null 2>&1; then
  usage() {
    if [ -n "$1" ]; then
      echo "$1"
    fi
    "$SPARK_HOME"/bin/spark-class org.apache.spark.deploy.SparkSubmit --help
    exit "$2"
  }
  export -f usage
fi

exec "$SPARK_HOME"/bin/spark-class org.apache.spark.deploy.SparkSubmit "$@"


On Mon, Jun 29, 2015 at 3:02 AM, Tomas Hudik <xhudik@gmail.com> wrote:

> Hi Ashish,
> i see. Well, I'd guess it has something to do with path. Is you specify an
> absolute path to your spark-shell - it will work (it seems to me "./"
> notation in linux is doing it for you to some extent ). In other words, I'd
> try to provide the absolute path and if working stick to it (the safest
> option).
>
> rgds, Tomas
>
> On Sun, Jun 28, 2015 at 4:40 PM, Ashish Soni <asoni.learn@gmail.com>
> wrote:
>
>> my spark home is as below
>> /usr/hdp/2.2.6.0-2800/spark
>> and when i go inside bin and execute the command in different way it
>> works see below i.e ./spark-shell works but *spark-shell* does not work
>> ...
>>
>>
>> [image: Inline image 1]
>>
>>
>> [image: Inline image 2]
>>
>> On Sun, Jun 28, 2015 at 8:53 AM, Tomas Hudik <xhudik@gmail.com> wrote:
>>
>>> /usr/bin/ - looks like strange directory. Did you copy some files to
>>> /usr/bin yourself?
>>> If you download (possible compile) spark - it will never be placed into
>>> /usr/bin
>>>
>>> On Sun, Jun 28, 2015 at 9:19 AM, Wojciech Pituła <w.pitula@gmail.com>
>>> wrote:
>>>
>>>> I assume that /usr/bin/load-spark-env.sh exists. Have you got the
>>>> rights to execute it?
>>>>
>>>> niedz., 28.06.2015 o 04:53 użytkownik Ashish Soni <
>>>> asoni.learn@gmail.com> napisał:
>>>>
>>>>> Not sure what is the issue but when i run the spark-submit or
>>>>> spark-shell i am getting below error
>>>>>
>>>>> /usr/bin/spark-class: line 24: /usr/bin/load-spark-env.sh: No such
>>>>> file or directory
>>>>>
>>>>> Can some one please help
>>>>>
>>>>> Thanks,
>>>>>
>>>>
>>>
>>
>

Mime
View raw message