spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niranjan Reddy <niranja...@gmail.com>
Subject Re: Spark on Yarn: java.lang.IllegalArgumentException: Invalid rule
Date Tue, 27 Jan 2015 15:21:13 GMT
Thanks, Ted. Kerberos is enabled on the cluster.

I'm new to the world of kerberos, so pease excuse my ignorance here. Do you
know if there are any additional steps I need to take in addition to
setting HADOOP_CONF_DIR? For instance, does hadoop.security.auth_to_local
require any specific setting (the current setting for this property was set
by the admin)? This cluster has Spark 1.0 installed and I can use it
without any errors.
On Jan 26, 2015 11:38 PM, "Ted Yu" <yuzhihong@gmail.com> wrote:

> Looks like Kerberos was enabled for your cluster.
>
> Can you check the config files under HADOOP_CONF_DIR ?
>
> Cheers
>
> On Mon, Jan 26, 2015 at 8:17 PM, maven <niranjanry@gmail.com> wrote:
>
>> All,
>>
>> I recently try to build Spark-1.2 on my enterprise server (which has
>> Hadoop
>> 2.3 with YARN). Here're the steps I followed for the build:
>>
>> $ mvn -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -DskipTests clean package
>> $ export SPARK_HOME=/path/to/spark/folder
>> $ export HADOOP_CONF_DIR=/etc/hadoop/conf
>>
>> However, when I try to work with this installation either locally or on
>> YARN, I get the following error:
>>
>> Exception in thread "main" java.lang.ExceptionInInitializerError
>>         at
>> org.apache.spark.util.Utils$.getSparkOrYarnConfig(Utils.scala:1784)
>>         at
>> org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:105)
>>         at
>> org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:180)
>>         at org.apache.spark.SparkEnv$.create(SparkEnv.scala:292)
>>         at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:159)
>>         at org.apache.spark.SparkContext.<init>(SparkContext.scala:232)
>>         at water.MyDriver$.main(MyDriver.scala:19)
>>         at water.MyDriver.main(MyDriver.scala)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>         at
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>         at java.lang.reflect.Method.invoke(Method.java:606)
>>         at
>> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:360)
>>         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
>>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>> Caused by: org.apache.spark.SparkException: Unable to load YARN support
>>         at
>>
>> org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:199)
>>         at
>> org.apache.spark.deploy.SparkHadoopUtil$.<init>(SparkHadoopUtil.scala:194)
>>         at
>> org.apache.spark.deploy.SparkHadoopUtil$.<clinit>(SparkHadoopUtil.scala)
>>         ... 15 more
>> Caused by: java.lang.IllegalArgumentException: Invalid rule: L
>> RULE:[2:$1@$0](.*@XXXCOMPANY.COM)s/(.*)@XXXCOMPANY.COM/$1/L
>> DEFAULT
>>         at
>>
>> org.apache.hadoop.security.authentication.util.KerberosName.parseRules(KerberosName.java:321)
>>         at
>>
>> org.apache.hadoop.security.authentication.util.KerberosName.setRules(KerberosName.java:386)
>>         at
>>
>> org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:75)
>>         at
>>
>> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:247)
>>         at
>>
>> org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:283)
>>         at
>> org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:43)
>>         at
>>
>> org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.<init>(YarnSparkHadoopUtil.scala:45)
>>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>> Method)
>>         at
>>
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>>         at
>>
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>         at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>>         at java.lang.Class.newInstance(Class.java:374)
>>         at
>>
>> org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:196)
>>         ... 17 more
>>
>> I noticed that when I unset HADOOP_CONF_DIR, I'm able to work in the local
>> mode without any errors. I'm able to work with pre-installed Spark 1.0,
>> locally and on yarn, without any issues. It looks like I may be missing a
>> configuration step somewhere. Any thoughts on what may be causing this?
>>
>> NR
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Yarn-java-lang-IllegalArgumentException-Invalid-rule-tp21382.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>

Mime
View raw message