spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 田毅 <tia...@asiainfo.com>
Subject Re: Spark Installation
Date Wed, 09 Jul 2014 06:41:35 GMT
Hi Srikrishna

the reason to this issue is you had uploaded assembly jar to HDFS twice.

paste your command could be better diagnosis



田毅
===========================================
橘云平台产品线
大数据产品部    
亚信联创科技(中国)有限公司
手机:13910177261
电话:010-82166322
传真:010-82166617
Q Q:20057509
MSN:yi.tian@hotmail.com
地址:北京市海淀区东北旺西路10号院东区  亚信联创大厦


===========================================

在 2014年7月9日,上午3:03,Srikrishna S <srikrishna097@gmail.com> 写道:

> Hi All,
> 
> 
> I tried the make distribution script and it worked well. I was able to
> compile the spark binary on our CDH5 cluster. Once I compiled Spark, I
> copied over the binaries in the dist folder to all the other machines
> in the cluster.
> 
> However, I run into an issue while submit a job in yarn-client mode. I
> get an error message that says the following
> Resource file:/opt/spark/spark-1.0.0-bin-hadoop2/lib/spark-assembly-1.1.0-SNAPSHOT-hadoop2.3.0.jar"
> changed on src filesystem (expected 1404845211000, was 1404845404000)
> 
> My end goal is to submit a job (that uses MLLib) in our Yarn cluster.
> 
> Any thoughts anyone?
> 
> Regards,
> Krishna
> 
> 
> 
> On Tue, Jul 8, 2014 at 9:49 AM, Sandy Ryza <sandy.ryza@cloudera.com> wrote:
>> 
>> Hi Srikrishna,
>> 
>> The binaries are built with something like
>> mvn package -Pyarn -Dhadoop.version=2.3.0-cdh5.0.1 -Dyarn.version=2.3.0-cdh5.0.1
>> 
>> -Sandy
>> 
>> 
>> On Tue, Jul 8, 2014 at 3:14 AM, 田毅 <tianyi@asiainfo.com> wrote:
>>> 
>>> try this command:
>>> 
>>> make-distribution.sh --hadoop 2.3.0-cdh5.0.0 --with-yarn --with-hive
>>> 
>>> 
>>> 
>>> 
>>> 田毅
>>> ===========================================
>>> 橘云平台产品线
>>> 大数据产品部
>>> 亚信联创科技(中国)有限公司
>>> 手机:13910177261
>>> 电话:010-82166322
>>> 传真:010-82166617
>>> Q Q:20057509
>>> MSN:yi.tian@hotmail.com
>>> 地址:北京市海淀区东北旺西路10号院东区  亚信联创大厦
>>> 
>>> 
>>> ===========================================
>>> 
>>> 在 2014年7月8日,上午11:53,Krishna Sankar <ksankar42@gmail.com>
写道:
>>> 
>>> Couldn't find any reference of CDH in pom.xml - profiles or the hadoop.version.Am
also wondering how the cdh compatible artifact was compiled.
>>> Cheers
>>> <k/>
>>> 
>>> 
>>> On Mon, Jul 7, 2014 at 8:07 PM, Srikrishna S <srikrishna097@gmail.com>
wrote:
>>>> 
>>>> Hi All,
>>>> 
>>>> Does anyone know what the command line arguments to mvn are to generate the
pre-built binary for spark on Hadoop 2-CHD5.
>>>> 
>>>> I would like to pull in a recent bug fix in spark-master and rebuild the
binaries in the exact same way that was used for that provided on the website.
>>>> 
>>>> I have tried the following:
>>>> 
>>>> mvn install -Pyarn -Dhadoop.version=2.3.0-cdh5.0.1
>>>> 
>>>> And it doesn't quite work.
>>>> 
>>>> Any thoughts anyone?
>>>> 
>>> 
>>> 
>> 
> 


Mime
View raw message