spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Can't access file in spark, but can in hadoop
Date Sat, 28 Mar 2015 13:29:30 GMT
Thanks for the follow-up, Dale.

bq. hdp 2.3.1

Minor correction: should be hdp 2.1.3

Cheers

On Sat, Mar 28, 2015 at 2:28 AM, Johnson, Dale <daljohnson@ebay.com> wrote:

>  Actually I did figure this out eventually.
>
>  I’m running on a Hortonworks cluster hdp 2.3.1 (hadoop 2.4.1).  Spark
> bundles the org/apache/hadoop/hdfs/… classes along with the spark-assembly
> jar.  This turns out to introduce a small incompatibility with hdp 2.3.1.
> I carved these classes out of the jar, and put a distro-provided jar into
> the class path for the hdfs classes, and this fixed the problem.
>
>  Ideally there would be an exclusion in the pom to deal with this.
>
>  Dale.
>
>   From: Zhan Zhang <zzhang@hortonworks.com>
> Date: Friday, March 27, 2015 at 4:28 PM
> To: "Johnson, Dale" <daljohnson@ebay.com>
> Cc: Ted Yu <yuzhihong@gmail.com>, user <user@spark.apache.org>
>
> Subject: Re: Can't access file in spark, but can in hadoop
>
>   Probably guava version conflicts issue. What spark version did you use,
> and which hadoop version it compile against?
>
>  Thanks.
>
>  Zhan Zhang
>
>  On Mar 27, 2015, at 12:13 PM, Johnson, Dale <daljohnson@ebay.com> wrote:
>
>  Yes, I could recompile the hdfs client with more logging, but I don’t
> have the day or two to spare right this week.
>
>  One more thing about this, the cluster is Horton Works 2.1.3 [.0]
>
>  They seem to have a claim of supporting spark on Horton Works 2.2
>
>  Dale.
>
>   From: Ted Yu <yuzhihong@gmail.com>
> Date: Thursday, March 26, 2015 at 4:54 PM
> To: "Johnson, Dale" <daljohnson@ebay.com>
> Cc: user <user@spark.apache.org>
> Subject: Re: Can't access file in spark, but can in hadoop
>
>   Looks like the following assertion failed:
>       Preconditions.checkState(storageIDsCount == locs.size());
>
>  locs is List<DatanodeInfoProto>
> Can you enhance the assertion to log more information ?
>
>  Cheers
>
> On Thu, Mar 26, 2015 at 3:06 PM, Dale Johnson <daljohnson@ebay.com> wrote:
>
>> There seems to be a special kind of "corrupted according to Spark" state
>> of
>> file in HDFS.  I have isolated a set of files (maybe 1% of all files I
>> need
>> to work with) which are producing the following stack dump when I try to
>> sc.textFile() open them.  When I try to open directories, most large
>> directories contain at least one file of this type.  Curiously, the
>> following two lines fail inside of a Spark job, but not inside of a Scoobi
>> job:
>>
>>     val conf = new org.apache.hadoop.conf.Configuration
>>     val fs = org.apache.hadoop.fs.FileSystem.get(conf)
>>
>> The stack trace follows:
>>
>> 15/03/26 14:22:43 INFO yarn.ApplicationMaster: Final app status: FAILED,
>> exitCode: 15, (reason: User class threw exception: null)
>> Exception in thread "Driver" java.lang.IllegalStateException
>>         at
>>
>> org.spark-project.guava.common.base.Preconditions.checkState(Preconditions.java:133)
>>         at
>> org.apache.hadoop.hdfs.protocolPB.PBHelper.convert(PBHelper.java:673)
>>         at
>>
>> org.apache.hadoop.hdfs.protocolPB.PBHelper.convertLocatedBlock(PBHelper.java:1100)
>>         at
>> org.apache.hadoop.hdfs.protocolPB.PBHelper.convert(PBHelper.java:1118)
>>         at
>> org.apache.hadoop.hdfs.protocolPB.PBHelper.convert(PBHelper.java:1251)
>>         at
>> org.apache.hadoop.hdfs.protocolPB.PBHelper.convert(PBHelper.java:1354)
>>         at
>> org.apache.hadoop.hdfs.protocolPB.PBHelper.convert(PBHelper.java:1363)
>>         at
>>
>> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:518)
>>         at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1743)
>>         at
>>
>> org.apache.hadoop.hdfs.DistributedFileSystem$15.<init>(DistributedFileSystem.java:738)
>>         at
>>
>> org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:727)
>>         at
>> org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1662)
>>         at org.apache.hadoop.fs.FileSystem$5.<init>(FileSystem.java:1724)
>>         at org.apache.hadoop.fs.FileSystem.listFiles(FileSystem.java:1721)
>>         at
>>
>> com.ebay.ss.niffler.miner.speller.SpellQueryLaunch$$anonfun$main$2.apply(SpellQuery.scala:1125)
>>         at
>>
>> com.ebay.ss.niffler.miner.speller.SpellQueryLaunch$$anonfun$main$2.apply(SpellQuery.scala:1123)
>>         at
>>
>> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
>>         at
>> scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
>>         at
>>
>> com.ebay.ss.niffler.miner.speller.SpellQueryLaunch$.main(SpellQuery.scala:1123)
>>         at
>> com.ebay.ss.niffler.miner.speller.SpellQueryLaunch.main(SpellQuery.scala)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>         at
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>         at java.lang.reflect.Method.invoke(Method.java:606)
>>         at
>>
>> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:427)
>> 15/03/26 14:22:43 INFO yarn.ApplicationMaster: Invoking sc stop from
>> shutdown hook
>>
>> It appears to have found the three copies of the given HDFS block, but is
>> performing some sort of validation with them before giving them back to
>> spark to schedule the job.  But there is an assert failing.
>>
>> I've tried this with 1.2.0, 1.2.1 and 1.3.0, and I get the exact same
>> error,
>> but I've seen the line numbers change on the HDFS libraries, but not the
>> function names.  I've tried recompiling myself with different hadoop
>> versions, and it's the same.  We're running hadoop 2.4.1 on our cluster.
>>
>> A google search turns up absolutely nothing on this.
>>
>> Any insight at all would be appreciated.
>>
>> Dale Johnson
>> Applied Researcher
>> eBay.com
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Can-t-access-file-in-spark-but-can-in-hadoop-tp22251.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
>

Mime
View raw message