spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Johnson, Dale" <daljohn...@ebay.com>
Subject Re: Can't access file in spark, but can in hadoop
Date Fri, 27 Mar 2015 19:13:26 GMT
Yes, I could recompile the hdfs client with more logging, but I don’t have the day or two
to spare right this week.

One more thing about this, the cluster is Horton Works 2.1.3 [.0]

They seem to have a claim of supporting spark on Horton Works 2.2

Dale.

From: Ted Yu <yuzhihong@gmail.com<mailto:yuzhihong@gmail.com>>
Date: Thursday, March 26, 2015 at 4:54 PM
To: "Johnson, Dale" <daljohnson@ebay.com<mailto:daljohnson@ebay.com>>
Cc: user <user@spark.apache.org<mailto:user@spark.apache.org>>
Subject: Re: Can't access file in spark, but can in hadoop

Looks like the following assertion failed:
      Preconditions.checkState(storageIDsCount == locs.size());

locs is List<DatanodeInfoProto>
Can you enhance the assertion to log more information ?

Cheers

On Thu, Mar 26, 2015 at 3:06 PM, Dale Johnson <daljohnson@ebay.com<mailto:daljohnson@ebay.com>>
wrote:
There seems to be a special kind of "corrupted according to Spark" state of
file in HDFS.  I have isolated a set of files (maybe 1% of all files I need
to work with) which are producing the following stack dump when I try to
sc.textFile() open them.  When I try to open directories, most large
directories contain at least one file of this type.  Curiously, the
following two lines fail inside of a Spark job, but not inside of a Scoobi
job:

    val conf = new org.apache.hadoop.conf.Configuration
    val fs = org.apache.hadoop.fs.FileSystem.get(conf)

The stack trace follows:

15/03/26 14:22:43 INFO yarn.ApplicationMaster: Final app status: FAILED,
exitCode: 15, (reason: User class threw exception: null)
Exception in thread "Driver" java.lang.IllegalStateException
        at
org.spark-project.guava.common.base.Preconditions.checkState(Preconditions.java:133)
        at org.apache.hadoop.hdfs.protocolPB.PBHelper.convert(PBHelper.java:673)
        at
org.apache.hadoop.hdfs.protocolPB.PBHelper.convertLocatedBlock(PBHelper.java:1100)
        at org.apache.hadoop.hdfs.protocolPB.PBHelper.convert(PBHelper.java:1118)
        at org.apache.hadoop.hdfs.protocolPB.PBHelper.convert(PBHelper.java:1251)
        at org.apache.hadoop.hdfs.protocolPB.PBHelper.convert(PBHelper.java:1354)
        at org.apache.hadoop.hdfs.protocolPB.PBHelper.convert(PBHelper.java:1363)
        at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:518)
        at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1743)
        at
org.apache.hadoop.hdfs.DistributedFileSystem$15.<init>(DistributedFileSystem.java:738)
        at
org.apache.hadoop.hdfs.DistributedFileSystem.listLocatedStatus(DistributedFileSystem.java:727)
        at org.apache.hadoop.fs.FileSystem.listLocatedStatus(FileSystem.java:1662)
        at org.apache.hadoop.fs.FileSystem$5.<init>(FileSystem.java:1724)
        at org.apache.hadoop.fs.FileSystem.listFiles(FileSystem.java:1721)
        at
com.ebay.ss.niffler.miner.speller.SpellQueryLaunch$$anonfun$main$2.apply(SpellQuery.scala:1125)
        at
com.ebay.ss.niffler.miner.speller.SpellQueryLaunch$$anonfun$main$2.apply(SpellQuery.scala:1123)
        at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
        at
com.ebay.ss.niffler.miner.speller.SpellQueryLaunch$.main(SpellQuery.scala:1123)
        at
com.ebay.ss.niffler.miner.speller.SpellQueryLaunch.main(SpellQuery.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at
org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:427)
15/03/26 14:22:43 INFO yarn.ApplicationMaster: Invoking sc stop from
shutdown hook

It appears to have found the three copies of the given HDFS block, but is
performing some sort of validation with them before giving them back to
spark to schedule the job.  But there is an assert failing.

I've tried this with 1.2.0, 1.2.1 and 1.3.0, and I get the exact same error,
but I've seen the line numbers change on the HDFS libraries, but not the
function names.  I've tried recompiling myself with different hadoop
versions, and it's the same.  We're running hadoop 2.4.1 on our cluster.

A google search turns up absolutely nothing on this.

Any insight at all would be appreciated.

Dale Johnson
Applied Researcher
eBay.com




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Can-t-access-file-in-spark-but-can-in-hadoop-tp22251.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org<mailto:user-unsubscribe@spark.apache.org>
For additional commands, e-mail: user-help@spark.apache.org<mailto:user-help@spark.apache.org>



Mime
View raw message