spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: wholeTextFiles like for binary files ?
Date Thu, 26 Jun 2014 06:58:40 GMT
You cannot read image files with wholeTextFiles because it uses
CombineFileInputFormat which cannot read gripped files because they are not
splittable <http://www.bigdataspeak.com/2013_01_01_archive.html> (source
proving it):

  override def createRecordReader(
      split: InputSplit,
      context: TaskAttemptContext): RecordReader[String, String] = {

    new CombineFileRecordReader[String, String](
      split.asInstanceOf[CombineFileSplit],
      context,
      classOf[WholeTextFileRecordReader])
  }

You may be able to use newAPIHadoopFile with wholefileinputformat
<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>
(not
built into hadoop but all over the internet) to get this to work correctly.
I don't think WholeFileInputFormat will work since it just gets the bytes
of the file, meaning you may have to write your own class possibly
extending WholeFileInputFormat.

Thanks
Best Regards


On Thu, Jun 26, 2014 at 3:31 AM, Jaonary Rabarisoa <jaonary@gmail.com>
wrote:

> Is there an equivalent of wholeTextFiles for binary files for example a
> set of images ?
>
> Cheers,
>
> Jaonary
>

Mime
View raw message