mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Harrington <ch...@heystaks.com>
Subject ClusterDumper writing to local instead of HDFS
Date Tue, 12 Mar 2013 12:37:47 GMT
Hi all,

The subject line says it all, ClusterDumper is writing to local file system instead of HDFS.

After looking at the source

From the ClusterDumper class

if (this.outputFile == null) {
      shouldClose = false;
      writer = new OutputStreamWriter(System.out);
    } else {
      shouldClose = true;
      if (outputFile.getName().startsWith("s3n://")) {
        Path p = outputPath;
        FileSystem fs = FileSystem.get(p.toUri(), conf);
        writer = new OutputStreamWriter(fs.create(p), Charsets.UTF_8);
      } else {
        writer = Files.newWriter(this.outputFile, Charsets.UTF_8);
      }
    }


From the Files class

  public static BufferedWriter newWriter(File file, Charset charset)
      throws FileNotFoundException {
   return new BufferedWriter(
        new OutputStreamWriter(new FileOutputStream(file), charset));
  }


So a few questions on the above.

1. Am I correct in saying if  the outputFile starts with "s3n://" it writes to the HDFS other
wise it writes to the local FS?

2. If the above is true then what is the meaning  of a URI starting with s3n:// 

3. Is there a way to force it to write to the HDFS even if the URI doesn't start with s3n://
or am I going to have to modify ClusterDumper class myself?




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message