beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From François Wagner (JIRA) <>
Subject [jira] [Created] (BEAM-2429) Conflicting filesystems with used of HadoopFileSystem
Date Fri, 09 Jun 2017 09:11:18 GMT
François Wagner created BEAM-2429:

             Summary: Conflicting filesystems with used of HadoopFileSystem
                 Key: BEAM-2429
             Project: Beam
          Issue Type: Bug
          Components: sdk-java-extensions
    Affects Versions: 2.0.0
            Reporter: François Wagner
            Assignee: Davor Bonaci

I'm facing issue when trying to use HadoopFileSystem in my pipeline. It looks like HadoopFileSystem
is registring itself under the `file` schema (,
hence the following Exception is thrown when trying to register HadoopFileSystem.

java.lang.IllegalStateException: Scheme: [file] has conflicting filesystems: [,]

What is the correct way to handle `hdfs` url out of the box with TextIO & AvroIO ?

    String[] args = new String[]{
        "--hdfsConfiguration=[{\"dfs.client.use.datanode.hostname\": \"true\"}]"};
    HadoopFileSystemOptions options = PipelineOptionsFactory
    Pipeline pipeline = Pipeline.create(options);
    Pipeline pipeline = Pipeline.create(options); 

This message was sent by Atlassian JIRA

View raw message