spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrei Badea (JIRA)" <>
Subject [jira] [Created] (SPARK-23641) Wrong username when making relative path to Hive LOAD DATA absolute
Date Fri, 09 Mar 2018 11:17:00 GMT
Andrei Badea created SPARK-23641:

             Summary: Wrong username when making relative path to Hive LOAD DATA absolute
                 Key: SPARK-23641
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.1.0
            Reporter: Andrei Badea

We have an application deployed in yarn-cluster mode.

At some point, the application invokes
spark.sql("LOAD DATA INPATH some/relative/path ...")
in an attempt to add that directory to a table. The relative path should be interpreted relatively
to the home directory of the user who ran the Spark application (this is what the Hive shell

The command runs without failing, but the directory is not added to the table. Investigation
showed that {{org.apache.spark.sql.execution.command.LoadDataCommand}} attempts to make the
path absolute by prepending {{s"/user/${System.getProperty("")}"}}. Since the application
was deployed in yarn-cluster mode, the value of the {{}} property is "yarn". This
is illustrated by the following message in the driver logs:
INFO metadata.Hive: No sources specified to move: hdfs://.../user/yarn/some/relative/path{noformat}
Interestingly, the same Spark application writes the data to the relative path (prior to calling
LOAD DATA), and that makes the path absolute as expected. It uses {{Path.makeQualified()}}.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message