sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venkat Ranganathan <vranganat...@hortonworks.com>
Subject Re: how to add default hadoop conf property value to SqoopOptions.java
Date Tue, 28 Jan 2014 23:04:00 GMT
You are trying to add a sqoop option when what you are trying to set is a
mapreduce configuration parameter.   You can set it in hadoop-site for
example


On Wed, Jan 22, 2014 at 5:50 PM, Suhas Satish <suhas.satish@gmail.com>wrote:

> Hi
> I can successfully run the following query in Sqoop -
>
> /opt/mapr/sqoop/sqoop-1.4.4/bin/sqoop import -D
> hadoop.job.history.user.location=none -connect jdbc:mysql://10.10.1.10/TestDB
> --username <http://10.10.1.10/TestDB--username> mapr --password mapr  --table t1
--hive-import
>
>
>
> But I am trying to make this property as the default value within Sqoop. Whats the best
way to do this in the code?
>
>
> hadoop.job.history.user.location=none
>
> I made the following modifications for this in SqoopOptions.java (diff attached) but
the property doesnt seem to get recognized by sqoop and I still get the stdout log msg which
says -
>
> "14/01/22 15:00:36 INFO mapred.JobClient: Creating job's user history location directory
at t1/_logs
>
> "
>
> With the property correctly set, _logs directory should not even be
> created.
>
>
>
> diff --git a/src/java/org/apache/sqoop/SqoopOptions.java
> b/src/java/org/apache/sqoop/SqoopOptions.java
> index 01805f9..4612973 100644
> --- a/src/java/org/apache/sqoop/SqoopOptions.java
> +++ b/src/java/org/apache/sqoop/SqoopOptions.java
> @@ -62,6 +62,8 @@
>    // Default hive and hcat locations.
>    public static final String DEF_HIVE_HOME = "/usr/lib/hive";
>    public static final String DEF_HCAT_HOME = "/usr/lib/hcatalog";
> +  public static final String DEF_HIST_LOGS = "none";
> +  public static final String PROP_HADOOP_JOB_HISTORY_LOGS_LOCATION =
> "hadoop.job.history.user.location";
>
>    public static final boolean METASTORE_PASSWORD_DEFAULT = false;
>
> @@ -186,6 +188,7 @@ public String toString() {
>    // class to use.
>    @StoredAsProperty("codegen.jar.file") private String existingJarFile;
>
> +  @StoredAsProperty("hadoop.job.history.user.location") private String
> hadoopJobHistLogsLoc;
>    @StoredAsProperty("mapreduce.num.mappers") private int numMappers;
>    @StoredAsProperty("enable.compression") private boolean useCompression;
>    @StoredAsProperty("compression.codec") private String compressionCodec;
> @@ -837,7 +840,12 @@ public static String getHCatHomeDefault() {
>      }
>      return hcatHome;
>    }
> -
> +  public static String getHadoopJobHistLogsLoc() {
> +    return DEF_HIST_LOGS;
> +  }
> +  private static void setHadoopJobHistLogsLoc(Configuration conf, String
> path) {
> +    conf.setStrings(PROP_HADOOP_JOB_HISTORY_LOGS_LOCATION, path);
> +  }
>    private void initDefaults(Configuration baseConfiguration) {
>      // first, set the true defaults if nothing else happens.
>      // default action is to run the full pipeline.
> @@ -886,6 +894,8 @@ private void initDefaults(Configuration
> baseConfiguration) {
>      } else {
>        this.conf = baseConfiguration;
>      }
> +    setHadoopJobHistLogsLoc(this.conf, DEF_HIST_LOGS);
> +    this.hadoopJobHistLogsLoc = getHadoopJobHistLogsLoc();
>
>      this.extraArgs = null;
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message