sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vijay <vijay.ragav...@gmail.com>
Subject Sqoop Export from Java Program - HDFS Path Assignment issue
Date Tue, 11 Jun 2013 05:09:44 GMT
I have written a java program to extract data from HDFS file system to
MySQL database.When I run the program it tries to read it from local file
system instead of assigned HDFS file system. Can any one please analyze the
below code and let me know the issue?

My Configurations

Running Single Node Server in Red Hat Linux 5
Hadoop 1.2.0
Sqoop 1.4.3
Running from Eclipse.
Program Follows :

     package com.archival.da;

     import java.io.IOException;
     import javax.servlet.ServletException;
     import javax.servlet.http.HttpServlet;
     import javax.servlet.http.HttpServletRequest;
     import javax.servlet.http.HttpServletResponse;
     import java.sql.Connection;
     import java.sql.PreparedStatement;
     import java.sql.ResultSet;
     import org.apache.hadoop.conf.*;
     import org.apache.hadoop.fs.*;

     import com.cloudera.sqoop.*;
     import com.cloudera.sqoop.tool.ExportTool;

     @SuppressWarnings("serial")
     public class DataExport extends HttpServlet {



 @SuppressWarnings("deprecation")
public void doPost(HttpServletRequest request, HttpServletResponse
response) throws ServletException, IOException {
     response.setContentType("text/html");


     String run_id = request.getParameter("run_id");
     Connection con=GetCon.getCon();
    PreparedStatement ps1;
    try {

        String driver = "com.mysql.jdbc.Driver";
        Class.forName(driver).newInstance();

            // Get running process Run ID to track and update status
            ps1=con.prepareStatement("SELECT
POLICY.SRC_TABLE,POLICY.SRC_DB,CON.SERVER,CON.PORT,RT.RUN_DATE,CON.USER,CON.PWD
FROM POLICY JOIN CONNECTION AS CON ON POLICY.C_ID=CON.C_ID JOIN RUN_TRACKER
AS RT ON POLICY.ID <http://policy.id/>=RT.POLICY_ID AND RUN_ID=?");
            ps1.setString(1,run_id);
            ResultSet rs1=ps1.executeQuery();
            rs1.next();
            String tgtTable=rs1.getString(1);
            String runDate=rs1.getDate(5).toString();
            String newRunDate=runDate.replace("-", "_");
            String restore_dir=tgtTable+"_"+newRunDate;
            String ServerNm =
"jdbc:mysql://"+rs1.getString(3)+":"+rs1.getString(4)+"/"+rs1.getString(2);
            String ConUser=rs1.getString(6);
            String ConPass=rs1.getString(7);


            Configuration config = new Configuration();
            config.addResource(new
Path("/ms/hadoop-1.2.0/conf/core-site.xml"));
            config.addResource(new
Path("/ms/hadoop-1.2.0/conf/hdfs-site.xml"));
            FileSystem dfs = FileSystem.get(config);
            String exportDir=dfs.getWorkingDirectory()+"/"+restore_dir;
            System.out.println(exportDir);
            Path path = new Path(exportDir);
            SqoopOptions options=new SqoopOptions();
            options.setDriverClassName(driver);
            options.setHadoopMapRedHome("/ms/hadoop-1.2.0");
            options.setConnectString(ServerNm);
            options.setUsername(ConUser);
            options.setPassword(ConPass);
            options.setExportDir(exportDir);
            options.setTableName(tgtTable);
            options.setInputFieldsTerminatedBy(',');
            options.setNumMappers(1);

            int status=new ExportTool().run(options);
            System.out.println(status);
            if(status==0){

            dfs.delete(path,true);


            }



              response.getWriter().write("Restore Process Completed");

              con.close();
        } catch (Exception e){
            e.printStackTrace();
        }

}

    }
Console Error Message : I also printed to ensure the exportDir is assigned
with hdfs path

hdfs://localhost:9000/user/root/city_2013_06_10
13/06/09 21:00:22 WARN sqoop.ConnFactory: $SQOOP_CONF_DIR has not been set
in the     environment. Cannot check for additional configuration.
13/06/09 21:00:23 WARN sqoop.ConnFactory: Parameter --driver is set to an
explicit driver however appropriate connection manager is not being set
(via --connection-manager).     Sqoop is going to fall back to
org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly
which connection manager should be used next time.
13/06/09 21:00:23 INFO manager.SqlManager: Using default fetchSize of 1000
13/06/09 21:00:23 INFO tool.CodeGenTool: Beginning code generation
13/06/09 21:00:23 INFO manager.SqlManager: Executing SQL statement: SELECT
t.* FROM city AS t WHERE 1=0
13/06/09 21:00:23 INFO manager.SqlManager: Executing SQL statement: SELECT
t.* FROM city   AS t WHERE 1=0
13/06/09 21:00:23 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
/ms/hadoop-1.2.0
Note: /tmp/sqoop-root/compile/03132a553452f11e2f5343a96172125e/city.java
uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
13/06/09 21:00:23 INFO orm.CompilationManager: Writing jar file:
/tmp/sqoop-root/compile/03132a553452f11e2f5343a96172125e/city.jar
13/06/09 21:00:23 INFO mapreduce.ExportJobBase: Beginning export of city
13/06/09 21:00:24 INFO manager.SqlManager: Executing SQL statement: SELECT
t.* FROM city AS t WHERE 1=0
13/06/09 21:00:24 WARN mapreduce.JobBase: SQOOP_HOME is unset. May not be
able to find all job dependencies.
13/06/09 21:00:25 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
13/06/09 21:00:25 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
13/06/09 21:00:27 INFO input.FileInputFormat: Total input paths to process
: 1
13/06/09 21:00:27 INFO input.FileInputFormat: Total input paths to process
: 1
13/06/09 21:00:27 WARN snappy.LoadSnappy: Snappy native library not loaded
13/06/09 21:00:27 INFO mapred.JobClient: Cleaning up the staging area
file:/tmp/hadoop-root/mapred/staging/root1015206311/.staging/job_local1015206311_0001
13/06/09 21:00:27 ERROR security.UserGroupInformation:
PriviledgedActionException as:root cause:java.io.FileNotFoundException:
File /user/root/city_2013_06_10/part-m-00000 does not exist.
13/06/09 21:00:27 ERROR tool.ExportTool: Encountered IOException running
export job: java.io.FileNotFoundException: File
/user/root/city_2013_06_10/part-m-00000 does not exist.
just addtional note..In Eclipse, it shows
com.cloudera.sqoop.tool.ExportTool; is deprecated.


On Tue, Jun 11, 2013 at 10:21 AM, Vijay <vijay.ragavann@gmail.com> wrote:

> I have written a java program to extract data from HDFS file system to
> MySQL database.When I run the program it tries to read it from local file
> system instead of assigned HDFS file system. Can any one please analyze the
> below code and let me know the issue?
>
> My Configurations
>
> Running Single Node Server in Red Hat Linux 5
> Hadoop 1.2.0
> Sqoop 1.4.3
> Running from Eclipse.
> Program Follows :
>
>      package com.archival.da;
>
>      import java.io.IOException;
>      import javax.servlet.ServletException;
>      import javax.servlet.http.HttpServlet;
>      import javax.servlet.http.HttpServletRequest;
>      import javax.servlet.http.HttpServletResponse;
>      import java.sql.Connection;
>      import java.sql.PreparedStatement;
>      import java.sql.ResultSet;
>      import org.apache.hadoop.conf.*;
>      import org.apache.hadoop.fs.*;
>
>      import com.cloudera.sqoop.*;
>      import com.cloudera.sqoop.tool.ExportTool;
>
>      @SuppressWarnings("serial")
>      public class DataExport extends HttpServlet {
>
>
>
>  @SuppressWarnings("deprecation")
> public void doPost(HttpServletRequest request, HttpServletResponse
> response) throws ServletException, IOException {
>      response.setContentType("text/html");
>
>
>      String run_id = request.getParameter("run_id");
>      Connection con=GetCon.getCon();
>     PreparedStatement ps1;
>     try {
>
>         String driver = "com.mysql.jdbc.Driver";
>         Class.forName(driver).newInstance();
>
>             // Get running process Run ID to track and update status
>             ps1=con.prepareStatement("SELECT
> POLICY.SRC_TABLE,POLICY.SRC_DB,CON.SERVER,CON.PORT,RT.RUN_DATE,CON.USER,CON.PWD
> FROM POLICY JOIN CONNECTION AS CON ON POLICY.C_ID=CON.C_ID JOIN RUN_TRACKER
> AS RT ON POLICY.ID=RT.POLICY_ID AND RUN_ID=?");
>             ps1.setString(1,run_id);
>             ResultSet rs1=ps1.executeQuery();
>             rs1.next();
>             String tgtTable=rs1.getString(1);
>             String runDate=rs1.getDate(5).toString();
>             String newRunDate=runDate.replace("-", "_");
>             String restore_dir=tgtTable+"_"+newRunDate;
>             String ServerNm =
> "jdbc:mysql://"+rs1.getString(3)+":"+rs1.getString(4)+"/"+rs1.getString(2);
>             String ConUser=rs1.getString(6);
>             String ConPass=rs1.getString(7);
>
>
>             Configuration config = new Configuration();
>             config.addResource(new
> Path("/ms/hadoop-1.2.0/conf/core-site.xml"));
>             config.addResource(new
> Path("/ms/hadoop-1.2.0/conf/hdfs-site.xml"));
>             FileSystem dfs = FileSystem.get(config);
>             String exportDir=dfs.getWorkingDirectory()+"/"+restore_dir;
>             System.out.println(exportDir);
>             Path path = new Path(exportDir);
>             SqoopOptions options=new SqoopOptions();
>             options.setDriverClassName(driver);
>             options.setHadoopMapRedHome("/ms/hadoop-1.2.0");
>             options.setConnectString(ServerNm);
>             options.setUsername(ConUser);
>             options.setPassword(ConPass);
>             options.setExportDir(exportDir);
>             options.setTableName(tgtTable);
>             options.setInputFieldsTerminatedBy(',');
>             options.setNumMappers(1);
>
>             int status=new ExportTool().run(options);
>             System.out.println(status);
>             if(status==0){
>
>             dfs.delete(path,true);
>
>
>             }
>
>
>
>               response.getWriter().write("Restore Process Completed");
>
>               con.close();
>         } catch (Exception e){
>             e.printStackTrace();
>         }
>
> }
>
>     }
> Console Error Message : I also printed to ensure the exportDir is assigned
> with hdfs path
>
> hdfs://localhost:9000/user/root/city_2013_06_10
> 13/06/09 21:00:22 WARN sqoop.ConnFactory: $SQOOP_CONF_DIR has not been set
> in the     environment. Cannot check for additional configuration.
> 13/06/09 21:00:23 WARN sqoop.ConnFactory: Parameter --driver is set to an
> explicit driver however appropriate connection manager is not being set
> (via --connection-manager).     Sqoop is going to fall back to
> org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly
> which connection manager should be used next time.
> 13/06/09 21:00:23 INFO manager.SqlManager: Using default fetchSize of 1000
> 13/06/09 21:00:23 INFO tool.CodeGenTool: Beginning code generation
> 13/06/09 21:00:23 INFO manager.SqlManager: Executing SQL statement: SELECT
> t.* FROM city AS t WHERE 1=0
> 13/06/09 21:00:23 INFO manager.SqlManager: Executing SQL statement: SELECT
> t.* FROM city   AS t WHERE 1=0
> 13/06/09 21:00:23 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
> /ms/hadoop-1.2.0
> Note: /tmp/sqoop-root/compile/03132a553452f11e2f5343a96172125e/city.java
> uses or overrides a deprecated API.
> Note: Recompile with -Xlint:deprecation for details.
> 13/06/09 21:00:23 INFO orm.CompilationManager: Writing jar file:
> /tmp/sqoop-root/compile/03132a553452f11e2f5343a96172125e/city.jar
> 13/06/09 21:00:23 INFO mapreduce.ExportJobBase: Beginning export of city
> 13/06/09 21:00:24 INFO manager.SqlManager: Executing SQL statement: SELECT
> t.* FROM city AS t WHERE 1=0
> 13/06/09 21:00:24 WARN mapreduce.JobBase: SQOOP_HOME is unset. May not be
> able to find all job dependencies.
> 13/06/09 21:00:25 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 13/06/09 21:00:25 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
> 13/06/09 21:00:27 INFO input.FileInputFormat: Total input paths to process
> : 1
> 13/06/09 21:00:27 INFO input.FileInputFormat: Total input paths to process
> : 1
> 13/06/09 21:00:27 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/06/09 21:00:27 INFO mapred.JobClient: Cleaning up the staging area
> file:/tmp/hadoop-root/mapred/staging/root1015206311/.staging/job_local1015206311_0001
> 13/06/09 21:00:27 ERROR security.UserGroupInformation:
> PriviledgedActionException as:root cause:java.io.FileNotFoundException:
> File /user/root/city_2013_06_10/part-m-00000 does not exist.
> 13/06/09 21:00:27 ERROR tool.ExportTool: Encountered IOException running
> export job: java.io.FileNotFoundException: File
> /user/root/city_2013_06_10/part-m-00000 does not exist.
>
>

Mime
View raw message