sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Alten-Lorenz <wget.n...@gmail.com>
Subject Re: Sqoop Export from Java Program - HDFS Path Assignment issue
Date Tue, 11 Jun 2013 05:45:32 GMT
Hi,

Do you have hadoop/conf/*in your path?

- Alex

On Jun 11, 2013, at 7:09 AM, Vijay <vijay.ragavann@gmail.com> wrote:

> I have written a java program to extract data from HDFS file system to MySQL database.When
I run the program it tries to read it from local file system instead of assigned HDFS file
system. Can any one please analyze the below code and let me know the issue?
> 
> My Configurations
> 
> Running Single Node Server in Red Hat Linux 5
> Hadoop 1.2.0
> Sqoop 1.4.3 
> Running from Eclipse.
> Program Follows :
> 
>      package com.archival.da;
> 
>      import java.io.IOException;
>      import javax.servlet.ServletException;
>      import javax.servlet.http.HttpServlet;
>      import javax.servlet.http.HttpServletRequest;
>      import javax.servlet.http.HttpServletResponse;
>      import java.sql.Connection;
>      import java.sql.PreparedStatement;
>      import java.sql.ResultSet;
>      import org.apache.hadoop.conf.*;
>      import org.apache.hadoop.fs.*;
> 
>      import com.cloudera.sqoop.*;
>      import com.cloudera.sqoop.tool.ExportTool;
> 
>      @SuppressWarnings("serial")
>      public class DataExport extends HttpServlet {
> 
> 
> 
>  @SuppressWarnings("deprecation")
> public void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException,
IOException {
>      response.setContentType("text/html");
> 
> 
>      String run_id = request.getParameter("run_id");
>      Connection con=GetCon.getCon();
>     PreparedStatement ps1;
>     try {
> 
>         String driver = "com.mysql.jdbc.Driver";
>         Class.forName(driver).newInstance();
> 
>             // Get running process Run ID to track and update status 
>             ps1=con.prepareStatement("SELECT POLICY.SRC_TABLE,POLICY.SRC_DB,CON.SERVER,CON.PORT,RT.RUN_DATE,CON.USER,CON.PWD
FROM POLICY JOIN CONNECTION AS CON ON POLICY.C_ID=CON.C_ID JOIN RUN_TRACKER AS RT ON POLICY.ID=RT.POLICY_ID
AND RUN_ID=?");
>             ps1.setString(1,run_id);
>             ResultSet rs1=ps1.executeQuery();
>             rs1.next();
>             String tgtTable=rs1.getString(1);
>             String runDate=rs1.getDate(5).toString();
>             String newRunDate=runDate.replace("-", "_");
>             String restore_dir=tgtTable+"_"+newRunDate;
>             String ServerNm = "jdbc:mysql://"+rs1.getString(3)+":"+rs1.getString(4)+"/"+rs1.getString(2);
>             String ConUser=rs1.getString(6);
>             String ConPass=rs1.getString(7);
> 
> 
>             Configuration config = new Configuration(); 
>             config.addResource(new Path("/ms/hadoop-1.2.0/conf/core-site.xml"));
>             config.addResource(new Path("/ms/hadoop-1.2.0/conf/hdfs-site.xml"));
>             FileSystem dfs = FileSystem.get(config);
>             String exportDir=dfs.getWorkingDirectory()+"/"+restore_dir;
>             System.out.println(exportDir);
>             Path path = new Path(exportDir);
>             SqoopOptions options=new SqoopOptions();
>             options.setDriverClassName(driver);
>             options.setHadoopMapRedHome("/ms/hadoop-1.2.0");
>             options.setConnectString(ServerNm);
>             options.setUsername(ConUser);
>             options.setPassword(ConPass);
>             options.setExportDir(exportDir);
>             options.setTableName(tgtTable);
>             options.setInputFieldsTerminatedBy(',');
>             options.setNumMappers(1);
> 
>             int status=new ExportTool().run(options);
>             System.out.println(status);
>             if(status==0){
> 
>             dfs.delete(path,true);
> 
> 
>             }
> 
> 
> 
>               response.getWriter().write("Restore Process Completed");
> 
>               con.close();          
>         } catch (Exception e){
>             e.printStackTrace(); 
>         }
> 
> }
> 
>     }
> Console Error Message : I also printed to ensure the exportDir is assigned with hdfs
path
> 
> hdfs://localhost:9000/user/root/city_2013_06_10
> 13/06/09 21:00:22 WARN sqoop.ConnFactory: $SQOOP_CONF_DIR has not been set in the   
 environment. Cannot check for additional configuration.
> 13/06/09 21:00:23 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver
however appropriate connection manager is not being set (via --connection-manager).     Sqoop
is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly
which connection manager should be used next time.
> 13/06/09 21:00:23 INFO manager.SqlManager: Using default fetchSize of 1000
> 13/06/09 21:00:23 INFO tool.CodeGenTool: Beginning code generation
> 13/06/09 21:00:23 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM city
AS t WHERE 1=0
> 13/06/09 21:00:23 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM city
  AS t WHERE 1=0
> 13/06/09 21:00:23 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /ms/hadoop-1.2.0
> Note: /tmp/sqoop-root/compile/03132a553452f11e2f5343a96172125e/city.java uses or overrides
a deprecated API.
> Note: Recompile with -Xlint:deprecation for details.
> 13/06/09 21:00:23 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/03132a553452f11e2f5343a96172125e/city.jar
> 13/06/09 21:00:23 INFO mapreduce.ExportJobBase: Beginning export of city
> 13/06/09 21:00:24 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM city
AS t WHERE 1=0
> 13/06/09 21:00:24 WARN mapreduce.JobBase: SQOOP_HOME is unset. May not be able to find
all job dependencies.
> 13/06/09 21:00:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for
your platform... using builtin-java classes where applicable
> 13/06/09 21:00:25 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments.
Applications should implement Tool for the same.
> 13/06/09 21:00:27 INFO input.FileInputFormat: Total input paths to process : 1
> 13/06/09 21:00:27 INFO input.FileInputFormat: Total input paths to process : 1
> 13/06/09 21:00:27 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/06/09 21:00:27 INFO mapred.JobClient: Cleaning up the staging area file:/tmp/hadoop-root/mapred/staging/root1015206311/.staging/job_local1015206311_0001
> 13/06/09 21:00:27 ERROR security.UserGroupInformation: PriviledgedActionException as:root
cause:java.io.FileNotFoundException: File /user/root/city_2013_06_10/part-m-00000 does not
exist.
> 13/06/09 21:00:27 ERROR tool.ExportTool: Encountered IOException running export job:
java.io.FileNotFoundException: File /user/root/city_2013_06_10/part-m-00000 does not exist.
> just addtional note..In Eclipse, it shows com.cloudera.sqoop.tool.ExportTool; is deprecated.
> 
> 
> On Tue, Jun 11, 2013 at 10:21 AM, Vijay <vijay.ragavann@gmail.com> wrote:
> I have written a java program to extract data from HDFS file system to MySQL database.When
I run the program it tries to read it from local file system instead of assigned HDFS file
system. Can any one please analyze the below code and let me know the issue?
> 
> My Configurations
> 
> Running Single Node Server in Red Hat Linux 5
> Hadoop 1.2.0
> Sqoop 1.4.3 
> Running from Eclipse.
> Program Follows :
> 
>      package com.archival.da;
> 
>      import java.io.IOException;
>      import javax.servlet.ServletException;
>      import javax.servlet.http.HttpServlet;
>      import javax.servlet.http.HttpServletRequest;
>      import javax.servlet.http.HttpServletResponse;
>      import java.sql.Connection;
>      import java.sql.PreparedStatement;
>      import java.sql.ResultSet;
>      import org.apache.hadoop.conf.*;
>      import org.apache.hadoop.fs.*;
> 
>      import com.cloudera.sqoop.*;
>      import com.cloudera.sqoop.tool.ExportTool;
> 
>      @SuppressWarnings("serial")
>      public class DataExport extends HttpServlet {
> 
> 
> 
>  @SuppressWarnings("deprecation")
> public void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException,
IOException {
>      response.setContentType("text/html");
> 
> 
>      String run_id = request.getParameter("run_id");
>      Connection con=GetCon.getCon();
>     PreparedStatement ps1;
>     try {
> 
>         String driver = "com.mysql.jdbc.Driver";
>         Class.forName(driver).newInstance();
> 
>             // Get running process Run ID to track and update status 
>             ps1=con.prepareStatement("SELECT POLICY.SRC_TABLE,POLICY.SRC_DB,CON.SERVER,CON.PORT,RT.RUN_DATE,CON.USER,CON.PWD
FROM POLICY JOIN CONNECTION AS CON ON POLICY.C_ID=CON.C_ID JOIN RUN_TRACKER AS RT ON POLICY.ID=RT.POLICY_ID
AND RUN_ID=?");
>             ps1.setString(1,run_id);
>             ResultSet rs1=ps1.executeQuery();
>             rs1.next();
>             String tgtTable=rs1.getString(1);
>             String runDate=rs1.getDate(5).toString();
>             String newRunDate=runDate.replace("-", "_");
>             String restore_dir=tgtTable+"_"+newRunDate;
>             String ServerNm = "jdbc:mysql://"+rs1.getString(3)+":"+rs1.getString(4)+"/"+rs1.getString(2);
>             String ConUser=rs1.getString(6);
>             String ConPass=rs1.getString(7);
> 
> 
>             Configuration config = new Configuration(); 
>             config.addResource(new Path("/ms/hadoop-1.2.0/conf/core-site.xml"));
>             config.addResource(new Path("/ms/hadoop-1.2.0/conf/hdfs-site.xml"));
>             FileSystem dfs = FileSystem.get(config);
>             String exportDir=dfs.getWorkingDirectory()+"/"+restore_dir;
>             System.out.println(exportDir);
>             Path path = new Path(exportDir);
>             SqoopOptions options=new SqoopOptions();
>             options.setDriverClassName(driver);
>             options.setHadoopMapRedHome("/ms/hadoop-1.2.0");
>             options.setConnectString(ServerNm);
>             options.setUsername(ConUser);
>             options.setPassword(ConPass);
>             options.setExportDir(exportDir);
>             options.setTableName(tgtTable);
>             options.setInputFieldsTerminatedBy(',');
>             options.setNumMappers(1);
> 
>             int status=new ExportTool().run(options);
>             System.out.println(status);
>             if(status==0){
> 
>             dfs.delete(path,true);
> 
> 
>             }
> 
> 
> 
>               response.getWriter().write("Restore Process Completed");
> 
>               con.close();          
>         } catch (Exception e){
>             e.printStackTrace(); 
>         }
> 
> }
> 
>     }
> Console Error Message : I also printed to ensure the exportDir is assigned with hdfs
path
> 
> hdfs://localhost:9000/user/root/city_2013_06_10
> 13/06/09 21:00:22 WARN sqoop.ConnFactory: $SQOOP_CONF_DIR has not been set in the   
 environment. Cannot check for additional configuration.
> 13/06/09 21:00:23 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver
however appropriate connection manager is not being set (via --connection-manager).     Sqoop
is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly
which connection manager should be used next time.
> 13/06/09 21:00:23 INFO manager.SqlManager: Using default fetchSize of 1000
> 13/06/09 21:00:23 INFO tool.CodeGenTool: Beginning code generation
> 13/06/09 21:00:23 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM city
AS t WHERE 1=0
> 13/06/09 21:00:23 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM city
  AS t WHERE 1=0
> 13/06/09 21:00:23 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /ms/hadoop-1.2.0
> Note: /tmp/sqoop-root/compile/03132a553452f11e2f5343a96172125e/city.java uses or overrides
a deprecated API.
> Note: Recompile with -Xlint:deprecation for details.
> 13/06/09 21:00:23 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/03132a553452f11e2f5343a96172125e/city.jar
> 13/06/09 21:00:23 INFO mapreduce.ExportJobBase: Beginning export of city
> 13/06/09 21:00:24 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM city
AS t WHERE 1=0
> 13/06/09 21:00:24 WARN mapreduce.JobBase: SQOOP_HOME is unset. May not be able to find
all job dependencies.
> 13/06/09 21:00:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for
your platform... using builtin-java classes where applicable
> 13/06/09 21:00:25 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments.
Applications should implement Tool for the same.
> 13/06/09 21:00:27 INFO input.FileInputFormat: Total input paths to process : 1
> 13/06/09 21:00:27 INFO input.FileInputFormat: Total input paths to process : 1
> 13/06/09 21:00:27 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/06/09 21:00:27 INFO mapred.JobClient: Cleaning up the staging area file:/tmp/hadoop-root/mapred/staging/root1015206311/.staging/job_local1015206311_0001
> 13/06/09 21:00:27 ERROR security.UserGroupInformation: PriviledgedActionException as:root
cause:java.io.FileNotFoundException: File /user/root/city_2013_06_10/part-m-00000 does not
exist.
> 13/06/09 21:00:27 ERROR tool.ExportTool: Encountered IOException running export job:
java.io.FileNotFoundException: File /user/root/city_2013_06_10/part-m-00000 does not exist.
> 

--
Alexander Alten-Lorenz
http://mapredit.blogspot.com
German Hadoop LinkedIn Group: http://goo.gl/N8pCF


Mime
View raw message