sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rakesh Sharma <raksha...@expedia.com>
Subject Re: Stable release for sqoop
Date Mon, 20 Oct 2014 07:36:54 GMT
Hi,

We are trying to generate an rpm for sqoop but it is taking time for us for the same.
If anyone has already created an rpm for sqoop, it will be of great help. Could you please
provide us the spec file and send us step by step process to create rpm.

We are working on a  tight schedule and any help will be really appreciated.

Thanks and Regards,
Rakesh.


From: Abraham Elmahrek <abe@cloudera.com<mailto:abe@cloudera.com>>
Date: Friday, September 26, 2014 at 5:25 AM
To: Rakesh Sharma <raksharma@expedia.com<mailto:raksharma@expedia.com>>
Cc: Atul Gupta <atugupta@expedia.com<mailto:atugupta@expedia.com>>, "dev@sqoop.apache.org<mailto:dev@sqoop.apache.org>"
<dev@sqoop.apache.org<mailto:dev@sqoop.apache.org>>, Shashank Tandon <standon@expedia.com<mailto:standon@expedia.com>>
Subject: Re: Stable release for sqoop

I'd still recommend Sqoop 1.4.5 or use a version of Sqoop that's provided by one of the vendors.

For reference though, could you provide the Schema of this table?

On Thu, Sep 25, 2014 at 3:41 AM, Rakesh Sharma <raksharma@expedia.com<mailto:raksharma@expedia.com>>
wrote:
Hi Abe,

Thanks for the prompt reply and support. We are using the following sqoop command  for code
from 1.4.5 branch:

sqoop import -Dmapred.job.queue.name<http://Dmapred.job.queue.name>=default --connect
jdbc:mysql://localhost/test --username root --password '' --table Temp_Employee  --incremental
lastmodified --check-column last_updated  --last-value "2014-001-01 00:00:00"  --hive-import
 --hive-table="Temp_Employee" -m 1

And we are getting the following exception :
14/09/25 03:35:11 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `Temp_Employee`
AS t LIMIT 1
14/09/25 03:35:11 ERROR manager.SqlManager: Error reading from database: java.sql.SQLException:
Streaming result set com.mysql.jdbc.RowDataDynamic@69222fc3 is still active. No statements
may be issued when any streaming result sets are open and in use on a given connection. Ensure
that you have called .close() on any active streaming result sets before attempting more queries.
java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@69222fc3 is still
active. No statements may be issued when any streaming result sets are open and in use on
a given connection. Ensure that you have called .close() on any active streaming result sets
before attempting more queries.
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingData(MysqlIO.java:2735)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1899)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2569)
at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1524)
at com.mysql.jdbc.ConnectionImpl.getMaxBytesPerChar(ConnectionImpl.java:3003)
at com.mysql.jdbc.Field.getMaxBytesPerCharacter(Field.java:602)
at com.mysql.jdbc.ResultSetMetaData.getPrecision(ResultSetMetaData.java:445)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:285)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:240)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:226)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1773)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1578)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
14/09/25 03:35:11 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException:
No columns to generate for ClassWriter
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1584)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

The same command works on 1.4.6(trunk) and even on 1.4.4 version of sqoop. Could you please
suggest what code is stable and we can make that as base of our development.

And Regarding the feature list, I will send you a separate mail with list of features. A couple
of them are very generic and we are planning to contribute them back to community. May be
we will need your help for that.

Thanks and Regards,
Rakesh.



From: Atul Gupta <atugupta@expedia.com<mailto:atugupta@expedia.com>>
Date: Tuesday, September 23, 2014 at 12:42 PM
To: Abraham Elmahrek <abe@cloudera.com<mailto:abe@cloudera.com>>
Cc: "dev@sqoop.apache.org<mailto:dev@sqoop.apache.org>" <dev@sqoop.apache.org<mailto:dev@sqoop.apache.org>>,
Shashank Tandon <standon@expedia.com<mailto:standon@expedia.com>>, Rakesh Sharma
<raksharma@expedia.com<mailto:raksharma@expedia.com>>
Subject: Re: Stable release for sqoop

++rakesh

Regards,
Atul

On 23-Sep-2014, at 12:16 pm, Abraham Elmahrek <abe@cloudera.com<mailto:abe@cloudera.com>>
wrote:

Hey there,

Could you provide your incremental import command? Also, what security enhancements and customizations
are you looking into? It might be something that fits generically into Sqoop. Sqoop 1.4.5
is the latest community release of Sqoop.

-Abe

On Mon, Sep 22, 2014 at 9:59 PM, Rakesh Sharma <raksharma@expedia.com<mailto:raksharma@expedia.com>>
wrote:
Hi,

We are planning to use Sqoop at Expedia but we might have to do some enhancements and customizations
into Sqoop. We were planning to base our work with 1.4.5 branch but we found a couple of issues
with code from that branch.
eg., We were not able to import data from mysql db using incremental import with 1.4.5 branch
but it is running properly with latest code and even in 1.4.4 branch as well.

Could you please suggest the stable version after 1.4.5 which we can pick to use as base for
our development.

Thanks and Regards,
Rakesh.






Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message