sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tijmen Ramakers (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SQOOP-3320) Commit behaviour leeds to errors on read-only databases
Date Thu, 26 Apr 2018 11:16:00 GMT
Tijmen Ramakers created SQOOP-3320:
--------------------------------------

             Summary: Commit behaviour leeds to errors on read-only databases
                 Key: SQOOP-3320
                 URL: https://issues.apache.org/jira/browse/SQOOP-3320
             Project: Sqoop
          Issue Type: Bug
          Components: connectors/generic
    Affects Versions: 1.4.7
            Reporter: Tijmen Ramakers


The current Sqoop behaviour is to do a commit after having retrieved the results of a query.
This leads to errors when the database (or the connection) is in read-only mode.

Example stack traces:
{code:java}
ERROR manager.SqlManager: SQLException closing ResultSet: org.netezza.error.NzSQLException:
ERROR:  Database 'XXXPRD' is available for read-only
        at org.netezza.internal.QueryExecutor.getNextResult(QueryExecutor.java:280)
        at org.netezza.internal.QueryExecutor.execute(QueryExecutor.java:76)
        at org.netezza.sql.NzConnection.execute(NzConnection.java:2819)
        at org.netezza.sql.NzConnection.execute(NzConnection.java:2791)
        at org.netezza.sql.NzConnection.commit(NzConnection.java:1161)
        at org.apache.sqoop.manager.SqlManager.getColumnNamesForRawQuery(SqlManager.java:182)
        at org.apache.sqoop.manager.SqlManager.getColumnNamesForQuery(SqlManager.java:142)

{code}
{code:java}
tool.EvalSqlTool: SQL exception executing statement: org.netezza.error.NzSQLException: ERROR: 
Database 'XXXPRD' is available for read-only         at org.netezza.internal.QueryExecutor.getNextResult(QueryExecutor.java:280)
        at org.netezza.internal.QueryExecutor.execute(QueryExecutor.java:76)        
at org.netezza.sql.NzConnection.execute(NzConnection.java:2819)         at org.netezza.sql.NzConnection.execute(NzConnection.java:2791)
        at org.netezza.sql.NzConnection.commit(NzConnection.java:1161)        
at org.apache.sqoop.tool.EvalSqlTool.run(EvalSqlTool.java:78)         at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
 
{code}
 

When using _sqoop eval_, it is not a major issue since it happens after the data is already
fetched and displayed. When using _sqoop import_ however, the spawned mapreduce jobs all fail,
so this is blocking the whole import.

This commit behaviour exists in several methods, at least the following:

org.apache.sqoop.tool.EvalSqlTool::run (line 78)

org.apache.sqoop.mapreduce.db.DBRecordReader::close (line 169)

org.apache.sqoop.manager.SqlManager::getColumnNamesForRawQuery (line 182)

org.apache.sqoop.manager.SqlManager::getColumnNamesForProcedure (line 226)

 

When a connection is in autocommit mode, then a manual commit is not needed anyway. When a
connection is in read-only mode, then it should still be possible to do a rollback in order
to close an outstanding transaction.

I propose to change the mentioned method lines ("connection.commit();" or equivalent), to:
{code:java}
if (!connection.getAutoCommit()) {
  if (connection.isReadOnly())
    connection.rollback();
  else
    connection.commit();
}
{code}
(or equivalent)

 

(Overall, I don't know why a commit is done in the above mentioned methods, as data should
have only been read, if the idea is to just close the transaction, why not do a rollback instead
just as a safety measure?)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message