sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Veena Basavaraj (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SQOOP-1736) Sqoop2: Driver missing during upgrade
Date Sat, 15 Nov 2014 00:49:35 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14213186#comment-14213186
] 

Veena Basavaraj edited comment on SQOOP-1736 at 11/15/14 12:49 AM:
-------------------------------------------------------------------

tested with the data set attached.  still needs confirmation

[root@vbsqoop-1 bin]# ./sqoop.sh tool upgrade
Sqoop home directory: /root/projects/sqoop/sqoop-1.99.4-bin-hadoop200
Setting SQOOP_HTTP_PORT:     12000
Setting SQOOP_ADMIN_PORT:     12001
Using   CATALINA_OPTS:       
Adding to CATALINA_OPTS:    -Dsqoop.http.port=12000 -Dsqoop.admin.port=12001
Nov 14, 2014 4:45:11 PM org.apache.catalina.startup.ClassLoaderFactory validateFile
WARNING: Problem with directory [/root/projects/sqoop/sqoop-1.99.4-bin-hadoop200/lib], exists:
[false], isDirectory: [false], canRead: [false]
Sqoop tool executor:
	Version: 1.99.4
	Revision: 68d5beeab40b9d9da9f1c00da544185b161fdce6
	Compiled on Fri Nov 14 15:46:21 PST 2014 by vbasavaraj
Running tool: class org.apache.sqoop.tools.tool.UpgradeTool
2014-11-14 16:45:12,123 INFO  [main] core.PropertiesConfigurationProvider (PropertiesConfigurationProvider.java:initialize(96))
- Starting config file poller thread
log4j: Parsing for [root] with value=[WARN, file].
log4j: Level token is [WARN].
log4j: Category root set to WARN
log4j: Parsing appender named "file".
log4j: Parsing layout options for "file".
log4j: Setting property [conversionPattern] to [%d{ISO8601} %-5p %c{2} [%l] %m%n].
log4j: End of parsing for "file".
log4j: Setting property [file] to [@LOGDIR@/sqoop.log].
log4j: Setting property [maxBackupIndex] to [5].
log4j: Setting property [maxFileSize] to [25MB].
log4j: setFile called: @LOGDIR@/sqoop.log, true
log4j: setFile ended
log4j: Parsed "file" options.
log4j: Parsing for [org.apache.sqoop] with value=[DEBUG].
log4j: Level token is [DEBUG].
log4j: Category org.apache.sqoop set to DEBUG
log4j: Handling log4j.additivity.org.apache.sqoop=[null]
log4j: Parsing for [org.apache.derby] with value=[INFO].
log4j: Level token is [INFO].
log4j: Category org.apache.derby set to INFO
log4j: Handling log4j.additivity.org.apache.derby=[null]
log4j: Finished configuring.
Tool class org.apache.sqoop.tools.tool.UpgradeTool has finished correctly.



14-11-14 16:45:23,850 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.findDriver(DerbyRepositoryHandler.java:1073)]
Looking up Driver and config 
2014-11-14 16:45:23,852 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.findDriver(DerbyRepositoryHandler.java:1129)]
Looked up Driver and config
2014-11-14 16:45:23,896 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)]
Tx count-commit: 3, rollback: false
2014-11-14 16:45:23,897 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)]
Tx count-close: 2, rollback: false
2014-11-14 16:45:23,897 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)]
Tx count-begin: 3, rollback: false
2014-11-14 16:45:24,035 INFO  repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:443)]
 Link upgrade for link:hdfs-connector for connector:hdfs-connector
2014-11-14 16:45:24,134 INFO  repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:467)]
 All Link and configs for this connector processed
2014-11-14 16:45:24,135 INFO  repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:473)]
 Job upgrade for job:m2h for connector:hdfs-connector
2014-11-14 16:45:24,138 WARN  configurable.ConfigurableUpgradeUtil [org.apache.sqoop.configurable.ConfigurableUpgradeUtil.doUpgrade(ConfigurableUpgradeUtil.java:56)]
Input: 'toJobConfig.customCompression' not present in old configurable. So it will not be
transferred by the upgrader.
2014-11-14 16:45:24,179 INFO  repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:523)]
 All Job and configs for this connector processed
2014-11-14 16:45:24,179 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)]
Tx count-commit: 3, rollback: false
2014-11-14 16:45:24,179 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)]
Tx count-close: 2, rollback: false
2014-11-14 16:45:24,180 INFO  repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:543)]
Connector upgrade finished for: hdfs-connector
2014-11-14 16:45:24,180 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)]
Tx count-commit: 2, rollback: false
2014-11-14 16:45:24,180 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)]
Tx count-close: 1, rollback: false
2014-11-14 16:45:24,180 DEBUG connector.ConnectorManager [org.apache.sqoop.connector.ConnectorManager.registerConnectors(ConnectorManager.java:215)]
Registered connector: connector-hdfs-connector:2:org.apache.sqoop.connector.hdfs.HdfsConnector,
Link: Configs: config-linkConfig:10:[input-linkConfig.uri:30:STRING:255]Type: LINK, From:
Configs: config-fromJobConfig:11:[input-fromJobConfig.inputDirectory:31:STRING:255]Type: JOB,
To: Configs: config-toJobConfig:12:[input-toJobConfig.outputFormat:32:ENUM:TEXT_FILE,SEQUENCE_FILE,
input-toJobConfig.compression:33:ENUM:NONE,DEFAULT,DEFLATE,GZIP,BZIP2,LZO,LZ4,SNAPPY,CUSTOM,
input-toJobConfig.customCompression:34:STRING:255, input-toJobConfig.outputDirectory:35:STRING:255]Type:
JOB
2014-11-14 16:45:24,181 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)]
Tx count-begin: 2, rollback: false
2014-11-14 16:45:24,181 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.findConnector(DerbyRepositoryHandler.java:988)]
Looking up connector: generic-jdbc-connector
2014-11-14 16:45:24,190 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.findConnector(DerbyRepositoryHandler.java:1002)]
Looking up connector: generic-jdbc-connector, found: null
2014-11-14 16:45:24,190 INFO  repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:413)]
Upgrading connector: generic-jdbc-connector
2014-11-14 16:45:24,191 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)]
Tx count-begin: 3, rollback: false
2014-11-14 16:45:24,215 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)]
Tx count-commit: 3, rollback: false
2014-11-14 16:45:24,215 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)]
Tx count-close: 2, rollback: false
2014-11-14 16:45:24,215 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)]
Tx count-begin: 3, rollback: false
2014-11-14 16:45:24,216 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.findDriver(DerbyRepositoryHandler.java:1073)]
Looking up Driver and config 
2014-11-14 16:45:24,218 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.findDriver(DerbyRepositoryHandler.java:1129)]
Looked up Driver and config
2014-11-14 16:45:24,234 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)]
Tx count-commit: 3, rollback: false
2014-11-14 16:45:24,235 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)]
Tx count-close: 2, rollback: false
2014-11-14 16:45:24,235 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)]
Tx count-begin: 3, rollback: false
2014-11-14 16:45:24,312 INFO  repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:443)]
 Link upgrade for link:mysql for connector:generic-jdbc-connector
2014-11-14 16:45:24,313 WARN  configurable.ConfigurableUpgradeUtil [org.apache.sqoop.configurable.ConfigurableUpgradeUtil.doUpgrade(ConfigurableUpgradeUtil.java:47)]
Config: 'linkConfig' not present in old configurable. So it and its inputs will not be transferred
by the upgrader.
2014-11-14 16:45:24,613 INFO  repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:443)]
 Link upgrade for link:mysql2 for connector:generic-jdbc-connector
2014-11-14 16:45:24,614 WARN  configurable.ConfigurableUpgradeUtil [org.apache.sqoop.configurable.ConfigurableUpgradeUtil.doUpgrade(ConfigurableUpgradeUtil.java:47)]
Config: 'linkConfig' not present in old configurable. So it and its inputs will not be transferred
by the upgrader.
2014-11-14 16:45:24,649 INFO  repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:443)]
 Link upgrade for link:postgresql for connector:generic-jdbc-connector
2014-11-14 16:45:24,651 WARN  configurable.ConfigurableUpgradeUtil [org.apache.sqoop.configurable.ConfigurableUpgradeUtil.doUpgrade(ConfigurableUpgradeUtil.java:47)]
Config: 'linkConfig' not present in old configurable. So it and its inputs will not be transferred
by the upgrader.
2014-11-14 16:45:24,741 INFO  repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:467)]
 All Link and configs for this connector processed
2014-11-14 16:45:24,742 INFO  repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:473)]
 Job upgrade for job:m2h for connector:generic-jdbc-connector
2014-11-14 16:45:24,742 WARN  configurable.ConfigurableUpgradeUtil [org.apache.sqoop.configurable.ConfigurableUpgradeUtil.doUpgrade(ConfigurableUpgradeUtil.java:47)]
Config: 'fromJobConfig' not present in old configurable. So it and its inputs will not be
transferred by the upgrader.
2014-11-14 16:45:24,753 INFO  repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:523)]
 All Job and configs for this connector processed
2014-11-14 16:45:24,753 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)]
Tx count-commit: 3, rollback: false
2014-11-14 16:45:24,753 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)]
Tx count-close: 2, rollback: false
2014-11-14 16:45:24,754 INFO  repository.Repository [org.apache.sqoop.repository.Repository.upgradeConnector(Repository.java:543)]
Connector upgrade finished for: generic-jdbc-connector
2014-11-14 16:45:24,754 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)]
Tx count-commit: 2, rollback: false
2014-11-14 16:45:24,754 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)]
Tx count-close: 1, rollback: false
2014-11-14 16:45:24,755 DEBUG connector.ConnectorManager [org.apache.sqoop.connector.ConnectorManager.registerConnectors(ConnectorManager.java:215)]
Registered connector: connector-generic-jdbc-connector:1:org.apache.sqoop.connector.jdbc.GenericJdbcConnector,
Link: Configs: config-linkConfig:13:[input-linkConfig.jdbcDriver:36:STRING:128, input-linkConfig.connectionString:37:STRING:128,
input-linkConfig.username:38:STRING:40, input-linkConfig.password:39:STRING:40, input-linkConfig.jdbcProperties:40:MAP]Type:
LINK, From: Configs: config-fromJobConfig:14:[input-fromJobConfig.schemaName:41:STRING:50,
input-fromJobConfig.tableName:42:STRING:50, input-fromJobConfig.sql:43:STRING:2000, input-fromJobConfig.columns:44:STRING:50,
input-fromJobConfig.partitionColumn:45:STRING:50, input-fromJobConfig.allowNullValueInPartitionColumn:46:BOOLEAN,
input-fromJobConfig.boundaryQuery:47:STRING:50]Type: JOB, To: Configs: config-toJobConfig:15:[input-toJobConfig.schemaName:48:STRING:50,
input-toJobConfig.tableName:49:STRING:2000, input-toJobConfig.sql:50:STRING:50, input-toJobConfig.columns:51:STRING:50,
input-toJobConfig.stageTableName:52:STRING:2000, input-toJobConfig.shouldClearStageTable:53:BOOLEAN]Type:
JOB
2014-11-14 16:45:24,755 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)]
Tx count-commit: 1, rollback: false
2014-11-14 16:45:24,755 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)]
Tx count-close: 0, rollback: false
2014-11-14 16:45:24,755 INFO  repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)]
Attempting transaction commit
2014-11-14 16:45:24,776 INFO  connector.ConnectorManager [org.apache.sqoop.connector.ConnectorManager.initialize(ConnectorManager.java:185)]
Connectors loaded: {hdfs-connector={hdfs-connector:org.apache.sqoop.connector.hdfs.HdfsConnector:jar:file:/root/projects/sqoop/sqoop-1.99.4-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-connector-hdfs-1.99.4.jar!/sqoopconnector.properties},
generic-jdbc-connector={generic-jdbc-connector:org.apache.sqoop.connector.jdbc.GenericJdbcConnector:jar:file:/root/projects/sqoop/sqoop-1.99.4-bin-hadoop200/server/webapps/sqoop/WEB-INF/lib/sqoop-connector-generic-jdbc-1.99.4.jar!/sqoopconnector.properties}}
2014-11-14 16:45:24,776 INFO  tool.UpgradeTool [org.apache.sqoop.tools.tool.UpgradeTool.runToolWithConfiguration(UpgradeTool.java:44)]
Initializing the Driver with upgrade option turned on.
2014-11-14 16:45:24,780 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:73)]
Tx count-begin: 1, rollback: false
2014-11-14 16:45:24,780 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.findDriver(DerbyRepositoryHandler.java:1073)]
Looking up Driver and config 
2014-11-14 16:45:24,782 DEBUG derby.DerbyRepositoryHandler [org.apache.sqoop.repository.derby.DerbyRepositoryHandler.findDriver(DerbyRepositoryHandler.java:1129)]
Looked up Driver and config
2014-11-14 16:45:24,783 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.commit(JdbcRepositoryTransaction.java:84)]
Tx count-commit: 1, rollback: false
2014-11-14 16:45:24,783 DEBUG repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:103)]
Tx count-close: 0, rollback: false
2014-11-14 16:45:24,783 INFO  repository.JdbcRepositoryTransaction [org.apache.sqoop.repository.JdbcRepositoryTransaction.close(JdbcRepositoryTransaction.java:111)]
Attempting transaction commit
2014-11-14 16:45:24,784 INFO  driver.Driver [org.apache.sqoop.driver.Driver.initialize(Driver.java:134)]
Driver initialized: OK
2014-11-14 16:45:24,784 INFO  tool.UpgradeTool [org.apache.sqoop.tools.tool.UpgradeTool.runToolWithConfiguration(UpgradeTool.java:47)]
Upgrade completed successfully.
2014-11-14 16


was (Author: vybs):
tested with the data set attached.  still needs confirmation

[root@vbsqoop-1 bin]# ./sqoop.sh tool upgrade
Sqoop home directory: /root/projects/sqoop/sqoop-1.99.4-bin-hadoop200
Setting SQOOP_HTTP_PORT:     12000
Setting SQOOP_ADMIN_PORT:     12001
Using   CATALINA_OPTS:       
Adding to CATALINA_OPTS:    -Dsqoop.http.port=12000 -Dsqoop.admin.port=12001
Nov 14, 2014 4:45:11 PM org.apache.catalina.startup.ClassLoaderFactory validateFile
WARNING: Problem with directory [/root/projects/sqoop/sqoop-1.99.4-bin-hadoop200/lib], exists:
[false], isDirectory: [false], canRead: [false]
Sqoop tool executor:
	Version: 1.99.4
	Revision: 68d5beeab40b9d9da9f1c00da544185b161fdce6
	Compiled on Fri Nov 14 15:46:21 PST 2014 by vbasavaraj
Running tool: class org.apache.sqoop.tools.tool.UpgradeTool
2014-11-14 16:45:12,123 INFO  [main] core.PropertiesConfigurationProvider (PropertiesConfigurationProvider.java:initialize(96))
- Starting config file poller thread
log4j: Parsing for [root] with value=[WARN, file].
log4j: Level token is [WARN].
log4j: Category root set to WARN
log4j: Parsing appender named "file".
log4j: Parsing layout options for "file".
log4j: Setting property [conversionPattern] to [%d{ISO8601} %-5p %c{2} [%l] %m%n].
log4j: End of parsing for "file".
log4j: Setting property [file] to [@LOGDIR@/sqoop.log].
log4j: Setting property [maxBackupIndex] to [5].
log4j: Setting property [maxFileSize] to [25MB].
log4j: setFile called: @LOGDIR@/sqoop.log, true
log4j: setFile ended
log4j: Parsed "file" options.
log4j: Parsing for [org.apache.sqoop] with value=[DEBUG].
log4j: Level token is [DEBUG].
log4j: Category org.apache.sqoop set to DEBUG
log4j: Handling log4j.additivity.org.apache.sqoop=[null]
log4j: Parsing for [org.apache.derby] with value=[INFO].
log4j: Level token is [INFO].
log4j: Category org.apache.derby set to INFO
log4j: Handling log4j.additivity.org.apache.derby=[null]
log4j: Finished configuring.
Tool class org.apache.sqoop.tools.tool.UpgradeTool has finished correctly.

> Sqoop2: Driver missing during upgrade
> -------------------------------------
>
>                 Key: SQOOP-1736
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1736
>             Project: Sqoop
>          Issue Type: Bug
>          Components: sqoop2-framework
>    Affects Versions: 1.99.4
>            Reporter: Abraham Elmahrek
>            Assignee: Veena Basavaraj
>             Fix For: 1.99.4
>
>         Attachments: 1993.tar.gz, SQOOP-1736-1.99.4-v2.patch, SQOOP-1736-1.99.4-v3.patch,
SQOOP-1736-1.99.4.patch
>
>
> There are two issues:
> # The driver doesn't exist for existing jobs and links to reference when upgrading. During
repository upgrade, a driver should be created (not done).
> # The original framework forms should be transformed to driver configs (done) and reference
the driver configurable (not done).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message