sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jérôme Verdier <verdier.jerom...@gmail.com>
Subject Re: import from Oracle to Hive : 2 errors
Date Tue, 18 Jun 2013 14:02:58 GMT
Hi Venkat,

thanks for  your  response.

i realise that my question was so stupid =).

i have rewrited the sqoop script with --hive-import option like this :

sqoop import --connect jdbc:oracle:thin:@xx.xx.xx.xx:1521/D_BI --username
visiteur --password visiteur --hive-import --create-hive-table --table
DT_PILOTAGE.DEMARQUE_MAG_JOUR --where "ROWNUM <= 1000" --hive-table
default.DEMARQUE_MAG_JOUR

But i have a new error :

13/06/18 15:59:33 ERROR tool.ImportTool: Encountered IOException running
import job: java.io.IOException: Hive exited with status 1
        at
org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:364)
        at
org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:314)
        at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:226)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:415)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:238)

In the Hive logs, we can read these informations :

Caused by: java.sql.SQLException: Failed to start database 'metastore_db',
see the next exception for details.
    at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
Source)
    at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
    ... 85 more
Caused by: java.sql.SQLException: Another instance of Derby may have
already booted the database /home/hduser/metastore_db.
    at
org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
Source)
    at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)
    at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source)
    at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
    ... 82 more

Thanks.

Jérôme


2013/6/18 Venkat <venkat.nr@gmail.com>

> Hi Jerome
>
> You can see the following message in your output.
>
> 13/06/18 12:05:21 WARN tool.BaseSqoopTool: It seems that you've specified
> at least one of following:
> 13/06/18 12:05:21 WARN tool.BaseSqoopTool:      --hive-home
> 13/06/18 12:05:21 WARN tool.BaseSqoopTool:      --hive-overwrite
> 13/06/18 12:05:21 WARN tool.BaseSqoopTool:      --create-hive-table
> 13/06/18 12:05:21 WARN tool.BaseSqoopTool:      --hive-table
> 13/06/18 12:05:21 WARN tool.BaseSqoopTool:      --hive-partition-key
> 13/06/18 12:05:21 WARN tool.BaseSqoopTool:      --hive-partition-value
> 13/06/18 12:05:21 WARN tool.BaseSqoopTool:      --map-column-hive
> --create-hive-table without --hive-import does not have any effect as the
> warning message showed.  So, you may want to add --hive-import to the
> command line which is the enabler of hive imports.   These additional
> options take effect only if you have enabled hive imports.
>
> THanks
>
> Venkat
>
>
> On Tue, Jun 18, 2013 at 3:36 AM, Jérôme Verdier <
> verdier.jerome66@gmail.com> wrote:
>
>> Hi Jarcec,
>>
>> Thanks for your explanations, it help me understand how Sqoop works.
>>
>> i'm trying import 1000 Rows for a quite Oracle big table which is divided
>> in partitions to keep reasonable query time.
>>
>> i am using this Sqoop script, with a query to select only the first 1000
>> rows :
>>
>> sqoop import --connect jdbc:oracle:thin:@xx.xx.xx.xx:1521/D_BI
>> --username xx --password xx --create-hive-table --query 'SELECT * FROM
>> DT_PILOTAGE.DEMARQUE_MAG_JOUR WHERE ROWNUM<1000 AND $CONDITIONS'
>> --target-dir /home/hduser --split-by DEMARQUE_MAG_JOUR.CO_SOCIETE
>> --hive-table default.DEMARQUE_MAG_JOUR
>>
>> the M/R job is working quite, but as we can see in the result below,
>> datas are not moved to Hive.
>>
>> Warning: /usr/lib/hbase does not exist! HBase imports will fail.
>> Please set $HBASE_HOME to the root of your HBase installation.
>> 13/06/18 12:05:21 WARN tool.BaseSqoopTool: Setting your password on the
>> command-line is insecure. Consider using -P instead.
>> 13/06/18 12:05:21 WARN tool.BaseSqoopTool: It seems that you've specified
>> at least one of following:
>> 13/06/18 12:05:21 WARN tool.BaseSqoopTool:      --hive-home
>> 13/06/18 12:05:21 WARN tool.BaseSqoopTool:      --hive-overwrite
>> 13/06/18 12:05:21 WARN tool.BaseSqoopTool:      --create-hive-table
>> 13/06/18 12:05:21 WARN tool.BaseSqoopTool:      --hive-table
>> 13/06/18 12:05:21 WARN tool.BaseSqoopTool:      --hive-partition-key
>> 13/06/18 12:05:21 WARN tool.BaseSqoopTool:      --hive-partition-value
>> 13/06/18 12:05:21 WARN tool.BaseSqoopTool:      --map-column-hive
>> 13/06/18 12:05:21 WARN tool.BaseSqoopTool: Without specifying parameter
>> --hive-import. Please note that
>> 13/06/18 12:05:21 WARN tool.BaseSqoopTool: those arguments will not be
>> used in this session. Either
>> 13/06/18 12:05:21 WARN tool.BaseSqoopTool: specify --hive-import to apply
>> them correctly or remove them
>> 13/06/18 12:05:21 WARN tool.BaseSqoopTool: from command line to remove
>> this warning.
>> 13/06/18 12:05:21 INFO manager.SqlManager: Using default fetchSize of 1000
>> 13/06/18 12:05:21 INFO tool.CodeGenTool: Beginning code generation
>> 13/06/18 12:05:40 INFO manager.OracleManager: Time zone has been set to
>> GMT
>> 13/06/18 12:05:40 INFO manager.SqlManager: Executing SQL statement:
>> SELECT * FROM DT_PILOTAGE.DEMARQUE_MAG_JOUR WHERE ROWNUM<1000 AND  (1 = 0)
>> 13/06/18 12:05:40 INFO manager.SqlManager: Executing SQL statement:
>> SELECT * FROM DT_PILOTAGE.DEMARQUE_MAG_JOUR WHERE ROWNUM<1000 AND  (1 = 0)
>> 13/06/18 12:05:40 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
>> /usr/local/hadoop
>> Note:
>> /tmp/sqoop-hduser/compile/b2b0decece541a7abda95580d7b1f0d2/QueryResult.java
>> uses or overrides a deprecated API.
>>
>> Note: Recompile with -Xlint:deprecation for details.
>> 13/06/18 12:05:41 INFO orm.CompilationManager: Writing jar file:
>> /tmp/sqoop-hduser/compile/b2b0decece541a7abda95580d7b1f0d2/QueryResult.jar
>> 13/06/18 12:05:41 INFO mapreduce.ImportJobBase: Beginning query import.
>> 13/06/18 12:05:42 INFO db.DataDrivenDBInputFormat: BoundingValsQuery:
>> SELECT MIN(t1.CO_SOCIETE), MAX(t1.CO_SOCIETE) FROM (SELECT * FROM
>> DT_PILOTAGE.DEMARQUE_MAG_JOUR WHERE ROWNUM<1000 AND  (1 = 1) ) t1
>> 13/06/18 12:05:42 WARN db.BigDecimalSplitter: Set BigDecimal splitSize to
>> MIN_INCREMENT
>> 13/06/18 12:05:42 INFO mapred.JobClient: Running job:
>> job_201306180922_0005
>> 13/06/18 12:05:43 INFO mapred.JobClient:  map 0% reduce 0%
>> 13/06/18 12:05:50 INFO mapred.JobClient:  map 100% reduce 0%
>> 13/06/18 12:05:51 INFO mapred.JobClient: Job complete:
>> job_201306180922_0005
>> 13/06/18 12:05:51 INFO mapred.JobClient: Counters: 18
>> 13/06/18 12:05:51 INFO mapred.JobClient:   Job Counters
>> 13/06/18 12:05:51 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=6570
>> 13/06/18 12:05:51 INFO mapred.JobClient:     Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>> 13/06/18 12:05:51 INFO mapred.JobClient:     Total time spent by all maps
>> waiting after reserving slots (ms)=0
>> 13/06/18 12:05:51 INFO mapred.JobClient:     Launched map tasks=1
>> 13/06/18 12:05:51 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0
>> 13/06/18 12:05:51 INFO mapred.JobClient:   File Output Format Counters
>> 13/06/18 12:05:52 INFO mapred.JobClient:     Bytes Written=174729
>> 13/06/18 12:05:52 INFO mapred.JobClient:   FileSystemCounters
>> 13/06/18 12:05:52 INFO mapred.JobClient:     HDFS_BYTES_READ=147
>> 13/06/18 12:05:52 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=61242
>> 13/06/18 12:05:52 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=174729
>> 13/06/18 12:05:52 INFO mapred.JobClient:   File Input Format Counters
>> 13/06/18 12:05:52 INFO mapred.JobClient:     Bytes Read=0
>> 13/06/18 12:05:52 INFO mapred.JobClient:   Map-Reduce Framework
>> 13/06/18 12:05:52 INFO mapred.JobClient:     Map input records=999
>> 13/06/18 12:05:52 INFO mapred.JobClient:     Physical memory (bytes)
>> snapshot=43872256
>> 13/06/18 12:05:52 INFO mapred.JobClient:     Spilled Records=0
>> 13/06/18 12:05:52 INFO mapred.JobClient:     CPU time spent (ms)=830
>> 13/06/18 12:05:52 INFO mapred.JobClient:     Total committed heap usage
>> (bytes)=16252928
>> 13/06/18 12:05:52 INFO mapred.JobClient:     Virtual memory (bytes)
>> snapshot=373719040
>> 13/06/18 12:05:52 INFO mapred.JobClient:     Map output records=999
>> 13/06/18 12:05:52 INFO mapred.JobClient:     SPLIT_RAW_BYTES=147
>> 13/06/18 12:05:52 INFO mapreduce.ImportJobBase: Transferred 170,6338 KB
>> in 10,6221 seconds (16,0641 KB/sec)
>> 13/06/18 12:05:52 INFO mapreduce.ImportJobBase: Retrieved 999 records.
>>
>> Why SQOOP doesn't move data to Hive, is there a problem with partitioned
>> table ??
>>
>> Thanks.
>>
>>
>>
>>
>>
>>
>> 2013/6/17 Jarek Jarcec Cecho <jarcec@apache.org>
>>
>>> Hi Jerome,
>>> there are many ways how to achieve that. Considering that you are doing
>>> table based import, you can take advantage of --where parameter to specify
>>> any arbitrary condition to choose the proper 1000 rows. Another option
>>> would be use query based import instead of table, where you can specify
>>> entire query that should be imported.
>>>
>>> Jarcec
>>>
>>> On Mon, Jun 17, 2013 at 05:59:45PM +0200, Jérôme Verdier wrote:
>>> > I have the solution to my problem.
>>> >
>>> > I resolved it by using the option --hive-table default.VENTES_EAN in my
>>> > sqoop script.
>>> >
>>> > But, i have another question :
>>> >
>>> > i want to import from oracle to hive only 1000 rows from my table, how
>>> we
>>> > can do this using sqoop ?
>>> >
>>> > Thanks.
>>> >
>>> > --
>>> > Jérôme
>>> >
>>> >
>>> > 2013/6/17 Jérôme Verdier <verdier.jerome66@gmail.com>
>>> >
>>> > > Hi Jarcec,
>>> > >
>>> > > Thanks for your answer, you're always very helpful =)
>>> > >
>>> > > i think that my Hive installation is OK, i can connect to hive server
>>> > > through my laptop using JDBC and Squirrel SQL.
>>> > >
>>> > > Here are the hive logs :
>>> > >
>>> > > 2013-06-17 17:16:40,452 WARN  conf.HiveConf
>>> (HiveConf.java:<clinit>(75)) -
>>> > > hive-site.xml not found on CLASSPATH
>>> > > 2013-06-17 17:20:51,228 WARN  conf.HiveConf
>>> (HiveConf.java:<clinit>(75)) -
>>> > > hive-site.xml not found on CLASSPATH
>>> > > 2013-06-17 17:20:53,296 ERROR DataNucleus.Plugin
>>> > > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core"
>>> requires
>>> > > "org.eclipse.core.resources" but it cannot be resolved.
>>> > > 2013-06-17 17:20:53,296 ERROR DataNucleus.Plugin
>>> > > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core"
>>> requires
>>> > > "org.eclipse.core.resources" but it cannot be resolved.
>>> > > 2013-06-17 17:20:53,297 ERROR DataNucleus.Plugin
>>> > > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core"
>>> requires
>>> > > "org.eclipse.core.runtime" but it cannot be resolved.
>>> > > 2013-06-17 17:20:53,297 ERROR DataNucleus.Plugin
>>> > > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core"
>>> requires
>>> > > "org.eclipse.core.runtime" but it cannot be resolved.
>>> > > 2013-06-17 17:20:53,297 ERROR DataNucleus.Plugin
>>> > > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core"
>>> requires
>>> > > "org.eclipse.text" but it cannot be resolved.
>>> > > 2013-06-17 17:20:53,297 ERROR DataNucleus.Plugin
>>> > > (Log4JLogger.java:error(115)) - Bundle "org.eclipse.jdt.core"
>>> requires
>>> > > "org.eclipse.text" but it cannot be resolved.
>>> > > 2013-06-17 17:20:58,645 ERROR exec.Task
>>> > > (SessionState.java:printError(401)) - FAILED: Error in metadata:
>>> > > InvalidObjectException(message:There is no database named themis)
>>> > > org.apache.hadoop.hive.ql.metadata.HiveException:
>>> > > InvalidObjectException(message:There is no database named themis)
>>> > >     at
>>> org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:576)
>>> > >     at
>>> > > org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3698)
>>> > >     at
>>> org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:253)
>>> > >     at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
>>> > >     at
>>> > >
>>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
>>> > >     at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1336)
>>> > >     at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1122)
>>> > >     at org.apache.hadoop.hive.ql.Driver.run(Driver.java:935)
>>> > >     at
>>> > >
>>> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
>>> > >     at
>>> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
>>> > >     at
>>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
>>> > >     at
>>> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:347)
>>> > >     at
>>> > >
>>> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:445)
>>> > >     at
>>> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:455)
>>> > >     at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:711)
>>> > >     at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
>>> > >     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > >     at
>>> > >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > >     at
>>> > >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > >     at java.lang.reflect.Method.invoke(Method.java:597)
>>> > >     at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>> > > Caused by: InvalidObjectException(message:There is no database named
>>> > > themis)
>>> > >     at
>>> > >
>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table(HiveMetaStore.java:1091)
>>> > >     at
>>> > >
>>> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table(HiveMetaStore.java:1070)
>>> > >     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > >     at
>>> > >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > >     at
>>> > >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > >     at java.lang.reflect.Method.invoke(Method.java:597)
>>> > >     at
>>> > >
>>> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
>>> > >     at $Proxy8.create_table(Unknown Source)
>>> > >     at
>>> > >
>>> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:432)
>>> > >     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > >     at
>>> > >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > >     at
>>> > >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > >     at java.lang.reflect.Method.invoke(Method.java:597)
>>> > >     at
>>> > >
>>> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:74)
>>> > >     at $Proxy9.createTable(Unknown Source)
>>> > >     at
>>> org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:570)
>>> > >     ... 20 more
>>> > >
>>> > > 2013-06-17 17:20:58,651 ERROR ql.Driver
>>> > > (SessionState.java:printError(401)) - FAILED: Execution Error,
>>> return code
>>> > > 1 from org.apache.hadoop.hive.ql.exec.DDLTask
>>> > >
>>> > > I try to import the oracle table VENTES_EAN located in the schema
>>> THEMIS
>>> > > ==> THEMIS.VENTES_EAN
>>> > >
>>> > > but as we can see on the logs, Hive is thinking that i want to import
>>> > > VENTES_EAN into the THEMIS databases, but i have only one database
:
>>> > > default.
>>> > >
>>> > > is there a Hive configuration problem here ?
>>> > >
>>> > > Thanks.
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > 2013/6/17 Jarek Jarcec Cecho <jarcec@apache.org>
>>> > >
>>> > >> Hi Jerome,
>>> > >> Hive import in Sqoop is done in two phases. The first phase will
>>> transfer
>>> > >> the data from your Oracle database to HDFS as would normal non
hive
>>> import.
>>> > >> Subsequently in the second phase Sqoop will invoke Hive to perform
>>> LOAD
>>> > >> DATA statement to move imported data into Hive. In you first Sqoop
>>> > >> invocation the first step obviously finished correctly, however
the
>>> second
>>> > >> phase has failed. This is the reason why the second Sqoop
>>> invocation is
>>> > >> failing as the intermediate directory between the two phases still
>>> exists.
>>> > >> You can unblock that by removing the directory using HDFS command,
>>> for
>>> > >> example:
>>> > >>
>>> > >>   hadoop dfs -rmr KPI.ENTITE
>>> > >>
>>> > >> The second phase seems to be failing for you on following exception:
>>> > >>
>>> > >> > java.lang.RuntimeException: Unable to instantiate
>>> > >>
>>> > >> I would therefore suggest to take a look into Hive logs
>>> > >> (/tmp/$USER/hive.log if I'm not mistaken) to see if there would
be
>>> more
>>> > >> details about the instantiation failure. Could you also verify
that
>>> your
>>> > >> Hive installation is configured correctly?
>>> > >>
>>> > >> Jarcec
>>> > >>
>>> > >> On Mon, Jun 17, 2013 at 03:46:28PM +0200, Jérôme Verdier wrote:
>>> > >> > Hi,
>>> > >> >
>>> > >> > I'm try to import various tables from Oracle to Hive using
Sqoop,
>>> but, i
>>> > >> > have some errors that i don't understand.
>>> > >> >
>>> > >> > Here is my query :
>>> > >> >
>>> > >> > sqoop import --connect jdbc:oracle:thin:@my.db.server:1521/xx
>>> > >> --username
>>> > >> > user --password password --create-hive-table --hive-import
--table
>>> > >> > schema.table_xx
>>> > >> >
>>> > >> > the first error  is this one :
>>> > >> >
>>> > >> > Please set $HBASE_HOME to the root of your HBase installation.
>>> > >> > 13/06/17 15:36:40 WARN tool.BaseSqoopTool: Setting your password
>>> on the
>>> > >> > command-line is insecure. Consider using -P instead.
>>> > >> > 13/06/17 15:36:40 INFO tool.BaseSqoopTool: Using Hive-specific
>>> > >> delimiters
>>> > >> > for output. You can override
>>> > >> > 13/06/17 15:36:40 INFO tool.BaseSqoopTool: delimiters with
>>> > >> > --fields-terminated-by, etc.
>>> > >> > 13/06/17 15:36:40 INFO manager.SqlManager: Using default
>>> fetchSize of
>>> > >> 1000
>>> > >> > 13/06/17 15:36:40 INFO tool.CodeGenTool: Beginning code generation
>>> > >> > 13/06/17 15:36:41 INFO manager.OracleManager: Time zone has
been
>>> set to
>>> > >> GMT
>>> > >> > 13/06/17 15:36:41 INFO manager.SqlManager: Executing SQL
>>> statement:
>>> > >> SELECT
>>> > >> > t.* FROM KPI.ENTITE t WHERE 1=0
>>> > >> > 13/06/17 15:36:41 INFO orm.CompilationManager: HADOOP_MAPRED_HOME
>>> is
>>> > >> > /usr/local/hadoop
>>> > >> > Note:
>>> > >> >
>>> > >>
>>> /tmp/sqoop-hduser/compile/85a6dcface4ca6ca28091ed383edce2e/KPI_ENTITE.java
>>> > >> > uses or overrides a deprecated API.
>>> > >> > Note: Recompile with -Xlint:deprecation for details.
>>> > >> > 13/06/17 15:36:42 INFO orm.CompilationManager: Writing jar
file:
>>> > >> >
>>> > >>
>>> /tmp/sqoop-hduser/compile/85a6dcface4ca6ca28091ed383edce2e/KPI.ENTITE.jar
>>> > >> > 13/06/17 15:36:42 INFO manager.OracleManager: Time zone has
been
>>> set to
>>> > >> GMT
>>> > >> > 13/06/17 15:36:42 WARN manager.OracleManager: The table KPI.ENTITE
>>> > >> contains
>>> > >> > a multi-column primary key. Sqoop will default to the column
>>> CO_SOCIETE
>>> > >> > only for this job.
>>> > >> > 13/06/17 15:36:42 INFO manager.OracleManager: Time zone has
been
>>> set to
>>> > >> GMT
>>> > >> > 13/06/17 15:36:42 WARN manager.OracleManager: The table KPI.ENTITE
>>> > >> contains
>>> > >> > a multi-column primary key. Sqoop will default to the column
>>> CO_SOCIETE
>>> > >> > only for this job.
>>> > >> > 13/06/17 15:36:42 INFO mapreduce.ImportJobBase: Beginning
import
>>> of
>>> > >> > KPI.ENTITE
>>> > >> > 13/06/17 15:36:42 INFO manager.OracleManager: Time zone has
been
>>> set to
>>> > >> GMT
>>> > >> > 13/06/17 15:36:44 INFO db.DataDrivenDBInputFormat:
>>> BoundingValsQuery:
>>> > >> > SELECT MIN(CO_SOCIETE), MAX(CO_SOCIETE) FROM KPI.ENTITE
>>> > >> > 13/06/17 15:36:44 INFO mapred.JobClient: Running job:
>>> > >> job_201306171456_0005
>>> > >> > 13/06/17 15:36:45 INFO mapred.JobClient:  map 0% reduce 0%
>>> > >> > 13/06/17 15:36:56 INFO mapred.JobClient:  map 25% reduce 0%
>>> > >> > 13/06/17 15:37:40 INFO mapred.JobClient:  map 50% reduce 0%
>>> > >> > 13/06/17 15:38:00 INFO mapred.JobClient:  map 75% reduce 0%
>>> > >> > 13/06/17 15:38:08 INFO mapred.JobClient:  map 100% reduce
0%
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient: Job complete:
>>> > >> job_201306171456_0005
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient: Counters: 18
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:   Job Counters
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:
>>> SLOTS_MILLIS_MAPS=151932
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:     Total time spent
by
>>> all
>>> > >> > reduces waiting after reserving slots (ms)=0
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:     Total time spent
by
>>> all
>>> > >> maps
>>> > >> > waiting after reserving slots (ms)=0
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:     Launched map
tasks=4
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:
>>> SLOTS_MILLIS_REDUCES=0
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:   File Output Format
>>> Counters
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:     Bytes Written=26648
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:   FileSystemCounters
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:     HDFS_BYTES_READ=462
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:
>>> FILE_BYTES_WRITTEN=244596
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:
>>> HDFS_BYTES_WRITTEN=26648
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:   File Input Format
>>> Counters
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:     Bytes Read=0
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:   Map-Reduce Framework
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:     Map input records=339
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:     Physical memory
>>> (bytes)
>>> > >> > snapshot=171716608
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:     Spilled Records=0
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:     CPU time spent
>>> (ms)=3920
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:     Total committed
heap
>>> usage
>>> > >> > (bytes)=65011712
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:     Virtual memory
>>> (bytes)
>>> > >> > snapshot=1492393984
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:     Map output
>>> records=339
>>> > >> > 13/06/17 15:38:09 INFO mapred.JobClient:     SPLIT_RAW_BYTES=462
>>> > >> > 13/06/17 15:38:09 INFO mapreduce.ImportJobBase: Transferred
>>> 26,0234 KB
>>> > >> in
>>> > >> > 86,6921 seconds (307,3869 bytes/sec)
>>> > >> > 13/06/17 15:38:09 INFO mapreduce.ImportJobBase: Retrieved
339
>>> records.
>>> > >> > 13/06/17 15:38:09 INFO manager.OracleManager: Time zone has
been
>>> set to
>>> > >> GMT
>>> > >> > 13/06/17 15:38:09 INFO manager.SqlManager: Executing SQL
>>> statement:
>>> > >> SELECT
>>> > >> > t.* FROM KPI.ENTITE t WHERE 1=0
>>> > >> > 13/06/17 15:38:09 WARN hive.TableDefWriter: Column CO_SOCIETE
had
>>> to be
>>> > >> > cast to a less precise type in Hive
>>> > >> > 13/06/17 15:38:09 INFO hive.HiveImport: Removing temporary
files
>>> from
>>> > >> > import process:
>>> hdfs://localhost:54310/user/hduser/KPI.ENTITE/_logs
>>> > >> > 13/06/17 15:38:09 INFO hive.HiveImport: Loading uploaded data
>>> into Hive
>>> > >> > 13/06/17 15:38:11 INFO hive.HiveImport: WARNING:
>>> > >> > org.apache.hadoop.metrics.jvm.EventCounter is deprecated.
Please
>>> use
>>> > >> > org.apache.hadoop.log.metrics.EventCounter in all the
>>> log4j.properties
>>> > >> > files.
>>> > >> > 13/06/17 15:38:12 INFO hive.HiveImport: Logging initialized
using
>>> > >> > configuration in
>>> > >> >
>>> > >>
>>> jar:file:/usr/local/hive/lib/hive-common-0.10.0.jar!/hive-log4j.properties
>>> > >> > 13/06/17 15:38:12 INFO hive.HiveImport: Hive history
>>> > >> > file=/tmp/hduser/hive_job_log_hduser_201306171538_49452696.txt
>>> > >> > 13/06/17 15:38:14 INFO hive.HiveImport: FAILED: Error in metadata:
>>> > >> > java.lang.RuntimeException: Unable to instantiate
>>> > >> > org.apache.hadoop.hive.metastore.HiveMetaStoreClient
>>> > >> > 13/06/17 15:38:14 INFO hive.HiveImport: FAILED: Execution
Error,
>>> return
>>> > >> > code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
>>> > >> > 13/06/17 15:38:14 ERROR tool.ImportTool: Encountered IOException
>>> running
>>> > >> > import job: java.io.IOException: Hive exited with status 1
>>> > >> >         at
>>> > >> >
>>> > >>
>>> org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:364)
>>> > >> >         at
>>> > >> >
>>> org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:314)
>>> > >> >         at
>>> > >> org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:226)
>>> > >> >         at
>>> > >> org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:415)
>>> > >> >         at
>>> org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
>>> > >> >         at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
>>> > >> >         at
>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>> > >> >         at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
>>> > >> >         at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
>>> > >> >         at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
>>> > >> >         at org.apache.sqoop.Sqoop.main(Sqoop.java:238)
>>> > >> >
>>> > >> > I don't understand because the M/R job is completed, but after
>>> this, it
>>> > >> > give me an I/O error.
>>> > >> >
>>> > >> > when i try a SHOW TABLES on hive, i have no tables.
>>> > >> >
>>> > >> > but, when i retry the SQOOP script, i get this error :
>>> > >> >
>>> > >> > Warning: /usr/lib/hbase does not exist! HBase imports will
fail.
>>> > >> > Please set $HBASE_HOME to the root of your HBase installation.
>>> > >> > 13/06/17 15:41:51 WARN tool.BaseSqoopTool: Setting your password
>>> on the
>>> > >> > command-line is insecure. Consider using -P instead.
>>> > >> > 13/06/17 15:41:51 INFO tool.BaseSqoopTool: Using Hive-specific
>>> > >> delimiters
>>> > >> > for output. You can override
>>> > >> > 13/06/17 15:41:51 INFO tool.BaseSqoopTool: delimiters with
>>> > >> > --fields-terminated-by, etc.
>>> > >> > 13/06/17 15:41:51 INFO manager.SqlManager: Using default
>>> fetchSize of
>>> > >> 1000
>>> > >> > 13/06/17 15:41:51 INFO tool.CodeGenTool: Beginning code generation
>>> > >> > 13/06/17 15:42:15 INFO manager.OracleManager: Time zone has
been
>>> set to
>>> > >> GMT
>>> > >> > 13/06/17 15:42:15 INFO manager.SqlManager: Executing SQL
>>> statement:
>>> > >> SELECT
>>> > >> > t.* FROM KPI.ENTITE t WHERE 1=0
>>> > >> > 13/06/17 15:42:15 INFO orm.CompilationManager: HADOOP_MAPRED_HOME
>>> is
>>> > >> > /usr/local/hadoop
>>> > >> > Note:
>>> > >> >
>>> > >>
>>> /tmp/sqoop-hduser/compile/10cd05e9146a878654b1155df5be7765/KPI_ENTITE.java
>>> > >> > uses or overrides a deprecated API.
>>> > >> > Note: Recompile with -Xlint:deprecation for details.
>>> > >> > 13/06/17 15:42:16 INFO orm.CompilationManager: Writing jar
file:
>>> > >> >
>>> > >>
>>> /tmp/sqoop-hduser/compile/10cd05e9146a878654b1155df5be7765/KPI.ENTITE.jar
>>> > >> > 13/06/17 15:42:16 INFO manager.OracleManager: Time zone has
been
>>> set to
>>> > >> GMT
>>> > >> > 13/06/17 15:42:16 WARN manager.OracleManager: The table KPI.ENTITE
>>> > >> contains
>>> > >> > a multi-column primary key. Sqoop will default to the column
>>> CO_SOCIETE
>>> > >> > only for this job.
>>> > >> > 13/06/17 15:42:16 INFO manager.OracleManager: Time zone has
been
>>> set to
>>> > >> GMT
>>> > >> > 13/06/17 15:42:16 WARN manager.OracleManager: The table KPI.ENTITE
>>> > >> contains
>>> > >> > a multi-column primary key. Sqoop will default to the column
>>> CO_SOCIETE
>>> > >> > only for this job.
>>> > >> > 13/06/17 15:42:16 INFO mapreduce.ImportJobBase: Beginning
import
>>> of
>>> > >> > KPI.ENTITE
>>> > >> > 13/06/17 15:42:16 INFO manager.OracleManager: Time zone has
been
>>> set to
>>> > >> GMT
>>> > >> > 13/06/17 15:42:17 INFO mapred.JobClient: Cleaning up the staging
>>> area
>>> > >> >
>>> > >>
>>> hdfs://localhost:54310/app/hadoop/tmp/mapred/staging/hduser/.staging/job_201306171456_0006
>>> > >> > 13/06/17 15:42:17 ERROR security.UserGroupInformation:
>>> > >> > PriviledgedActionException as:hduser
>>> > >> > cause:org.apache.hadoop.mapred.FileAlreadyExistsException:
Output
>>> > >> directory
>>> > >> > KPI.ENTITE already exists
>>> > >> > 13/06/17 15:42:17 ERROR tool.ImportTool: Encountered IOException
>>> running
>>> > >> > import job: org.apache.hadoop.mapred.FileAlreadyExistsException:
>>> Output
>>> > >> > directory KPI.ENTITE already exists
>>> > >> >         at
>>> > >> >
>>> > >>
>>> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:137)
>>> > >> >         at
>>> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:949)
>>> > >> >         at
>>> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:912)
>>> > >> >         at java.security.AccessController.doPrivileged(Native
>>> Method)
>>> > >> >         at javax.security.auth.Subject.doAs(Subject.java:396)
>>> > >> >         at
>>> > >> >
>>> > >>
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>>> > >> >         at
>>> > >> >
>>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:912)
>>> > >> >         at org.apache.hadoop.mapreduce.Job.submit(Job.java:500)
>>> > >> >         at
>>> > >> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:530)
>>> > >> >         at
>>> > >> >
>>> > >>
>>> org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:173)
>>> > >> >         at
>>> > >> >
>>> org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:151)
>>> > >> >         at
>>> > >> >
>>> > >>
>>> org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:221)
>>> > >> >         at
>>> > >> >
>>> org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:545)
>>> > >> >         at
>>> > >> >
>>> > >>
>>> org.apache.sqoop.manager.OracleManager.importTable(OracleManager.java:380)
>>> > >> >         at
>>> > >> org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:403)
>>> > >> >         at
>>> org.apache.sqoop.tool.ImportTool.run(ImportTool.java:476)
>>> > >> >         at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
>>> > >> >         at
>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>> > >> >         at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
>>> > >> >         at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
>>> > >> >         at org.apache.sqoop.Sqoop.runTool(Sqoop.java:22
>>> > >> >
>>> > >> > The output explains that the output already exists.
>>> > >> >
>>> > >> > But, Hive command SHOW TABLES give me zero tables !
>>> > >> >
>>> > >> > Thanks for your help ;-)
>>> > >> >
>>> > >> >
>>> > >> > --
>>> > >> > Jérôme
>>> > >>
>>> > >
>>> > >
>>>
>>
>>
>>
>> --
>> *Jérôme VERDIER*
>> 06.72.19.17.31
>> verdier.jerome66@gmail.com
>>
>>
>
>
> --
> Regards
>
> Venkat
>

Mime
View raw message