sqoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <zoltan.fe...@thomsonreuters.com>
Subject Sqoop import fails on permissions over /user directory ?
Date Tue, 10 Mar 2015 16:53:39 GMT
Hi,
We are running Sqoop version: 1.4.4-cdh5.1.3 and I'm trying to import from Oracle into Hive
using Sqoop, but I receive permission denied on the "/user" folder.
I don't see what Sqoop is even trying to write into the /user folder. My user ("myuser") has
no write access to the /user folder (that cannot be changed), but it has its own /user/myuser
folder and I don't see what sqoop is trying to write in /user folder.
Any way to tell sqoop not to write anything to "/user"?

Thanks,
Zoltan


Import command I use:
sudo HADOOP_USER_NAME=myuser sqoop import \
--connect jdbc:oracle:thin:@xxx.com:1521/xxx.com \
--username myuser \
--password mypassword\
--table TEST_TABLE \
--split-by TEST_ID \
--hive-import \
--hive-overwrite \
--hive-table myuser.test_table \
--warehouse-dir /user/hive/warehouse/mydb.db


Error:
hive.HiveImport: FAILED: RuntimeException org.apache.hadoop.security.AccessControlException:
Permission denied: user=myuser, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x


Complete log:
Warning: /usr/lib/sqoop/../hive-hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
15/03/10 12:39:15 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4-cdh5.1.3
15/03/10 12:39:15 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure.
Consider using -P instead.
15/03/10 12:39:15 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You
can override
15/03/10 12:39:15 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
15/03/10 12:39:15 WARN tool.BaseSqoopTool: It seems that you're doing hive import directly
into default
15/03/10 12:39:15 WARN tool.BaseSqoopTool: hive warehouse directory which is not supported.
Sqoop is
15/03/10 12:39:15 WARN tool.BaseSqoopTool: firstly importing data into separate directory
and then
15/03/10 12:39:15 WARN tool.BaseSqoopTool: inserting data into hive. Please consider removing
15/03/10 12:39:15 WARN tool.BaseSqoopTool: --target-dir or --warehouse-dir into /user/hive/warehouse
in
15/03/10 12:39:15 WARN tool.BaseSqoopTool: case that you will detect any issues.
15/03/10 12:39:15 INFO manager.SqlManager: Using default fetchSize of 1000
15/03/10 12:39:15 INFO tool.CodeGenTool: Beginning code generation
15/03/10 12:39:52 INFO manager.OracleManager: Time zone has been set to GMT
15/03/10 12:39:52 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM TEST_TABLE
t WHERE 1=0
15/03/10 12:39:52 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
Note: /tmp/sqoop-root/compile/cfadbffae50670e6e0cacc4c94c93824/TEST_TABLE.java uses or overrides
a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
15/03/10 12:39:54 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/cfadbffae50670e6e0cacc4c94c93824/TEST_TABLE.jar
15/03/10 12:39:54 INFO mapreduce.ImportJobBase: Beginning import of TEST_TABLE
15/03/10 12:39:54 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
15/03/10 12:39:54 INFO manager.OracleManager: Time zone has been set to GMT
15/03/10 12:39:55 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead,
use mapreduce.job.maps
15/03/10 12:39:55 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm726
15/03/10 12:39:57 INFO db.DBInputFormat: Using read commited transaction isolation
15/03/10 12:39:57 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(TEST_ID),
MAX(TEST_ID) FROM TEST_TABLE
15/03/10 12:39:57 INFO mapreduce.JobSubmitter: number of splits:4
15/03/10 12:39:57 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1424828239953_41912
15/03/10 12:39:58 INFO impl.YarnClientImpl: Submitted application application_1424828239953_41912
15/03/10 12:39:58 INFO mapreduce.Job: The url to track the job: http://xxxx.com:8088/proxy/application_1424828239953_41912/
15/03/10 12:39:58 INFO mapreduce.Job: Running job: job_1424828239953_41912
15/03/10 12:40:11 INFO mapreduce.Job: Job job_1424828239953_41912 running in uber mode : false
15/03/10 12:40:11 INFO mapreduce.Job:  map 0% reduce 0%
15/03/10 12:40:22 INFO mapreduce.Job:  map 75% reduce 0%
15/03/10 12:40:23 INFO mapreduce.Job:  map 100% reduce 0%
15/03/10 12:40:23 INFO mapreduce.Job: Job job_1424828239953_41912 completed successfully
15/03/10 12:40:23 INFO mapreduce.Job: Counters: 30
                File System Counters
                                FILE: Number of bytes read=0
                                FILE: Number of bytes written=458289
                                FILE: Number of read operations=0
                                FILE: Number of large read operations=0
                                FILE: Number of write operations=0
                                HDFS: Number of bytes read=490
                                HDFS: Number of bytes written=16221504
                                HDFS: Number of read operations=16
                                HDFS: Number of large read operations=0
                                HDFS: Number of write operations=8
                Job Counters
                                Launched map tasks=4
                                Other local map tasks=4
                                Total time spent by all maps in occupied slots (ms)=0
                                Total time spent by all reduces in occupied slots (ms)=0
                                Total time spent by all map tasks (ms)=25947
                                Total vcore-seconds taken by all map tasks=25947
                                Total megabyte-seconds taken by all map tasks=106278912
                Map-Reduce Framework
                                Map input records=51500
                                Map output records=51500
                                Input split bytes=490
                                Spilled Records=0
                                Failed Shuffles=0
                                Merged Map outputs=0
                                GC time elapsed (ms)=127
                                CPU time spent (ms)=15480
                                Physical memory (bytes) snapshot=2619047936
                                Virtual memory (bytes) snapshot=17864171520
                                Total committed heap usage (bytes)=8233418752
                File Input Format Counters
                                Bytes Read=0
                File Output Format Counters
                                Bytes Written=16221504
15/03/10 12:40:23 INFO mapreduce.ImportJobBase: Transferred 15.47 MB in 28.572 seconds (554.4347
KB/sec)
15/03/10 12:40:23 INFO mapreduce.ImportJobBase: Retrieved 51500 records.
15/03/10 12:40:23 INFO manager.OracleManager: Time zone has been set to GMT
15/03/10 12:40:23 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM TEST_TABLE
t WHERE 1=0
15/03/10 12:40:23 WARN hive.TableDefWriter: Column FEATURE_ID had to be cast to a less precise
type in Hive
15/03/10 12:40:23 WARN hive.TableDefWriter: Column DATE_FIRST_SEEN had to be cast to a less
precise type in Hive
15/03/10 12:40:23 WARN hive.TableDefWriter: Column DATE_LAST_SEEN had to be cast to a less
precise type in Hive
15/03/10 12:40:23 WARN hive.TableDefWriter: Column DATE_EDITED had to be cast to a less precise
type in Hive
15/03/10 12:40:23 WARN hive.TableDefWriter: Column CREATED_DT had to be cast to a less precise
type in Hive
15/03/10 12:40:23 WARN hive.TableDefWriter: Column UPDATED_DT had to be cast to a less precise
type in Hive
15/03/10 12:40:23 INFO hive.HiveImport: Loading uploaded data into Hive
15/03/10 12:40:23 INFO hive.HiveImport: /usr/lib/hive/conf/hive-env.sh: line 5: /usr/lib/hive/lib/hive-hbase-handler-0.12.0-cdh5.1.3.jar,/usr/lib/hbase/hbase-hadoop2-compat.jar,/usr/lib/hbase/lib/htrace-core.jar,/usr/lib/hbase/lib/htrace-core-2.04.jar,/usr/lib/hbase/hbase-client.jar,/usr/lib/hbase/hbase-hadoop-compat.jar,/usr/lib/hbase/hbase-protocol.jar,/usr/lib/hbase/hbase-common.jar,/usr/lib/hbase/hbase-server.jar:
No such file or directory
15/03/10 12:40:25 INFO hive.HiveImport: 15/03/10 12:40:25 INFO Configuration.deprecation:
mapred.input.dir.recursive is deprecated. Instead, use mapreduce.input.fileinputformat.input.dir.recursive
15/03/10 12:40:25 INFO hive.HiveImport: 15/03/10 12:40:25 INFO Configuration.deprecation:
mapred.max.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.maxsize
15/03/10 12:40:25 INFO hive.HiveImport: 15/03/10 12:40:25 INFO Configuration.deprecation:
mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
15/03/10 12:40:25 INFO hive.HiveImport: 15/03/10 12:40:25 INFO Configuration.deprecation:
mapred.min.split.size.per.rack is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.rack
15/03/10 12:40:25 INFO hive.HiveImport: 15/03/10 12:40:25 INFO Configuration.deprecation:
mapred.min.split.size.per.node is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize.per.node
15/03/10 12:40:25 INFO hive.HiveImport: 15/03/10 12:40:25 INFO Configuration.deprecation:
mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
15/03/10 12:40:25 INFO hive.HiveImport: 15/03/10 12:40:25 INFO Configuration.deprecation:
mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
15/03/10 12:40:26 INFO hive.HiveImport: 15/03/10 12:40:26 WARN conf.HiveConf: DEPRECATED:
Configuration property hive.metastore.local no longer has any effect. Make sure to provide
a valid value for hive.metastore.uris if you are connecting to a remote metastore.
15/03/10 12:40:26 INFO hive.HiveImport:
15/03/10 12:40:26 INFO hive.HiveImport: Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-0.12.0-cdh5.1.3.jar!/hive-log4j.properties
15/03/10 12:40:27 INFO hive.HiveImport: OK
15/03/10 12:40:27 INFO hive.HiveImport: Time taken: 0.856 seconds
15/03/10 12:40:28 INFO hive.HiveImport: FAILED: RuntimeException org.apache.hadoop.security.AccessControlException:
Permission denied: user=myuser, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
15/03/10 12:40:28 INFO hive.HiveImport:              at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:271)
15/03/10 12:40:28 INFO hive.HiveImport:              at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:257)
15/03/10 12:40:28 INFO hive.HiveImport:              at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:238)
15/03/10 12:40:28 INFO hive.HiveImport:              at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:179)
15/03/10 12:40:28 INFO hive.HiveImport:              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5607)
15/03/10 12:40:28 INFO hive.HiveImport:              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5589)
15/03/10 12:40:28 INFO hive.HiveImport:              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5563)
15/03/10 12:40:28 INFO hive.HiveImport:              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3685)
15/03/10 12:40:28 INFO hive.HiveImport:              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3655)
15/03/10 12:40:28 INFO hive.HiveImport:              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3629)
15/03/10 12:40:28 INFO hive.HiveImport:              at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:741)
15/03/10 12:40:28 INFO hive.HiveImport:              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)
15/03/10 12:40:28 INFO hive.HiveImport:              at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
15/03/10 12:40:28 INFO hive.HiveImport:              at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
15/03/10 12:40:28 INFO hive.HiveImport:              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
15/03/10 12:40:28 INFO hive.HiveImport:              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
15/03/10 12:40:28 INFO hive.HiveImport:              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
15/03/10 12:40:28 INFO hive.HiveImport:              at java.security.AccessController.doPrivileged(Native
Method)
15/03/10 12:40:28 INFO hive.HiveImport:              at javax.security.auth.Subject.doAs(Subject.java:415)
15/03/10 12:40:28 INFO hive.HiveImport:              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
15/03/10 12:40:28 INFO hive.HiveImport:              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
15/03/10 12:40:28 INFO hive.HiveImport:
15/03/10 12:40:28 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException:
Hive exited with status 64
                at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:385)
                at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:335)
                at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:239)
                at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:425)
                at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:506)
                at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
                at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
                at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
                at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
                at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
                at org.apache.sqoop.Sqoop.main(Sqoop.java:240)

Mime
View raw message