sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jarek Cecho <jar...@apache.org>
Subject Re: Review Request 41971: Sqoop2: Enable kerberos for HadoopMiniCluster
Date Wed, 03 Feb 2016 23:07:12 GMT


> On Jan. 6, 2016, 4:07 p.m., Jarek Cecho wrote:
> > test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java, line 74
> > <https://reviews.apache.org/r/41971/diff/1/?file=1183938#file1183938line74>
> >
> >     Why do we need to change the directory to /tmp/ ?
> >     
> >     Aren't we by mistake using this for local path somewhere?
> 
> Dian Fu wrote:
>     In kerberos enabled cluster, we will get exceptions like this with "/mapreduce-job-io":
>     ```
>     2016-01-07 10:00:46,381 INFO  [org.apache.hadoop.hdfs.server.namenode.FSNamesystem$DefaultAuditLogger.logAuditMessage(FSNamesystem.java:9332)]
allowed=false    ugi=sqoopclient (auth:TOKEN) via hadoop/bdpe20.sh.intel.com@EXAMPLE.COM (auth:TOKEN)
   ip=/127.0.0.1   cmd=create      src=/mapreduce-job-io/org.apache.sqoop.integration.connector.hdfs.AppendModeTest/test/.577fcfcd-6ef8-491c-ad6c-c6f4983dfdf0/22200b07-c11b-4c39-b0f3-c8236b34f44e.txt
   dst=null        perm=null       proto=rpc
>     2016-01-07 10:00:46,381 DEBUG [org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:634)]
Served: create queueTime= 1 procesingTime= 1 exception= AccessControlException
>     2016-01-07 10:00:46,381 DEBUG [org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1632)]
PrivilegedActionException as:sqoopclient (auth:TOKEN) via hadoop/bdpe20.sh.intel.com@EXAMPLE.COM
(auth:TOKEN) cause:org.apache.hadoop.security.AccessControlException: Permission denied: user=sqoopclient,
access=WRITE, inode="/":root:supergroup:drwxr-xr-x
>     ```
>     This reason to this exception is that MR job tries to create directory "/mapreduce-job-io"
with user **sqoopclient**. But only superuser is allowed to create directory under "/". While
I found a better way to handle this: just create directory "mapreduce-job-io" with superuser
and changes its permission to "0777" during cluster initial phase.

Got it, I'm fine with /tmp/ then :)


- Jarek


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41971/#review113083
-----------------------------------------------------------


On Jan. 29, 2016, 8:46 a.m., Dian Fu wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41971/
> -----------------------------------------------------------
> 
> (Updated Jan. 29, 2016, 8:46 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-2744
>     https://issues.apache.org/jira/browse/SQOOP-2744
> 
> 
> Repository: sqoop-sqoop2
> 
> 
> Description
> -------
> 
> The aim of this JIRA is to enable kerberos for HadoopMiniCluster.
> 
> 
> Diffs
> -----
> 
>   test/pom.xml 644a9c7 
>   test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java 2c0c4e6

>   test/src/main/java/org/apache/sqoop/test/infrastructure/SqoopTestCase.java 74fe29b

>   test/src/main/java/org/apache/sqoop/test/kdc/KdcRunner.java aa5c6fc 
>   test/src/main/java/org/apache/sqoop/test/kdc/MiniKdcRunner.java 299fd9f 
>   test/src/main/java/org/apache/sqoop/test/kdc/NoKerberosKdcRunner.java 29a005e 
>   test/src/main/java/org/apache/sqoop/test/minicluster/SqoopMiniCluster.java 4ff97e7

>   test/src/test/java/org/apache/sqoop/integration/connector/hdfs/OutputDirectoryTest.java
e5e3d26 
>   test/src/test/java/org/apache/sqoop/integration/connector/hive/FromRDBMSToKiteHiveTest.java
ec9f733 
>   test/src/test/java/org/apache/sqoop/integration/connectorloading/BlacklistedConnectorTest.java
1c7a483 
>   test/src/test/java/org/apache/sqoop/integration/connectorloading/ClasspathTest.java
5e49064 
>   test/src/test/java/org/apache/sqoop/integration/connectorloading/ConnectorClasspathIsolationTest.java
bbbf749 
>   test/src/test/resources/hive-tests-suite.xml PRE-CREATION 
>   test/src/test/resources/integration-tests-suite.xml 1cf3299 
> 
> Diff: https://reviews.apache.org/r/41971/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Dian Fu
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message