sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From daniel voros <daniel.vo...@gmail.com>
Subject Re: Review Request 66300: Upgrade to Hadoop 3.0.0
Date Tue, 17 Jul 2018 13:40:26 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66300/#review206151
-----------------------------------------------------------



I've been working on the failure of `TestHiveMiniCluster#testInsertedRowCanBeReadFromTable[KerberosAuthenticationConfiguration]`
and wanted to give an update.

I think this is the meaningful part of the quite verbose logs:

```
java.lang.Exception: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in
shuffle in localfetcher#1
	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559)
Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle
in localfetcher#1
	at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ExceptionInInitializerError
	at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:71)
	at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:62)
	at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:57)
	at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:125)
	at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103)
	at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86)
Caused by: java.lang.RuntimeException: Secure IO is not possible without native code extensions.
	at org.apache.hadoop.io.SecureIOUtils.<clinit>(SecureIOUtils.java:71)
	... 6 more
```

This is happening in MR's shuffle phase. I was trying to find out how tests in Hive don't
run into this and found out that Secure MR is not supported there. See here: https://github.com/apache/hive/blob/dceeefbdf5e4f6fea83cb6ca5c11fbac10e77677/itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java#L178-L180

I'm trying to get Tez working with our MiniHiveCluster, but did not succeed yet. The kerberos
ticket is not picked up for some reason:

```
java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate
via:[TOKEN, KERBEROS]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:755)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
        at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:718)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:811)
        at org.apache.hadoop.ipc.Client$Connection.access$3500(Client.java:409)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1552)
        at org.apache.hadoop.ipc.Client.call(Client.java:1383)
        at org.apache.hadoop.ipc.Client.call(Client.java:1347)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
        at com.sun.proxy.$Proxy81.getAMStatus(Unknown Source)
        at org.apache.tez.client.TezClient.getAppMasterStatus(TezClient.java:772)
        at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:909)
        at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:880)
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:434)
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:360)
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.openInternal(TezSessionPoolSession.java:124)
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:237)
        at org.apache.hadoop.hive.ql.exec.tez.TezTask.ensureSessionHasResources(TezTask.java:364)
        at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:191)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2479)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2150)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1826)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1567)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1561)
        at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:221)
        at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
        at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:313)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
        at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:326)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN,
KERBEROS]
        at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173)
        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
        at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:613)
        at org.apache.hadoop.ipc.Client$Connection.access$2200(Client.java:409)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:798)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:794)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:794)
        ... 36 more
```

- daniel voros


On March 27, 2018, 8:50 a.m., daniel voros wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66300/
> -----------------------------------------------------------
> 
> (Updated March 27, 2018, 8:50 a.m.)
> 
> 
> Review request for Sqoop.
> 
> 
> Bugs: SQOOP-3305
>     https://issues.apache.org/jira/browse/SQOOP-3305
> 
> 
> Repository: sqoop-trunk
> 
> 
> Description
> -------
> 
> To be able to eventually support the latest versions of Hive, HBase and Accumulo, we
should start by upgrading our Hadoop dependencies to 3.0.0. See https://hadoop.apache.org/docs/r3.0.0/index.html
> 
> 
> Diffs
> -----
> 
>   ivy.xml 1f587f3e 
>   ivy/libraries.properties 565a8bf5 
>   src/java/org/apache/sqoop/SqoopOptions.java d9984af3 
>   src/java/org/apache/sqoop/config/ConfigurationHelper.java fb2ab031 
>   src/java/org/apache/sqoop/hive/HiveImport.java 5da00a74 
>   src/java/org/apache/sqoop/mapreduce/JobBase.java 6d1e0499 
>   src/java/org/apache/sqoop/mapreduce/hcat/DerbyPolicy.java PRE-CREATION 
>   src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 784b5f2a 
>   src/java/org/apache/sqoop/mapreduce/parquet/kite/KiteParquetUtils.java e68bba90 
>   src/java/org/apache/sqoop/util/SqoopJsonUtil.java adf186b7 
>   src/test/org/apache/sqoop/TestSqoopOptions.java bb7c20dd 
>   src/test/org/apache/sqoop/hive/minicluster/HiveMiniCluster.java 19bb7605 
>   src/test/org/apache/sqoop/hive/minicluster/KerberosAuthenticationConfiguration.java
549a8c6c 
>   src/test/org/apache/sqoop/hive/minicluster/PasswordAuthenticationConfiguration.java
79881f7b 
>   src/test/org/apache/sqoop/util/TestSqoopJsonUtil.java fdf972c1 
>   testdata/hcatalog/conf/hive-site.xml edac7aa9 
> 
> 
> Diff: https://reviews.apache.org/r/66300/diff/7/
> 
> 
> Testing
> -------
> 
> Normal and third-party unit tests.
> 
> 
> Thanks,
> 
> daniel voros
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message