spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From KhajaAsmath Mohammed <mdkhajaasm...@gmail.com>
Subject Re: Spark Streaming Kerberos Issue
Date Wed, 22 Nov 2017 18:33:21 GMT
We use oracle JDK. we are on unix.

On Wed, Nov 22, 2017 at 12:31 PM, Georg Heiler <georg.kf.heiler@gmail.com>
wrote:

> Do you use oracle or open jdk? We recently had an issue with open jdk:
> formerly, java Security extensions were installed by default - no longer so
> on centos 7.3
>
> Are these installed?
>
> KhajaAsmath Mohammed <mdkhajaasmath@gmail.com> schrieb am Mi. 22. Nov.
> 2017 um 19:29:
>
>> I passed keytab, renewal is enabled by running the script every eight
>> hours. User gets renewed by the script every eight hours.
>>
>> On Wed, Nov 22, 2017 at 12:27 PM, Georg Heiler <georg.kf.heiler@gmail.com
>> > wrote:
>>
>>> Did you pass a keytab? Is renewal enabled in your kdc?
>>> KhajaAsmath Mohammed <mdkhajaasmath@gmail.com> schrieb am Mi. 22. Nov.
>>> 2017 um 19:25:
>>>
>>>> Hi,
>>>>
>>>> I have written spark stream job and job is running successfully for
>>>> more than 36 hours. After around 36 hours job gets failed with kerberos
>>>> issue. Any solution on how to resolve it.
>>>>
>>>> org.apache.spark.SparkException: Task failed while wri\
>>>>
>>>> ting rows.
>>>>
>>>>                 at org.apache.spark.sql.hive.
>>>> SparkHiveDynamicPartitionWriterContainer.writeToFile(
>>>> hiveWriterContainers.scala:328)
>>>>
>>>>                 at org.apache.spark.sql.hive.
>>>> execution.InsertIntoHiveTable$$anonfun$saveAsHiveFile$3.
>>>> apply(InsertIntoHiveTable.scala:210)
>>>>
>>>>                 at org.apache.spark.sql.hive.
>>>> execution.InsertIntoHiveTable$$anonfun$saveAsHiveFile$3.
>>>> apply(InsertIntoHiveTable.scala:210)
>>>>
>>>>                 at org.apache.spark.scheduler.
>>>> ResultTask.runTask(ResultTask.scala:87)
>>>>
>>>>                 at org.apache.spark.scheduler.Task.run(Task.scala:99)
>>>>
>>>>                 at org.apache.spark.executor.Executor$TaskRunner.run(
>>>> Executor.scala:322)
>>>>
>>>>                 at java.util.concurrent.ThreadPoolExecutor.runWorker(
>>>> ThreadPoolExecutor.java:1145)
>>>>
>>>>                 at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>> ThreadPoolExecutor.java:615)
>>>>
>>>>                 at java.lang.Thread.run(Thread.java:745)
>>>>
>>>> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
>>>> java.io.IOException: org.apache.hadoop.security.authentication.client.\
>>>>
>>>> AuthenticationException: org.apache.hadoop.security.
>>>> token.SecretManager$InvalidToken: token (kms-dt owner=va_dflt,
>>>> renewer=yarn, re\
>>>>
>>>> alUser=, issueDate=1511262017635, maxDate=1511866817635,
>>>> sequenceNumber=1854601, masterKeyId=3392) is expired
>>>>
>>>>                 at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.
>>>> getHiveRecordWriter(HiveFileFormatUtils.java:248)
>>>>
>>>>                 at org.apache.spark.sql.hive.
>>>> SparkHiveDynamicPartitionWriterContainer.newOutputWriter$1(
>>>> hiveWriterContainers.scala:346)
>>>>
>>>>                 at org.apache.spark.sql.hive.
>>>> SparkHiveDynamicPartitionWriterContainer.writeToFile(
>>>> hiveWriterContainers.scala:304)
>>>>
>>>>                 ... 8 more
>>>>
>>>> Caused by: java.io.IOException: org.apache.hadoop.security.
>>>> authentication.client.AuthenticationException:
>>>> org.apache.hadoop.securit\
>>>>
>>>> y.token.SecretManager$InvalidToken: token (kms-dt owner=va_dflt,
>>>> renewer=yarn, realUser=, issueDate=1511262017635, maxDate=15118668\
>>>>
>>>> 17635, sequenceNumber=1854601, masterKeyId=3392) is expired
>>>>
>>>>                 at org.apache.hadoop.crypto.key.kms.
>>>> LoadBalancingKMSClientProvider.decryptEncryptedKey(
>>>> LoadBalancingKMSClientProvider.java:216)
>>>>
>>>>                 at org.apache.hadoop.crypto.key.
>>>> KeyProviderCryptoExtension.decryptEncryptedKey(
>>>> KeyProviderCryptoExtension.java:388)
>>>>
>>>>                 at org.apache.hadoop.hdfs.DFSClient.
>>>> decryptEncryptedDataEncryptionKey(DFSClient.java:1440)
>>>>
>>>>                 at org.apache.hadoop.hdfs.DFSClient.
>>>> createWrappedOutputStream(DFSClient.java:1542)
>>>>
>>>>                 at org.apache.hadoop.hdfs.DFSClient.
>>>> createWrappedOutputStream(DFSClient.java:1527)
>>>>
>>>>                 at org.apache.hadoop.hdfs.DistributedFileSystem$7.
>>>> doCall(DistributedFileSystem.java:428)
>>>>
>>>>                 at org.apache.hadoop.hdfs.DistributedFileSystem$7.
>>>> doCall(DistributedFileSystem.java:421)
>>>>
>>>>                 at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(
>>>> FileSystemLinkResolver.java:81)
>>>>
>>>>                 at org.apache.hadoop.hdfs.DistributedFileSystem.create(
>>>> DistributedFileSystem.java:421)
>>>>
>>>>                 at org.apache.hadoop.hdfs.DistributedFileSystem.create(
>>>> DistributedFileSystem.java:362)
>>>>
>>>>                 at org.apache.hadoop.fs.FileSystem.create(FileSystem.
>>>> java:925)
>>>>
>>>>                 at org.apache.hadoop.fs.FileSystem.create(FileSystem.
>>>> java:906)
>>>>
>>>>                 at parquet.hadoop.ParquetFileWriter.<init>(
>>>> ParquetFileWriter.java:220)
>>>>
>>>>                 at parquet.hadoop.ParquetOutputFormat.getRecordWriter(
>>>> ParquetOutputFormat.java:311)
>>>>
>>>>                 at parquet.hadoop.ParquetOutputFormat.getRecordWriter(
>>>> ParquetOutputFormat.java:287)
>>>>
>>>>                 at org.apache.hadoop.hive.ql.io.parquet.write.
>>>> ParquetRecordWriterWrapper.<init>(ParquetRecordWriterWrapper.java:65)
>>>>
>>>>                 at org.apache.hadoop.hive.ql.io.parquet.
>>>> MapredParquetOutputFormat.getParquerRecordWriterWrapper(
>>>> MapredParquetOutputFormat.java:125)
>>>>
>>>>                 at org.apache.hadoop.hive.ql.io.parquet.
>>>> MapredParquetOutputFormat.getHiveRecordWriter(
>>>> MapredParquetOutputFormat.java:114)
>>>>
>>>>                 at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.
>>>> getRecordWriter(HiveFileFormatUtils.java:260)
>>>>
>>>>                 at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.
>>>> getHiveRecordWriter(HiveFileFormatUtils.java:245)
>>>>
>>>>                 ... 10 more
>>>>
>>>> Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException:
>>>> org.apache.hadoop.security.token.SecretManager\
>>>>
>>>> $InvalidToken: token (kms-dt owner=va_dflt, renewer=yarn, realUser=,
>>>> issueDate=1511262017635, maxDate=1511866817635, sequenceNumber\
>>>>
>>>> =1854601, masterKeyId=3392) is expired
>>>>
>>>>                 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>>> Method)
>>>>
>>>>                 at sun.reflect.NativeConstructorAccessorImpl.
>>>> newInstance(NativeConstructorAccessorImpl.java:57)
>>>>
>>>>                 at sun.reflect.DelegatingConstructorAccessorI
>>>> mpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>>
>>>>                 at java.lang.reflect.Constructor.
>>>> newInstance(Constructor.java:526)
>>>>
>>>>                 at org.apache.hadoop.util.HttpExceptionUtils.
>>>> validateResponse(HttpExceptionUtils.java:157)
>>>>
>>>>                 at org.apache.hadoop.crypto.key.
>>>> kms.KMSClientProvider.call(KMSClientProvider.java:627)
>>>>
>>>>                 at org.apache.hadoop.crypto.key.
>>>> kms.KMSClientProvider.call(KMSClientProvider.java:585)
>>>>
>>>>                 at org.apache.hadoop.crypto.key.kms.KMSClientProvider.
>>>> decryptEncryptedKey(KMSClientProvider.java:852)
>>>>
>>>>                 at org.apache.hadoop.crypto.key.kms.
>>>> LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider
>>>> .java:209)
>>>>
>>>>                 at org.apache.hadoop.crypto.key.kms.
>>>> LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider
>>>> .java:205)
>>>>
>>>>                 at org.apache.hadoop.crypto.key.kms.
>>>> LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider
>>>> .java:94)
>>>>
>>>>                 at org.apache.hadoop.crypto.key.kms.
>>>> LoadBalancingKMSClientProvider.decryptEncryptedKey(
>>>> LoadBalancingKMSClientProvider.java:205)
>>>>
>>>>                 ... 29 more
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Asmath
>>>>
>>>
>>

Mime
View raw message