AccessControlException problem in Hadoop index task (occurs with certain library)

[LEEAHRI]

Affected Version

Druid 31.0.0

Description

Hello,
I ran into a Kerberos authentication problem while trying to run a Hadoop index task. The problem seems to occur during the process of getting the MR job report. However, there is no direct issue with the task itself, and the task completes successfully, so the problem seems to be related to the authentication step rather than the execution of the task.

Below is the error stack trace (hadoop-client-api-3.3.6):

2024-11-18T15:11:03,789 WARN [task-runner-0-priority-0] org.apache.hadoop.ipc.Client - Exception encountered while connecting to the server {server:port}
org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
	at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:179) ~[hadoop-client-api-3.3.6.jar:?]
	at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:392) ~[hadoop-client-api-3.3.6.jar:?]
	at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:561) ~[hadoop-client-api-3.3.6.jar:?]
	at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:347) [hadoop-client-api-3.3.6.jar:?]
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:783) ~[hadoop-client-api-3.3.6.jar:?]
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:779) ~[hadoop-client-api-3.3.6.jar:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_242]
	at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_242]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) ~[hadoop-client-api-3.3.6.jar:?]
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:779) [hadoop-client-api-3.3.6.jar:?]
	at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:347) [hadoop-client-api-3.3.6.jar:?]
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1632) [hadoop-client-api-3.3.6.jar:?]
	at org.apache.hadoop.ipc.Client.call(Client.java:1457) [hadoop-client-api-3.3.6.jar:?]
	at org.apache.hadoop.ipc.Client.call(Client.java:1410) [hadoop-client-api-3.3.6.jar:?]
	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:258) [hadoop-client-api-3.3.6.jar:?]
	at org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:139) [hadoop-client-api-3.3.6.jar:?]
	at com.sun.proxy.$Proxy263.getJobReport(Unknown Source) [?:?]
	at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:135) [hadoop-client-api-3.3.6.jar:?]
	at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) ~[?:?]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_242]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_242]
	at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:326) [hadoop-client-api-3.3.6.jar:?]
	at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:430) [hadoop-client-api-3.3.6.jar:?]
	at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:872) [hadoop-client-api-3.3.6.jar:?]
	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:333) [hadoop-client-api-3.3.6.jar:?]
	at org.apache.hadoop.mapreduce.Job$1.run(Job.java:330) [hadoop-client-api-3.3.6.jar:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_242]
	at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_242]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899) [hadoop-client-api-3.3.6.jar:?]
	at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:330) [hadoop-client-api-3.3.6.jar:?]
	at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:614) [hadoop-client-api-3.3.6.jar:?]
	at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1737) [hadoop-client-api-3.3.6.jar:?]
	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1699) [hadoop-client-api-3.3.6.jar:?]
	at org.apache.druid.indexer.DetermineHashedPartitionsJob.run(DetermineHashedPartitionsJob.java:136) [druid-indexing-hadoop-31.0.0-SNAPSHOT.jar:31.0.0-SNAPSHOT]

When I lowered the library version to hadoop-client:2.8.5 in the hadoopDependencyCoordinates, I could not reproduce the authentication issue and the task ran successfully with no errors.

  "hadoopDependencyCoordinates": ["org.apache.hadoop:hadoop-client:2.8.5"]

Is there any guidance you can provide on how to resolve the issue with the Kerberos authentication?

[wujiandaod]

I got the same issue with Druid-v27.0.0 and hadoop-client-api 3.3.6