Sent too soon.
Hi Surneni,
This looks to be an issue with your network setup. The address or hostname known to a Drillbit
about another Drillbit is not resolved correctly while trying to make connection.
Nodes with failure: drillslave-7dccb67657-9qv4l, drillslave-7dccb67657-rhfgt
Can you please verify if above 2 nodes can ping each other using the hostname?
If not may be you can try to update the /etc/hosts file to provide a mapping between hostname
and corresponding IP address.
with org.apache.drill.exec.rpc.RpcException: Command failed while establishing connection.
Failure type CONNECTION..
2018-04-11 07:26:17,705 [BitServer-6] ERROR o.a.d.e.rpc.ConnectionMultiListener - Failed to
establish connection
java.util.concurrent.ExecutionException: java.nio.channels.UnresolvedAddressException
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
Thanks,
Sorabh
________________________________
From: Sorabh Hamirwasia
Sent: Wednesday, April 11, 2018 9:04 AM
To: user@drill.apache.org
Subject: Re: Apache Drill- Error setting up remote intermediate fragment execution
Hi Surneni,
This looks to be an issue with your network setup. The address or hostname known to a Drillbit
about another Drillbit is not resolved correctly while trying to make connection.
Thanks,
Sorabh
________________________________
From: Surneni Tilak <surneni.tilak@ericsson.com>
Sent: Wednesday, April 11, 2018 8:30 AM
To: user@drill.apache.org
Subject: FW: Apache Drill- Error setting up remote intermediate fragment execution
Hi Team,
I have successfully launched a Drill cluster on 3 docker containers with Zoo keeper Quorum
and accessing data from a remote node on which Drill is not installed . I am able to list
all the drill bits from each node and run queries on a single drill bit , But I am facing
issue when I am trying to run a query which distributes the query to the other drill bits
by creating the intermediate fragments. I have gone through the documentation but It couldn't
help. I got the below error message in Drill shell and the detailed error log mentioned
below. Please help me on resolving this.
Error in Drill Shell:
Error: CONNECTION ERROR: Error setting up remote intermediate fragment execution
Nodes with failures drillslave-7dccb67657-9qv4l, drillslave-7dccb67657-rhfgt
[Error Id: 5d701a0d-988b-4845-83f6-b029ad6825a8 on drillmaster-76d6885c59-wpksz:31010] (state=,code=0)
Error Log:
2018-04-11 07:26:17,391 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatus
es() took 0 ms, numFiles: 1
2018-04-11 07:26:17,392 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatus
es() took 0 ms, numFiles: 1
2018-04-11 07:26:17,392 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatus
es() took 0 ms, numFiles: 1
2018-04-11 07:26:17,392 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatus
es() took 0 ms, numFiles: 1
2018-04-11 07:26:17,392 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatus
es() took 0 ms, numFiles: 1
2018-04-11 07:26:17,392 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatus
es() took 0 ms, numFiles: 1
2018-04-11 07:26:17,401 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatus
es() took 0 ms, numFiles: 1
2018-04-11 07:26:17,404 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.e.s.schedule.BlockMapBuilder
- Get block maps: Execute
d 1 out of 1 using 1 threads. Time: 2ms total, 2.798853ms avg, 2ms max.
2018-04-11 07:26:17,404 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.e.s.schedule.BlockMapBuilder
- Get block maps: Execute
d 1 out of 1 using 1 threads. Earliest start: 0.699000 ?s, Latest start: 0.699000 ?s, Average
start: 0.699000 ?s .
2018-04-11 07:26:17,408 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.e.p.l.partition.PruneScanRule
- Beginning partition pr
uning, pruning class: org.apache.drill.exec.planner.logical.partition.PruneScanRule$DirPruneScanFilterOnScanRule
2018-04-11 07:26:17,408 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.d.e.p.l.partition.PruneScanRule
- No partition columns a
re projected from the scan..continue. Total pruning elapsed time: 0 ms
2018-04-11 07:26:17,611 [BitServer-3] ERROR o.a.d.e.rpc.ConnectionMultiListener - Failed to
establish connection
java.util.concurrent.ExecutionException: java.nio.channels.UnresolvedAddressException
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:89)
[drill-
rpc-1.12.0.jar:1.12.0]
at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:76)
[drill-
rpc-1.12.0.jar:1.12.0]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
[netty-common-4.0.48.Final.jar:4.0.48.Fina
l]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) [netty-common-4.0.48.Final.jar:4.0.48.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:241)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226)
[netty-transport-4.0.48.Final.ja
r:4.0.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
[netty-transport-4.0.48.F
inal.jar:4.0.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47)
[netty-transport-4.0.48.Final.ja
r:4.0.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
[netty-transport-4.0.48.F
inal.jar:4.0.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
[netty-transport-4.0.48.F
inal.jar:4.0.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:506)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:970)
[netty-transport-4.0.48.Final.jar:4.0.48.Fina
l]
at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:214) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:166) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
[netty-common-4.0.48.Final.
at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:166)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
[netty-common-4.0.48.Final.
jar:4.0.48.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
[netty-common-4.0.48.Final.jar:4.
0.48.Final]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.nio.channels.UnresolvedAddressException: null
at sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_161]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) ~[na:1.8.0_161]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_161]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:243)
~[netty-transport-4.0.48.Final.jar:4.0.48.Fi
nal]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
... 17 common frames omitted
2018-04-11 07:26:17,693 [BitServer-4] ERROR o.a.d.e.rpc.ConnectionMultiListener - Failed to
establish connection
java.util.concurrent.ExecutionException: java.nio.channels.UnresolvedAddressException
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:89)
[drill-
rpc-1.12.0.jar:1.12.0]
at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:76)
[drill-
rpc-1.12.0.jar:1.12.0]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
[netty-common-4.0.48.Final.jar:4.0.48.Fina
l]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) [netty-common-4.0.48.Final.jar:4.0.48.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:241)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226)
[netty-transport-4.0.48.Final.ja
r:4.0.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
[netty-transport-4.0.48.F
inal.jar:4.0.48.Final]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.nio.channels.UnresolvedAddressException: null
at sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_161]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) ~[na:1.8.0_161]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_161]
at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:243)
~[netty-transport-4.0.48.Final.jar:4.0.48.Fi
nal]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
... 17 common frames omitted
2018-04-11 07:26:17,694 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO o.a.drill.exec.work.foreman.Foreman
- User Error Occurred: E
rror setting up remote intermediate fragment execution (Command failed while establishing
connection. Failure type CONNECTION.)
org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Error setting up remote
intermediate fragment execution
Nodes with failures drillslave-7dccb67657-9qv4l, drillslave-7dccb67657-rhfgt
[Error Id: 7e6e629d-1558-4a76-b9a9-6d34f86015a2 ]
at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
~[drill-common-1.12.0.jar:1.12.0]
at org.apache.drill.exec.work.foreman.Foreman.scheduleRemoteIntermediateFragments(Foreman.java:1126)
[drill-java-exec-1.12.0.jar:
1.12.0]
at org.apache.drill.exec.work.foreman.Foreman.setupNonRootFragments(Foreman.java:1217)
[drill-java-exec-1.12.0.jar:1.12.0]
at org.apache.drill.exec.work.foreman.Foreman.runPhysicalPlan(Foreman.java:454) [drill-java-exec-1.12.0.jar:1.12.0]
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1019) [drill-java-exec-1.12.0.jar:1.12.0]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:289) [drill-java-exec-1.12.0.jar:1.12.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_161]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: org.apache.drill.exec.rpc.RpcException: Command failed while establishing connection.
Failure type CONNECTION.
at org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:67) ~[drill-rpc-1.12.0.jar:1.12.0]
at org.apache.drill.exec.rpc.ListeningCommand.connectionFailed(ListeningCommand.java:66)
~[drill-rpc-1.12.0.jar:1.12.0]
at org.apache.drill.exec.rpc.ReconnectingConnection$ConnectionListeningFuture.connectionFailed(ReconnectingConnection.java:152)
~
[drill-rpc-1.12.0.jar:1.12.0]
at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:118)
~[dril
l-rpc-1.12.0.jar:1.12.0]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205)
~[netty-transport-4.0.48.Final.
jar:4.0.48.Final]
... 17 common frames omitted
2018-04-11 07:26:17,702 [BitServer-9] INFO o.a.d.e.w.fragment.FragmentExecutor - 25324766-53a8-f988-20b4-65bbe0aa6cf9:0:0:
State change
requested AWAITING_ALLOCATION --> CANCELLATION_REQUESTED
2018-04-11 07:26:17,702 [BitServer-9] INFO o.a.d.e.w.f.FragmentStatusReporter - 25324766-53a8-f988-20b4-65bbe0aa6cf9:0:0:
State to repor
t: CANCELLATION_REQUESTED
2018-04-11 07:26:17,702 [BitServer-9] INFO o.a.d.e.w.fragment.FragmentExecutor - 25324766-53a8-f988-20b4-65bbe0aa6cf9:0:0:
State change
requested CANCELLATION_REQUESTED --> FINISHED
2018-04-11 07:26:17,702 [BitServer-9] INFO o.a.d.e.w.f.FragmentStatusReporter - 25324766-53a8-f988-20b4-65bbe0aa6cf9:0:0:
State to repor
t: CANCELLED
2018-04-11 07:26:17,703 [BitServer-5] ERROR o.a.d.e.rpc.ConnectionMultiListener - Failed to
establish connection
java.util.concurrent.ExecutionException: java.nio.channels.UnresolvedAddressException
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:89)
[drill-
rpc-1.12.0.jar:1.12.0]
at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:76)
[drill-
rpc-1.12.0.jar:1.12.0]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
[netty-common-4.0.48.Final.jar:4.0.48.Fina
l]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) [netty-common-4.0.48.Final.jar:4.0.48.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:241)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226)
[netty-transport-4.0.48.Final.ja
r:4.0.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
[netty-transport-4.0.48.F
inal.jar:4.0.48.Final]
at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
... 17 common frames omitted
2018-04-11 07:26:17,704 [BitServer-5] ERROR o.a.d.exec.work.foreman.QueryManager - Failure
while attempting to CANCEL fragment query_id {
part1: 2680283233054685576
part2: 2356620362624167161
}
major_fragment_id: 1
minor_fragment_id: 0
on endpoint address: "drillslave-7dccb67657-9qv4l"
user_port: 31010
control_port: 31011
data_port: 31012
version: "1.12.0"
state: ONLINE
with org.apache.drill.exec.rpc.RpcException: Command failed while establishing connection.
Failure type CONNECTION..
2018-04-11 07:26:17,705 [BitServer-6] ERROR o.a.d.e.rpc.ConnectionMultiListener - Failed to
establish connection
java.util.concurrent.ExecutionException: java.nio.channels.UnresolvedAddressException
at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:89)
[drill-
rpc-1.12.0.jar:1.12.0]
at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:76)
[drill-
rpc-1.12.0.jar:1.12.0]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
[netty-common-4.0.48.Final.jar:4.0.48.Fina
l]
Please respond back ASAP with your findings.
Best regards,
_________________________
Tilak
|