drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Surneni Tilak <surneni.ti...@ericsson.com>
Subject FW: Apache Drill- Error setting up remote intermediate fragment execution
Date Wed, 11 Apr 2018 15:30:36 GMT
Hi Team,

I have successfully launched  a Drill cluster on 3 docker containers with Zoo keeper  Quorum
and accessing data from a remote node on which Drill is not installed . I am able to list
all the drill bits from each node and  run  queries on a single drill bit , But I am facing
issue when I am trying to run a query which distributes the query to the other drill bits
by creating the intermediate fragments.  I have gone through the documentation but It couldn't
help. I got the below error message in Drill shell  and the detailed  error log mentioned
below. Please help me on resolving this.

Error in Drill Shell:

Error: CONNECTION ERROR: Error setting up remote intermediate fragment execution

Nodes with failures drillslave-7dccb67657-9qv4l, drillslave-7dccb67657-rhfgt

[Error Id: 5d701a0d-988b-4845-83f6-b029ad6825a8 on drillmaster-76d6885c59-wpksz:31010] (state=,code=0)


Error Log:





2018-04-11 07:26:17,391 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatus
es() took 0 ms, numFiles: 1
2018-04-11 07:26:17,392 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatus
es() took 0 ms, numFiles: 1
2018-04-11 07:26:17,392 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatus
es() took 0 ms, numFiles: 1
2018-04-11 07:26:17,392 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatus
es() took 0 ms, numFiles: 1
2018-04-11 07:26:17,392 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatus
es() took 0 ms, numFiles: 1
2018-04-11 07:26:17,392 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatus
es() took 0 ms, numFiles: 1
2018-04-11 07:26:17,401 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO  o.a.d.exec.store.dfs.FileSelection
- FileSelection.getStatus
es() took 0 ms, numFiles: 1
2018-04-11 07:26:17,404 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO  o.a.d.e.s.schedule.BlockMapBuilder
- Get block maps: Execute
d 1 out of 1 using 1 threads. Time: 2ms total, 2.798853ms avg, 2ms max.
2018-04-11 07:26:17,404 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO  o.a.d.e.s.schedule.BlockMapBuilder
- Get block maps: Execute
d 1 out of 1 using 1 threads. Earliest start: 0.699000 ?s, Latest start: 0.699000 ?s, Average
start: 0.699000 ?s .
2018-04-11 07:26:17,408 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO  o.a.d.e.p.l.partition.PruneScanRule
- Beginning partition pr
uning, pruning class: org.apache.drill.exec.planner.logical.partition.PruneScanRule$DirPruneScanFilterOnScanRule
2018-04-11 07:26:17,408 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO  o.a.d.e.p.l.partition.PruneScanRule
- No partition columns a
re projected from the scan..continue. Total pruning elapsed time: 0 ms
2018-04-11 07:26:17,611 [BitServer-3] ERROR o.a.d.e.rpc.ConnectionMultiListener - Failed to
establish connection
java.util.concurrent.ExecutionException: java.nio.channels.UnresolvedAddressException
        at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
        at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:89)
[drill-
rpc-1.12.0.jar:1.12.0]
        at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:76)
[drill-
rpc-1.12.0.jar:1.12.0]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
[netty-common-4.0.48.Final.jar:4.0.48.Fina
l]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) [netty-common-4.0.48.Final.jar:4.0.48.Final]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:241)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226)
[netty-transport-4.0.48.Final.ja
r:4.0.48.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
[netty-transport-4.0.48.F
inal.jar:4.0.48.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
        at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:47)
[netty-transport-4.0.48.Final.ja
r:4.0.48.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
[netty-transport-4.0.48.F
inal.jar:4.0.48.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
        at io.netty.channel.ChannelDuplexHandler.connect(ChannelDuplexHandler.java:50) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
[netty-transport-4.0.48.F
inal.jar:4.0.48.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:506)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
        at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:970)
[netty-transport-4.0.48.Final.jar:4.0.48.Fina
l]
        at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:214) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
        at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:166) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
[netty-common-4.0.48.Final.
                                        at io.netty.bootstrap.Bootstrap$2.run(Bootstrap.java:166)
[netty-transport-4.0.48.Final.jar:4.0.48.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
[netty-common-4.0.48.Final.
jar:4.0.48.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) [netty-transport-4.0.48.Final.jar:4.0.48.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
[netty-common-4.0.48.Final.jar:4.
0.48.Final]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.nio.channels.UnresolvedAddressException: null
        at sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_161]
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) ~[na:1.8.0_161]
        at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
        at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
        at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_161]
        at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
        at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:243)
~[netty-transport-4.0.48.Final.jar:4.0.48.Fi
nal]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
        ... 17 common frames omitted
2018-04-11 07:26:17,693 [BitServer-4] ERROR o.a.d.e.rpc.ConnectionMultiListener - Failed to
establish connection
java.util.concurrent.ExecutionException: java.nio.channels.UnresolvedAddressException
        at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
        at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:89)
[drill-
rpc-1.12.0.jar:1.12.0]
        at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:76)
[drill-
rpc-1.12.0.jar:1.12.0]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
[netty-common-4.0.48.Final.jar:4.0.48.Fina
l]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) [netty-common-4.0.48.Final.jar:4.0.48.Final]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:241)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226)
[netty-transport-4.0.48.Final.ja
r:4.0.48.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
[netty-transport-4.0.48.F
inal.jar:4.0.48.Final]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: java.nio.channels.UnresolvedAddressException: null
        at sun.nio.ch.Net.checkAddress(Net.java:101) ~[na:1.8.0_161]
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) ~[na:1.8.0_161]
        at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:83) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
        at io.netty.util.internal.SocketUtils$3.run(SocketUtils.java:80) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
        at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_161]
        at io.netty.util.internal.SocketUtils.connect(SocketUtils.java:80) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
        at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:243)
~[netty-transport-4.0.48.Final.jar:4.0.48.Fi
nal]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
        ... 17 common frames omitted
2018-04-11 07:26:17,694 [25324766-53a8-f988-20b4-65bbe0aa6cf9:foreman] INFO  o.a.drill.exec.work.foreman.Foreman
- User Error Occurred: E
rror setting up remote intermediate fragment execution (Command failed while establishing
connection.  Failure type CONNECTION.)
org.apache.drill.common.exceptions.UserException: CONNECTION ERROR: Error setting up remote
intermediate fragment execution

Nodes with failures drillslave-7dccb67657-9qv4l, drillslave-7dccb67657-rhfgt

[Error Id: 7e6e629d-1558-4a76-b9a9-6d34f86015a2 ]
        at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586)
~[drill-common-1.12.0.jar:1.12.0]
        at org.apache.drill.exec.work.foreman.Foreman.scheduleRemoteIntermediateFragments(Foreman.java:1126)
[drill-java-exec-1.12.0.jar:
1.12.0]
        at org.apache.drill.exec.work.foreman.Foreman.setupNonRootFragments(Foreman.java:1217)
[drill-java-exec-1.12.0.jar:1.12.0]
        at org.apache.drill.exec.work.foreman.Foreman.runPhysicalPlan(Foreman.java:454) [drill-java-exec-1.12.0.jar:1.12.0]
       at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1019) [drill-java-exec-1.12.0.jar:1.12.0]
        at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:289) [drill-java-exec-1.12.0.jar:1.12.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_161]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161]
Caused by: org.apache.drill.exec.rpc.RpcException: Command failed while establishing connection.
 Failure type CONNECTION.
        at org.apache.drill.exec.rpc.RpcException.mapException(RpcException.java:67) ~[drill-rpc-1.12.0.jar:1.12.0]
        at org.apache.drill.exec.rpc.ListeningCommand.connectionFailed(ListeningCommand.java:66)
~[drill-rpc-1.12.0.jar:1.12.0]
        at org.apache.drill.exec.rpc.ReconnectingConnection$ConnectionListeningFuture.connectionFailed(ReconnectingConnection.java:152)
~
[drill-rpc-1.12.0.jar:1.12.0]
        at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:118)
~[dril
l-rpc-1.12.0.jar:1.12.0]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:205)
~[netty-transport-4.0.48.Final.
jar:4.0.48.Final]
        ... 17 common frames omitted
2018-04-11 07:26:17,702 [BitServer-9] INFO  o.a.d.e.w.fragment.FragmentExecutor - 25324766-53a8-f988-20b4-65bbe0aa6cf9:0:0:
State change
requested AWAITING_ALLOCATION --> CANCELLATION_REQUESTED
2018-04-11 07:26:17,702 [BitServer-9] INFO  o.a.d.e.w.f.FragmentStatusReporter - 25324766-53a8-f988-20b4-65bbe0aa6cf9:0:0:
State to repor
t: CANCELLATION_REQUESTED
2018-04-11 07:26:17,702 [BitServer-9] INFO  o.a.d.e.w.fragment.FragmentExecutor - 25324766-53a8-f988-20b4-65bbe0aa6cf9:0:0:
State change
requested CANCELLATION_REQUESTED --> FINISHED
2018-04-11 07:26:17,702 [BitServer-9] INFO  o.a.d.e.w.f.FragmentStatusReporter - 25324766-53a8-f988-20b4-65bbe0aa6cf9:0:0:
State to repor
t: CANCELLED
2018-04-11 07:26:17,703 [BitServer-5] ERROR o.a.d.e.rpc.ConnectionMultiListener - Failed to
establish connection
java.util.concurrent.ExecutionException: java.nio.channels.UnresolvedAddressException
        at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
        at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:89)
[drill-
rpc-1.12.0.jar:1.12.0]
        at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:76)
[drill-
rpc-1.12.0.jar:1.12.0]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
[netty-common-4.0.48.Final.jar:4.0.48.Fina
l]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) [netty-common-4.0.48.Final.jar:4.0.48.Final]
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.connect(AbstractNioChannel.java:241)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1226)
[netty-transport-4.0.48.Final.ja
r:4.0.48.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:539)
[netty-transport-4.0.48.F
inal.jar:4.0.48.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:524)
[netty-transport-4.0.48.Final.j
ar:4.0.48.Final]
        ... 17 common frames omitted
2018-04-11 07:26:17,704 [BitServer-5] ERROR o.a.d.exec.work.foreman.QueryManager - Failure
while attempting to CANCEL fragment query_id {
  part1: 2680283233054685576
  part2: 2356620362624167161
}
major_fragment_id: 1
minor_fragment_id: 0
on endpoint address: "drillslave-7dccb67657-9qv4l"
user_port: 31010
control_port: 31011
data_port: 31012
version: "1.12.0"
state: ONLINE
with org.apache.drill.exec.rpc.RpcException: Command failed while establishing connection.
 Failure type CONNECTION..
2018-04-11 07:26:17,705 [BitServer-6] ERROR o.a.d.e.rpc.ConnectionMultiListener - Failed to
establish connection
java.util.concurrent.ExecutionException: java.nio.channels.UnresolvedAddressException
        at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:54) ~[netty-common-4.0.48.Final.jar:4.0.48.Final]
        at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:89)
[drill-
rpc-1.12.0.jar:1.12.0]
        at org.apache.drill.exec.rpc.ConnectionMultiListener$ConnectionHandler.operationComplete(ConnectionMultiListener.java:76)
[drill-
rpc-1.12.0.jar:1.12.0]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507)
[netty-common-4.0.48.Final.jar:4.0.48.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:481)
[netty-common-4.0.48.Final.jar:4.0.48.Fina
l]



Please respond back ASAP with your findings.




Best regards,
_________________________
Tilak


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message