storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Назар Кушпір <nazarkush...@gmail.com>
Subject Some executors stay idle
Date Wed, 19 Oct 2016 10:07:31 GMT
Hi,

I'm using Storm 1.0.2 and streamparse 3.0.1 (bolts, spouts and topology
being written in Python).
There is nothing special about the topology - it's just some serial data
processing:
*spout -> bolt_1 -> bolt_2 -> bolt_3 -> ... *

Since components are written using streamparse, I can't set number of tasks
per component, so number of tasks is equal to number of executors.

The problem is the following: all the executors log the message:
*"o.a.s.d.executor [INFO] Preparing bolt bolt_*:(34)"*

but some components never get prepared and thus stay idle - they don't
receive any tuple, i.e. I expect 5 "bolt_2" executors to run - I get 3
executors working and 2 - idle.
Components that run correctly log a message:
*"o.a.s.d.executor [INFO] Prepared bolt bolt_*: (10)"*

My question is: *How can I make ALL the components run and process data?*

Topology runs on 4 hosts - 2 workers per host.

storm.yaml:
--------------------------
storm.zookeeper.servers:
      - "zkhost-fqdn"
storm.zookeeper.root: "/topology-name"
storm.local.dir: "/data/storm"

nimbus.seeds : ["nimbus-host-fqdn"]

supervisor.slots.ports:
    - 6700
    - 6701
    - 6702
    - 6703

nimbus.childopts: "-Xmx1024m -Djava.net.preferIPv4Stack=true"
supervisor.childopts: "-Xmx1024m -Djava.net.preferIPv4Stack=true"
ui.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true"
worker.childopts: "-Xmx5120m -Djava.net.preferIPv4Stack=true
-XX:+PrintGCDetails -Xloggc:artifacts/gc.log -XX:+PrintGCDateStamps
-XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=1M -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=artifacts/heapdump"

topology.max.spout.pending: 500

supervisor.worker.timeout.secs: 600
nimbus.task.timeout.secs: 600
nimbus.supervisor.timeout.secs: 600
--------------------------

- Nazar

Mime
View raw message