storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alec Swan <alecs...@gmail.com>
Subject Configuring Python environment
Date Mon, 02 May 2016 22:34:59 GMT
Hello,

I am having a real hard time configuring Storm to use python2.7 virtual
environment that I installed on my CentOS 6.5 host.

CentOS 6.5 ships with python2.6 but storm requires at least python2.7.
Switching to python2.7 breaks some of CentOS functionality, so I had to
install python2.7 as a separate environment.

I also read that if I set BASH_ENV env var to point to the script that
activates the virtual environment, then it will be loaded before scripts
are executed. I also changed the first line in storm.py to:
#!/usr/bin/env python
so that the right python executable is being used. However, I still have no
luck and keep getting the error shown below after topology is successfully
submitted.

Has anyone figured out how to use Storm with python virtual environments?

2016-05-02 21:57:43.503 b.s.util [ERROR] Async loop died!
java.lang.RuntimeException: Error when launching multilang subprocess
python: error while loading shared libraries: libpython2.7.so.1.0: cannot
open shared object file: No such file or directory

        at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:68)
~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:117)
~[storm-core-0.10.0.jar:0.10.0]
        at
backtype.storm.daemon.executor$fn__5694$fn__5707.invoke(executor.clj:757)
~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.util$async_loop$fn__545.invoke(util.clj:477)
[storm-core-0.10.0.jar:0.10.0]
        at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_73]
Caused by: java.io.IOException: Broken pipe
        at java.io.FileOutputStream.writeBytes(Native Method) ~[?:1.8.0_73]
        at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[?:1.8.0_73]
        at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
~[?:1.8.0_73]
        at
java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
~[?:1.8.0_73]
        at java.io.DataOutputStream.flush(DataOutputStream.java:123)
~[?:1.8.0_73]
        at
backtype.storm.multilang.JsonSerializer.writeString(JsonSerializer.java:96)
~[storm-core-0.10.0.jar:0.10.0]
        at
backtype.storm.multilang.JsonSerializer.writeMessage(JsonSerializer.java:89)
~[storm-core-0.10.0.jar:0.10.0]
        at
backtype.storm.multilang.JsonSerializer.connect(JsonSerializer.java:61)
~[storm-core-0.10.0.jar:0.10.0]
        at backtype.storm.utils.ShellProcess.launch(ShellProcess.java:64)
~[storm-core-0.10.0.jar:0.10.0]
        ... 5 more


Thanks,

Alec

Mime
View raw message