storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alec Swan <alecs...@gmail.com>
Subject Re: Configuring Python environment
Date Tue, 03 May 2016 03:25:07 GMT
I did try streamparse, but it failed to deploy the topology to the remote
Storm cluster because of some internal errors which I wasn't able to track
down. I am not surprised though, because even manual install of python 2.7
or virtual environment is painful on CentOS 6.5.
So, I took the topology JAR produced by Streamparse and tried to manually
deploy on Storm, but ran into the Python issues that I am trying to resolve
in this thread.

Thanks,

Alec

On Mon, May 2, 2016 at 8:40 PM, Jiaming Lin <linnjiaming@gmail.com> wrote:

> You can try streamparse, I haven't used it, but it looks charming.
>
>
> On Tuesday, May 3, 2016, Alec Swan <alecswan@gmail.com> wrote:
>
>> Thanks, Joaquin, but I've seen all these articles since I spent the
>> entire weekend trying to get my topology to run on CentOS/Python2.7.
>> Getting it to work on Mac OSX was a breeze because I can easily install
>> Python 2.7 on it.
>>
>> It'd be great if somebody with the actually CentOS/Python2.7 experience
>> could share their thoughts.
>>
>> Thanks,
>>
>> Alec
>>
>> On Mon, May 2, 2016 at 7:59 PM, Joaquin Menchaca <jmenchaca@gobalto.com>
>> wrote:
>>
>>> virtualenvwrappers
>>>   - http://virtualenvwrapper.readthedocs.io/en/latest/
>>> - https://pypi.python.org/pypi/virtualenvwrapper/
>>> Pyenv
>>>   - https://amaral.northwestern.edu/resources/guides/pyenv-tutorial
>>>   - https://github.com/yyuu/pyenv
>>> Docker
>>>
>>> You can have a Python in another directory, then put that directory in
>>> the path.  Maybe linuxbrew can help with that.
>>>
>>> Linuxbrew - http://linuxbrew.sh
>>>
>>> Disclaimer, I haven't tested any of these with Storm.  I personally
>>> don't use CentOS (or RHEL) any more, because they are dinosaurs when it
>>> comes to scripting languages and web development.  But many love the slow
>>> churn and stability.
>>>
>>> I also found these:
>>> -
>>> https://github.com/h2oai/h2o-2/wiki/Installing-python-2.7-on-centos-6.3.-Follow-this-sequence-exactly-for-centos-machine-only
>>> -
>>> https://www.digitalocean.com/community/tutorials/how-to-set-up-python-2-7-6-and-3-3-3-on-centos-6-4
>>> - http://toomuchdata.com/2014/02/16/how-to-install-python-on-centos/
>>> On May 2, 2016 3:35 PM, "Alec Swan" <alecswan@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I am having a real hard time configuring Storm to use python2.7 virtual
>>>> environment that I installed on my CentOS 6.5 host.
>>>>
>>>> CentOS 6.5 ships with python2.6 but storm requires at least python2.7.
>>>> Switching to python2.7 breaks some of CentOS functionality, so I had to
>>>> install python2.7 as a separate environment.
>>>>
>>>> I also read that if I set BASH_ENV env var to point to the script that
>>>> activates the virtual environment, then it will be loaded before scripts
>>>> are executed. I also changed the first line in storm.py to:
>>>> #!/usr/bin/env python
>>>> so that the right python executable is being used. However, I still
>>>> have no luck and keep getting the error shown below after topology is
>>>> successfully submitted.
>>>>
>>>> Has anyone figured out how to use Storm with python virtual
>>>> environments?
>>>>
>>>> 2016-05-02 21:57:43.503 b.s.util [ERROR] Async loop died!
>>>> java.lang.RuntimeException: Error when launching multilang subprocess
>>>> python: error while loading shared libraries: libpython2.7.so.1.0:
>>>> cannot open shared object file: No such file or directory
>>>>
>>>>         at
>>>> backtype.storm.utils.ShellProcess.launch(ShellProcess.java:68)
>>>> ~[storm-core-0.10.0.jar:0.10.0]
>>>>         at backtype.storm.task.ShellBolt.prepare(ShellBolt.java:117)
>>>> ~[storm-core-0.10.0.jar:0.10.0]
>>>>         at
>>>> backtype.storm.daemon.executor$fn__5694$fn__5707.invoke(executor.clj:757)
>>>> ~[storm-core-0.10.0.jar:0.10.0]
>>>>         at backtype.storm.util$async_loop$fn__545.invoke(util.clj:477)
>>>> [storm-core-0.10.0.jar:0.10.0]
>>>>         at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
>>>>         at java.lang.Thread.run(Thread.java:745) [?:1.8.0_73]
>>>> Caused by: java.io.IOException: Broken pipe
>>>>         at java.io.FileOutputStream.writeBytes(Native Method)
>>>> ~[?:1.8.0_73]
>>>>         at java.io.FileOutputStream.write(FileOutputStream.java:326)
>>>> ~[?:1.8.0_73]
>>>>         at
>>>> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
>>>> ~[?:1.8.0_73]
>>>>         at
>>>> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
>>>> ~[?:1.8.0_73]
>>>>         at java.io.DataOutputStream.flush(DataOutputStream.java:123)
>>>> ~[?:1.8.0_73]
>>>>         at
>>>> backtype.storm.multilang.JsonSerializer.writeString(JsonSerializer.java:96)
>>>> ~[storm-core-0.10.0.jar:0.10.0]
>>>>         at
>>>> backtype.storm.multilang.JsonSerializer.writeMessage(JsonSerializer.java:89)
>>>> ~[storm-core-0.10.0.jar:0.10.0]
>>>>         at
>>>> backtype.storm.multilang.JsonSerializer.connect(JsonSerializer.java:61)
>>>> ~[storm-core-0.10.0.jar:0.10.0]
>>>>         at
>>>> backtype.storm.utils.ShellProcess.launch(ShellProcess.java:64)
>>>> ~[storm-core-0.10.0.jar:0.10.0]
>>>>         ... 5 more
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Alec
>>>>
>>>
>>

Mime
View raw message