flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Márton Balassi <balassi.mar...@gmail.com>
Subject Re: Problem with deploying a flink topology in a remote execution environment
Date Thu, 13 Nov 2014 15:12:44 GMT
Dear Attila,

Both the issues you have reported are fixed with the streaming commits just
pushed to the current master. Could you please retry running the jobs?

Best,

Marton

On Mon, Nov 10, 2014 at 4:51 PM, Márton Balassi <balassi.marton@gmail.com>
wrote:

> Hey Attila,
>
> Thanks for the detailed bug report. I've picked up the issue and fixed the
> build for the streaming example packages. [1]
> There is an outstanding issue with the most likely the class loading when
> executing streaming code remotely. I've started working on it, Robert was
> kind enough to give me some guidance on it.
>
> [1]
> https://github.com/mbalassi/incubator-flink/commit/2be50b644807aeead825f512e13b21702404560f
>
> Getting back to you as soon as it is fixed,
>
> Marton
>
> On Wed, Nov 5, 2014 at 5:08 PM, axt <axt@load.hu> wrote:
>
>>  Hi Marton! Sorry for my late response.
>>
>> My classes are serializable, thats not what causing the error.
>>
>> I've managed to narrow the problem (to the official examples, without
>> using any own code):
>>
>> I can run the wordcount example (with input supplied) and the twitter
>> example, locally.
>> Unfortunatelly  when I try to submit them to the flink-on-yarn instance,
>> in both cases, I get the ClassNotFound errors. Stack traces are here:
>> http://pastebin.com/MMLaNTcv
>>
>> I've tried it with two different hadoop installs.
>>
>> I'm starting flink-on-yarn with this command:
>> HADOOP_HOME=/usr/lib/hadoop
>> /opt/flink-yarn-0.7.0-incubating/bin/yarn-session.sh -n 2 -jm 1024 -tm 4096
>> -s 4
>>
>> Can you give me the versions what you are using in your test
>> environments? Do you have any specific configuration options?
>>
>> Maybe any ideas how should I debug this problem?
>>
>> Just a note: the
>> "org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator.groupBy([I)Lorg/apache/flink/streaming/api/datastream/GroupedDataStream;"
>> errors have disappeared, it was caused just by an inconsistent build.
>>
>> Greetings,
>>   Attila
>>
>>
>>
>> On 2014-11-03 16:57, Márton Balassi wrote:
>>
>>  Hey Attila,
>>
>>  Thanks for trying out streaming!
>>
>>  As for your issues:
>>
>>     * Your simple topology: make sure that your user defined functions
>> (e.g. ItemsKafkaSource)  and the classes containing them are
>> serializable, so that they can be shipped. For anything more specific if
>> you can give send over your code to me I am more than happy to have a look
>> at it.
>>    * Streaming wordcount not working without provided dataset: Yes,
>> thanks for reporting it is my fault - which was known, but now I also have
>> a JIRA [1] for it. :) However if you give it some input it should work.
>>    * Can you run the Twitter example locally by the way?
>>
>>  [1] https://issues.apache.org/jira/browse/FLINK-1204
>>
>>  Cheers,
>>
>>  Marton
>>
>>
>> On Mon, Nov 3, 2014 at 2:53 PM, axt <axt@load.hu> wrote:
>>
>>> ClassNotFoundException:
>>> com.gravityrd.flink.ItemsKafkaSource
>>>
>>
>>
>>
>>
>

Mime
View raw message