samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Kniep <ckn...@gaikai.com>
Subject Re: Docker-Image for 'hello-samza'
Date Fri, 15 Jan 2016 10:04:14 GMT
Hi Yi,

I added documentation on how to spin up the image, I guess we could skip a couple of RTTs
by using a common Docker Image. :)
https://github.com/qnib/u-samza/blob/master/SetupDocker.md

Cheers
Christian

--




Christian Kniep  |  Release Engineer

www.gaikai.com <http://www.gaikai.com/>
> On 14 Jan 2016, at 19:12, Yi Pan <nickpan47@gmail.com> wrote:
> 
> Hi, Christian,
> 
> From the logs I can find on github, I only saw the job is submitted successfully, which
means that the Kafka systems are working fine. I don't see the following information:
> a) How do you determine that the job failed?
> b) There should be an userlogs under ${HELLO-SAMZA}/deploy/yarn/logs, in which you should
be able to find the SamzaAppMaster and SamzaContainer logs. That would tell us more about
errors.
> 
> Could you post those logs somewhere together w/ your configuration file s.t. we can help
to dig into it more?
> 
> Thanks a lot!
> 
> -Yi
> 
> On Thu, Jan 14, 2016 at 3:07 AM, Christian Kniep <ckniep@gaikai.com <mailto:ckniep@gaikai.com>>
wrote:
> The kafka error seems not to be the important piece… https://issues.apache.org/jira/browse/KAFKA-2251
<https://issues.apache.org/jira/browse/KAFKA-2251>
> 
> I created an image that holds all services needed to rule out that somehow the job assumes
‘localhost’ connections somewhere:
> https://hub.docker.com/r/qnib/samza-mono/builds/ <https://hub.docker.com/r/qnib/samza-mono/builds/>
> 
> Still get the same error. :/
> 
> --
> 
> 
> 
> 
> Christian Kniep  |  Release Engineer
> 
> www.gaikai.com <http://www.gaikai.com/>
>> On 14 Jan 2016, at 09:50, Christian Kniep <ckniep@gaikai.com <mailto:ckniep@gaikai.com>>
wrote:
>> 
>> Hey Yi,
>> 
>> as you can see within the github repo I explore software stacks by tackle them with
docker. Quite fun, but challenging from times to times.
>> Especially if it’s handled by layers of Java - not per se a problem, but my Java
knowledge lacks experience.
>> 
>> But to the topic… I checked the logs of ZK and Kafka and Kafka throws:
>> ####
>> [2016-01-14 09:42:28,748] INFO Closing socket connection to /172.17.0.7 <http://172.17.0.7/>.
(kafka.network.Processor)
>> [2016-01-14 09:42:28,807] INFO Closing socket connection to /172.17.0.7 <http://172.17.0.7/>.
(kafka.network.Processor)
>> [2016-01-14 09:42:29,993] INFO Closing socket connection to /172.17.0.7 <http://172.17.0.7/>.
(kafka.network.Processor)
>> [2016-01-14 09:42:30,010] INFO Closing socket connection to /172.17.0.7 <http://172.17.0.7/>.
(kafka.network.Processor)
>> [2016-01-14 09:42:30,010] INFO Closing socket connection to /172.17.0.7 <http://172.17.0.7/>.
(kafka.network.Processor)
>> [2016-01-14 09:42:36,183] INFO Closing socket connection to /172.17.0.7 <http://172.17.0.7/>.
(kafka.network.Processor)
>> [2016-01-14 09:42:36,270] INFO Closing socket connection to /172.17.0.7 <http://172.17.0.7/>.
(kafka.network.Processor)
>> [2016-01-14 09:42:40,591] INFO Closing socket connection to /172.17.0.7 <http://172.17.0.7/>.
(kafka.network.Processor)
>> [2016-01-14 09:42:40,592] INFO Closing socket connection to /172.17.0.7 <http://172.17.0.7/>.
(kafka.network.Processor)
>> [2016-01-14 09:42:46,451] ERROR Closing socket for /172.17.0.7 <http://172.17.0.7/>
because of error (kafka.network.Processor)
>> java.io.IOException: Connection reset by peer
>> 	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>> 	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>> 	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>> 	at sun.nio.ch.IOUtil.read(IOUtil.java:197)
>> 	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
>> 	at kafka.utils.Utils$.read(Utils.scala:380)
>> 	at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
>> 	at kafka.network.Processor.read(SocketServer.scala:444)
>> 	at kafka.network.Processor.run(SocketServer.scala:340)
>> 	at java.lang.Thread.run(Thread.java:745)
>> [2016-01-14 09:42:46,451] INFO Closing socket connection to /172.17.0.7 <http://172.17.0.7/>.
(kafka.network.Processor)
>> [2016-01-14 09:42:46,452] INFO Closing socket connection to /172.17.0.7 <http://172.17.0.7/>.
(kafka.network.Processor)
>> ####
>> 
>> Even thought the Kafka-Manager seems to work as expected (see picture).
>> The ZookeeperUI also works as (I would) expect it to work. I used the stack of Kafka&Zookeeper
plus all the Manager/Monitors and UI for other purposes, therefore I like to think they are
fine. :)
>> 
>> If we can figure the stack out with hello-szama it would be my pleasure to write
this up nicely and create a blog post, add it to the samza documentation and showcase samza
as a BigData project during my FOSDEM talk (https://fosdem.org/2016/schedule/track/hpc,_big_data_and_data_science/
<https://fosdem.org/2016/schedule/track/hpc,_big_data_and_data_science/>).
>> 
>> Cheers
>> Christian
>> 
>> 
>> <Screen Shot 2016-01-14 at 09.45.48.png>
>> 
>> --
>> 
>> <Gaikai_Logo.png>
>> 
>> 
>> Christian Kniep  |  Release Engineer
>> 
>> www.gaikai.com <http://www.gaikai.com/>
>>> On 14 Jan 2016, at 06:42, Yi Pan <nickpan47@gmail.com <mailto:nickpan47@gmail.com>>
wrote:
>>> 
>>> Hi, Christian,
>>> 
>>> This is interesting area (i.e. creating docker image for hello-samza). I did
not see the container log, which I assume that would include the error message? How did you
hello-samza job fail?
>>> 
>>> -Yi
>>> 
>>> On Tue, Jan 12, 2016 at 8:56 AM, Christian Kniep <ckniep@gaikai.com <mailto:ckniep@gaikai.com>>
wrote:
>>> Hey there,
>>> 
>>> I try to replay the hello-samza stuff with my docker image, but failed. :(
>>> Could someone provide me a hint what I do wrong?
>>> 
>>> I have to admit, that I am not the biggest Java developer, therefore I might
do stupid mistakes… :)
>>> 
>>> My Steps: https://github.com/ChristianKniep/orchestra/tree/master/samza <https://github.com/ChristianKniep/orchestra/tree/master/samza>
>>> 
>>> Cheers
>>> Christian
>>> 
>>> --
>>> 
>>> <Gaikai_Logo.png>
>>> 
>>> 
>>> Christian Kniep  |  Release Engineer
>>> 
>>> www.gaikai.com <http://www.gaikai.com/>
> 


Mime
View raw message