flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rmetzger <...@git.apache.org>
Subject [GitHub] flink pull request: [FLINK-377] [FLINK-671] Generic Interface / PA...
Date Mon, 02 Feb 2015 20:44:25 GMT
Github user rmetzger commented on the pull request:

    https://github.com/apache/flink/pull/202#issuecomment-72533930
  
    Wordcount with build-in data works :+1: nice.
    ```
    robert@robert-tower ...k-0.9-SNAPSHOT-bin/flink-0.9-SNAPSHOT (git)-[papipr] % ./bin/pyflink3.sh
pyflink.py - /home/robert/flink-workdir/yarnLog /tmp/yarnPyWC
    02/02/2015 21:20:34	Job execution switched to status RUNNING.
    02/02/2015 21:20:34	DataSource (TextSource)(1/1) switched to SCHEDULED 
    02/02/2015 21:20:34	DataSource (TextSource)(1/1) switched to DEPLOYING 
    02/02/2015 21:20:34	DataSource (TextSource)(1/1) switched to RUNNING 
    02/02/2015 21:20:34	MapPartition (PythonFlatMap -> PythonCombine)(1/1) switched to
SCHEDULED 
    02/02/2015 21:20:34	MapPartition (PythonFlatMap -> PythonCombine)(1/1) switched to
DEPLOYING 
    02/02/2015 21:20:34	MapPartition (PythonFlatMap -> PythonCombine)(1/1) switched to
RUNNING 
    02/02/2015 21:20:34	DataSource (TextSource)(1/1) switched to FINISHED 
    ```
    
    I wanted to run wordcount locally on some serious data, but sadly it seems that the job
somehow deadlocked.
    
    ```
    "MapPartition (PythonFlatMap -> PythonCombine) (1/1)" #85 daemon prio=5 os_prio=0 tid=0x0000000001b59000
nid=0x855 runnable [0x00007fd73b3f4000]
       java.lang.Thread.State: RUNNABLE
            at java.net.PlainDatagramSocketImpl.receive0(Native Method)
            - locked <0x00000000fad36828> (a java.net.PlainDatagramSocketImpl)
            at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:143)
            - locked <0x00000000fad36828> (a java.net.PlainDatagramSocketImpl)
            at java.net.DatagramSocket.receive(DatagramSocket.java:781)
            - locked <0x00000000fad367a0> (a java.net.DatagramPacket)
            - locked <0x00000000fad367c8> (a java.net.DatagramSocket)
            at org.apache.flink.languagebinding.api.java.common.streaming.Streamer.streamBufferWithoutGroups(Streamer.java:172)
            at org.apache.flink.languagebinding.api.java.python.functions.PythonMapPartition.mapPartition(PythonMapPartition.java:55)
            at org.apache.flink.runtime.operators.MapPartitionDriver.run(MapPartitionDriver.java:98)
            at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:496)
            at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360)
            at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204)
            at java.lang.Thread.run(Thread.java:745)
    ```
    
    ```
    "Thread-23" #86 daemon prio=5 os_prio=0 tid=0x00007fd634001800 nid=0x857 runnable [0x00007fd73b4f4000]
       java.lang.Thread.State: RUNNABLE
            at java.io.FileInputStream.readBytes(Native Method)
            at java.io.FileInputStream.read(FileInputStream.java:234)
            at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
            at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
            - locked <0x00000000fad3a440> (a java.lang.UNIXProcess$ProcessPipeInputStream)
            at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
            at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
            at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
            - locked <0x00000000fad3e968> (a java.io.InputStreamReader)
            at java.io.InputStreamReader.read(InputStreamReader.java:184)
            at java.io.BufferedReader.fill(BufferedReader.java:161)
            at java.io.BufferedReader.readLine(BufferedReader.java:324)
            - locked <0x00000000fad3e968> (a java.io.InputStreamReader)
            at java.io.BufferedReader.readLine(BufferedReader.java:389)
            at org.apache.flink.languagebinding.api.java.common.streaming.StreamPrinter.run(StreamPrinter.java:34)
    ```
    
    I have to dig further to understand whats going on.
    
    
    My understanding of the pull request right now is the following: I see that this change
was a LOT of work and that there had been some iterations of improvement.
    What the code certainly needs are a few more developers. This will probably automatically
lead to cleaner code, better code comments, better error handling and so on.
    I'm still not convinced to merge the code in the state its currently in. Therefore, I'm
just facing too many issues right now. That the example in the documentation is broken is
certainly not the dealbreaker here. Issues like hard to find error messages or the issues
I had with the wordcount (I don't know if its the runtime or an issue of the Python code)
    
    Please don't get my feedback here wrong. I appreciate and see that this has been a lot
of work, and we are probably close to 100% (we are probably in the 90s already). Whats needed
are people to test this in different environments, with different expectations etc. .. then
we'll probably quickly achieve 100% (for a first prototype!)
    
    In my opinion, we have two options now:
    - We try quickly find people (mailing lists, contacts) willing to test the Python API
and @zentol fixes these issues until we have enough confidence to merge it. 
    - We go a step back with the Python API and follow the approach similar to the streaming
system and the graph API ("gelly"): We create a separate github project where the project
can exist for some time. Such a repository would allow @zentol to easily collaborate with
other contributors.
    
    I think the two options very much depend on the number of people we find to test the python
API and the willingness of @zentol to fix the issues alone.
    If others share my concern with limited number of contributors, we should probably put
the code into a separate directory for a while.
    
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message