flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wei Zhong (Jira)" <j...@apache.org>
Subject [jira] [Created] (FLINK-14590) Unify the working directory of Java process and Python process when submitting python jobs via "flink run -py"
Date Fri, 01 Nov 2019 02:25:00 GMT
Wei Zhong created FLINK-14590:
---------------------------------

             Summary: Unify the working directory of Java process and Python process when
submitting python jobs via "flink run -py"
                 Key: FLINK-14590
                 URL: https://issues.apache.org/jira/browse/FLINK-14590
             Project: Flink
          Issue Type: Bug
          Components: API / Python
            Reporter: Wei Zhong


Assume we enter this flink directory with following structure:
{code:java}
flink/
      bin/
          flink
          pyflink-shell.sh
          python-gateway-server.sh
          ...
      bad_case/
               word_count.py
               data.txt
      lib/...
      opt/...{code}
 And the word_count.py has such a piece of code:
{code:java}
    t_config = TableConfig()
    env = StreamExecutionEnvironment.get_execution_environment()
    t_env = StreamTableEnvironment.create(env, t_config)

    env._j_stream_execution_environment.registerCachedFile("data", "bad_case/data.txt")

    with open("bad_case/data.txt", "r") as f:
        content = f.read()
    elements = [(word, 1) for word in content.split(" ")]
    t_env.from_elements(elements, ["word", "count"]){code}
Then we enter the "flink" directory and run:
{code:java}
bin/flink run -py bad_case/word_count.py
{code}
The program will fail at the line of "with open("bad_case/data.txt", "r") as f:".

It is because the working directory of Java process is current directory but the working directory
of Python process is a temporary directory.

So there is no problem when relative path is used in the api call to java process. But if
relative path is used in other place such as native file access, it will fail, because the
working directory of python process has been change to a temporary directory that is not known
to users.

I think it will cause some confusion for users, especially after we support dependency management.
It will be great if we unify the working directory of Java process and Python process.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message