beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Work logged] (BEAM-4511) Create a tox environment that uses Py3 interpreter for pre/post commit test suites, once codebase supports Py3.
Date Wed, 05 Sep 2018 15:29:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-4511?focusedWorklogId=141331&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141331
]

ASF GitHub Bot logged work on BEAM-4511:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 05/Sep/18 15:28
            Start Date: 05/Sep/18 15:28
    Worklog Time Spent: 10m 
      Work Description: aaltay closed pull request #6266: [BEAM-4511] added py3 tox env for
first test
URL: https://github.com/apache/beam/pull/6266
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/__init__.py b/sdks/python/apache_beam/__init__.py
index 8bb1a53d9d0..569d6c4770c 100644
--- a/sdks/python/apache_beam/__init__.py
+++ b/sdks/python/apache_beam/__init__.py
@@ -75,9 +75,11 @@
 from __future__ import absolute_import
 
 import sys
+import os
 
 
-if not (sys.version_info[0] == 2 and sys.version_info[1] == 7):
+if not ((sys.version_info[0] == 2 and sys.version_info[1] == 7) or
+        (sys.version_info[0] == 3 and os.environ['BEAM_EXPERIMENTAL_PY3'])):
   raise RuntimeError(
       'The Apache Beam SDK for Python is supported only on Python 2.7. '
       'It is not supported on Python ['+ str(sys.version_info) + '].')
diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index 0423cc01b90..a32bd91273d 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -127,7 +127,7 @@ task testGcp(dependsOn: 'setupVirtualenv') {
 }
 test.dependsOn testGcp
 
-task testPython(dependsOn: 'setupVirtualenv') {
+task testPython2(dependsOn: 'setupVirtualenv') {
   doLast {
     exec {
       executable 'sh'
@@ -135,7 +135,17 @@ task testPython(dependsOn: 'setupVirtualenv') {
     }
   }
 }
-test.dependsOn testPython
+test.dependsOn testPython2
+
+task testPython3(dependsOn: 'setupVirtualenv') {
+  doLast {
+    exec {
+      executable 'sh'
+      args '-c', ". ${envdir}/bin/activate && tox ${tox_opts} -e py3"
+    }
+  }
+}
+test.dependsOn testPython3
 
 task testCython(dependsOn: 'setupVirtualenv') {
   doLast {
@@ -149,7 +159,7 @@ test.dependsOn testCython
 // Ensure that testCython runs exclusively to other tests. This line is not
 // actually required, since gradle doesn't do parallel execution within a
 // project.
-testCython.mustRunAfter testPython, testGcp
+testCython.mustRunAfter testPython2, testGcp
 
 task docs(dependsOn: 'setupVirtualenv') {
   doLast {
@@ -173,7 +183,8 @@ task cover(dependsOn: 'setupVirtualenv') {
 task preCommit() {
   dependsOn "docs"
   dependsOn "testCython"
-  dependsOn "testPython"
+  dependsOn "testPython2"
+  dependsOn "testPython3"
   dependsOn "testGcp"
   dependsOn "lint"
 }
diff --git a/sdks/python/gen_protos.py b/sdks/python/gen_protos.py
index 34c6f748344..5314bed590d 100644
--- a/sdks/python/gen_protos.py
+++ b/sdks/python/gen_protos.py
@@ -17,6 +17,7 @@
 
 """Generates Python proto modules and grpc stubs for Beam protos."""
 from __future__ import absolute_import
+from __future__ import print_function
 
 import glob
 import logging
@@ -71,10 +72,11 @@ def generate_proto_files(force=False):
       raise RuntimeError(
           'No proto files found in %s.' % proto_dirs)
 
-  # Regenerate iff the proto files are newer.
+  # Regenerate iff the proto files or this file are newer.
   elif force or not out_files or len(out_files) < len(proto_files) or (
       min(os.path.getmtime(path) for path in out_files)
-      <= max(os.path.getmtime(path) for path in proto_files)):
+      <= max(os.path.getmtime(path)
+             for path in proto_files + [os.path.realpath(__file__)])):
     try:
       from grpc_tools import protoc
     except ImportError:
@@ -111,14 +113,17 @@ def generate_proto_files(force=False):
             'Protoc returned non-zero status (see logs for details): '
             '%s' % ret_code)
 
-    if sys.version_info[0] >= 3:
-      ret_code = subprocess.call(
-          ["futurize", "--both-stages", "--write", "--verbose", "--no-diff",
-           out_dir])
+    ret_code = subprocess.call(["pip", "install", "future==0.16.0"])
+    if ret_code:
+      raise RuntimeError(
+          'Error installing future during proto generation')
 
-      if ret_code:
-        raise RuntimeError(
-            'Error applying futurize to generated protobuf python files.')
+    ret_code = subprocess.call(
+        ["futurize", "--both-stages", "--write", "--verbose", "--no-diff",
+         out_dir])
+    if ret_code:
+      raise RuntimeError(
+          'Error applying futurize to generated protobuf python files.')
 
 
 # Though wheels are available for grpcio-tools, setup_requires uses
diff --git a/sdks/python/setup.py b/sdks/python/setup.py
index e397b982159..4a3439e6e39 100644
--- a/sdks/python/setup.py
+++ b/sdks/python/setup.py
@@ -22,6 +22,7 @@
 
 import os
 import platform
+import sys
 import warnings
 from distutils.version import StrictVersion
 
@@ -100,9 +101,15 @@ def get_version():
   except ImportError:
     cythonize = lambda *args, **kwargs: []
 
+REQUIRED_PACKAGES_PY2_ONLY = [
+    'avro>=1.8.1,<2.0.0'
+]
+
+REQUIRED_PACKAGES_PY3_ONLY = [
+    'avro-python3>=1.8.1,<2.0.0'
+]
 
 REQUIRED_PACKAGES = [
-    'avro>=1.8.1,<2.0.0',
     'crcmod>=1.7,<2.0',
     'dill>=0.2.6,<=0.2.8.2',
     'fastavro==0.21.4',
@@ -139,6 +146,11 @@ def get_version():
     'google-cloud-bigquery==0.25.0',
 ]
 
+if sys.version_info[0] == 2:
+  REQUIRED_PACKAGES = REQUIRED_PACKAGES + REQUIRED_PACKAGES_PY2_ONLY
+elif sys.version_info[0] >= 3:
+  REQUIRED_PACKAGES = REQUIRED_PACKAGES + REQUIRED_PACKAGES_PY3_ONLY
+
 
 # We must generate protos after setup_requires are installed.
 def generate_protos_first(original_cmd):
diff --git a/sdks/python/tox.ini b/sdks/python/tox.ini
index 2c344968c7b..f425ba7673b 100644
--- a/sdks/python/tox.ini
+++ b/sdks/python/tox.ini
@@ -17,7 +17,7 @@
 
 [tox]
 # new environments will be excluded by default unless explicitly added to envlist.
-envlist = py27,py27-{gcp,cython,lint,lint3},py3-lint,docs
+envlist = py27,py3,py27-{gcp,cython,lint,lint3},py3-lint,docs
 toxworkdir = {toxinidir}/target/.tox
 
 [pycodestyle]
@@ -53,6 +53,16 @@ commands =
   python setup.py nosetests
   {toxinidir}/scripts/run_tox_cleanup.sh
 
+[testenv:py3]
+setenv =
+  BEAM_EXPERIMENTAL_PY3=1
+commands =
+  python --version
+  pip --version
+  {toxinidir}/scripts/run_tox_cleanup.sh
+  python setup.py nosetests --tests apache_beam.runners.direct.direct_metrics_test:DirectMetricsTest.test_combiner_functions
+  {toxinidir}/scripts/run_tox_cleanup.sh
+
 [testenv:py27-cython]
 # cython tests are only expected to work in linux (2.x and 3.x)
 # If we want to add other platforms in the future, it should be:


 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 141331)
    Time Spent: 1h 50m  (was: 1h 40m)

> Create a tox environment that uses Py3 interpreter for pre/post commit test suites, once
codebase supports Py3. 
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: BEAM-4511
>                 URL: https://issues.apache.org/jira/browse/BEAM-4511
>             Project: Beam
>          Issue Type: Sub-task
>          Components: sdk-py-core
>            Reporter: Valentyn Tymofieiev
>            Assignee: Matthias Feys
>            Priority: Major
>          Time Spent: 1h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message