spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Driesprong, Fokko" <>
Subject Re: [DISCUSS] Migrate development scripts under dev/ from Python2 to Python 3
Date Thu, 15 Aug 2019 09:24:54 GMT
Sorry for the late reply, was a bit busy lately, but I still would like to
share my thoughts on this.

For Apache Airflow we're dropping support for Python 2 in the next major
release. We're now supporting Python 3.5+. Mostly because:

   - Easier to maintain and test, and less if/else constructions for the
   different Python versions. Also, not having to test against Python 2.x
   reduces the build matrix.
   - Python 3 has support for typing. From Python 3.5 you can include
   provisional type hints. An excellent presentation by Guido himself: From Python 3.5 it is still
   provisional, but it is a really good idea. From Airflow we've noticed that
   using mypy is catching bugs early:
      - This will put less stress on the (boring part of the) reviewing
      process since a lot of this stuff is checked automatically.
      - For new developers, it is easier to read the code because of the
      - Can be used as an input for generated documentation (or check if it
      still in sync with the docstrings)
      - Easier to extend the code since you know what kind of types you can
      expect, and your IDE will also pick up the hinting.
   - Python 2.x will be EOL end this year

I have a strong preference to migrate everything to Python 3.

Cheers, Fokko

Op wo 7 aug. 2019 om 12:14 schreef Weichen Xu <>:

> All right we could support both Python 2 and Python 3 for spark 3.0.
> On Wed, Aug 7, 2019 at 6:10 PM Hyukjin Kwon <> wrote:
>> We didn't drop Python 2 yet although it's deprecated. So I think It
>> should support both Python 2 and Python 3 at the current status.
>> 2019년 8월 7일 (수) 오후 6:54, Weichen Xu <>님이
>>> Hi all,
>>> I would like to discuss the compatibility for dev scripts. Because we
>>> already decided to deprecate python2 in spark 3.0, for development scripts
>>> under dev/ , we have two choice:
>>> 1) Migration from Python 2 to Python 3
>>> 2) Support both Python 2 and Python 3
>>> I tend to option (2) which is more friendly to maintenance.
>>> Regards,
>>> Weichen

View raw message