spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hyukjin Kwon <>
Subject [DISCUSS] Drop Python 2, 3.4 and 3.5
Date Thu, 02 Jul 2020 02:11:11 GMT
Hi all,

I would like to discuss dropping deprecated Python versions 2, 3.4 and 3.5
at I assume people support it
in general
but I am writing this to make sure everybody is happy.

Fokko made a very good investigation on it, see
Assuming from the statistics, I think we're pretty safe to drop them.
Also note that dropping Python 2 was actually declared at

Roughly speaking, there are many main advantages by dropping them:
  1. It removes a bunch of hacks we added around 700 lines in PySpark.
  2. PyPy2 has a critical bug that causes a flaky test, given my testing and
  3. Users can use Python type hints with Pandas UDFs without thinking
about Python version
  4. Users can leverage one latest cloudpickle, With Python 3.8+ it can also
leverage C pickle.
  5. ...

So it benefits both users and dev. WDYT guys?

View raw message