spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hyukjin Kwon <gurwls...@gmail.com>
Subject Re: Tests failing with run-tests.py SyntaxError
Date Sat, 29 Jul 2017 02:27:01 GMT
Or maybe in https://github.com/apache/spark/blob/master/dev/run-tests#L23

On 29 Jul 2017 11:16 am, "Hyukjin Kwon" <gurwls223@gmail.com> wrote:

I am sorry for saying just based on my wild guess because I have no way to
check and take a look into Jenkins but I think we might have to set the
explicit Python version in https://github.com/apache/
spark/blob/master/dev/run-tests-jenkins#L29

I guess we set the explicit Python version for running unit tests but it
looks the script itself is dependent python executable in the path.



On 29 Jul 2017 7:39 am, "Dong Joon Hyun" <dhyun@hortonworks.com> wrote:

I saw that error in the latest branch-2.1 build failure, too.



https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Tes
t%20(Dashboard)/job/spark-branch-2.1-test-sbt-hadoop-2.7/579/console



But, the code was written in Jan 2016. Didn’t we run it on Python 2.6
without any problem?



ee74498de37 (Josh Rosen       2016-01-26 14:20:11 -0800 124)         {m:
set(m.dependencies).intersection(modules_to_test) for m in
modules_to_test}, sort=True)



Bests,

Dongjoon.



*From: *Hyukjin Kwon <gurwls223@gmail.com>
*Date: *Friday, July 28, 2017 at 7:06 AM
*To: *Sean Owen <sowen@cloudera.com>
*Cc: *dev <dev@spark.apache.org>
*Subject: *Re: Tests failing with run-tests.py SyntaxError



Yes, that's my guess just given information here without a close look.



On 28 Jul 2017 11:03 pm, "Sean Owen" <sowen@cloudera.com> wrote:

I see, does that suggest that a machine has 2.6, when it should use 2.7?



On Fri, Jul 28, 2017 at 2:58 PM Hyukjin Kwon <gurwls223@gmail.com> wrote:

That looks appearently due to dict comprehension which is, IIRC, not
allowed in Python 2.6.x. I checked the release note for sure before -
https://issues.apache.org/jira/browse/SPARK-20149



On 28 Jul 2017 9:56 pm, "Sean Owen" <sowen@cloudera.com> wrote:

  File "./dev/run-tests.py", line 124

    {m: set(m.dependencies).intersection(modules_to_test) for m in
modules_to_test}, sort=True)

                                                            ^

SyntaxError: invalid syntax


It seems like tests are failing intermittently with this type of error,
which leads me to guess there's some difference in the Python interpreter
on one or more machines but not all (?)


Does the error suggest anything to anyone who knows Python better than I?



The line has been around for a year so I don't think it's the script per se.

Mime
View raw message