spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Felix Cheung <felixche...@apache.org>
Subject Re: [discuss] SparkR CRAN feasibility check server problem
Date Fri, 02 Nov 2018 04:50:54 GMT
Thanks for being this up and much appreciate with keeping on top of this at
all times.

Can upgrading R able to fix the issue. Is this perhaps  not necessarily
malform but some new format for new versions perhaps? Anyway we should
consider upgrading R version if that fixes the problem.

As an option we could also disable the repo check in Jenkins but I can see
that could also be problematic.


On Thu, Nov 1, 2018 at 7:35 PM Hyukjin Kwon <gurwls223@gmail.com> wrote:

> Hi all,
>
> I want to raise the CRAN failure issue because it started to block Spark
> PRs time to time. Since the number
> of PRs grows hugely in Spark community, this is critical to not block
> other PRs.
>
> There has been a problem at CRAN (See
> https://github.com/apache/spark/pull/20005 for analysis).
> To cut it short, the root cause is malformed package info from
> https://cran.r-project.org/src/contrib/PACKAGES
> from server side, and this had to be fixed by requesting it to CRAN
> sysaadmin's help.
>
> https://issues.apache.org/jira/browse/SPARK-24152 <- newly open. I am
> pretty sure it's the same issue
> https://issues.apache.org/jira/browse/SPARK-25923 <- reopen/resolved 2
> times
> https://issues.apache.org/jira/browse/SPARK-22812
>
> This happened 5 times for roughly about 10 months, causing blocking almost
> all PRs in Apache Spark.
> Historically, it blocked whole PRs for few days once, and whole Spark
> community had to stop working.
>
> I assume this has been not a super big big issue so far for other projects
> or other people because apparently
> higher version of R has some logics to handle this malformed documents (at
> least I verified R 3.4.0 works fine).
>
> For our side, Jenkins has low R version (R 3.1.1 if that's not updated
> from what I have seen before),
> which is unable to parse the malformed server's response.
>
> So, I want to talk about how we are going to handle this. Possible
> solutions are:
>
> 1. We should start a talk with CRAN sysadmin to permanently prevent this
> issue
> 2. We upgrade R to 3.4.0 in Jenkins (however we will not be able to test
> low R versions)
> 3. ...
>
> If if we fine, I would like to suggest to forward this email to CRAN
> sysadmin to discuss further about this.
>
> Adding Liang-Chi Felix and Shivaram who I already talked about this few
> times before.
>
> Thanks all.
>
>
>
>

Mime
View raw message