spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koert Kuipers <ko...@tresata.com>
Subject Re: shapeless in spark 2.1.0
Date Thu, 29 Dec 2016 21:14:19 GMT
we also use spray for webservices that execute on spark, and spray depends
on even older (and incompatible) shapeless 1.x
to get rid of the old shapeless i would have to upgrade from spray to
akka-http, which means going to java 8

this might also affect spark-job-server, which it seems uses spray.

On Thu, Dec 29, 2016 at 1:17 PM, Maciej Szymkiewicz <mszymkiewicz@gmail.com>
wrote:

> Breeze 0.13 (RC-1 right now) bumps shapeless to 2.2.0 and 2.2.5 for Scala
> 2.10 and 2.11 respectively:
>
> https://github.com/scalanlp/breeze/pull/509
>
>
> On 12/29/2016 07:13 PM, Ryan Williams wrote:
>
> Other option would presumably be for someone to make a release of breeze
> with old-shapeless shaded... unless shapeless classes are exposed in
> breeze's public API, in which case you'd have to copy the relevant
> shapeless classes into breeze and then publish that?
>
> On Thu, Dec 29, 2016, 1:05 PM Sean Owen <sowen@cloudera.com> wrote:
>
>> It is breeze, but, what's the option? It can't be excluded. I think this
>> falls in the category of things an app would need to shade in this
>> situation.
>>
>> On Thu, Dec 29, 2016, 16:49 Koert Kuipers <koert@tresata.com> wrote:
>>
>> i just noticed that spark 2.1.0 bring in a new transitive dependency on
>> shapeless 2.0.0
>>
>> shapeless is a popular library for scala users, and shapeless 2.0.0 is
>> old (2014) and not compatible with more current versions.
>>
>> so this means a spark user that uses shapeless in his own development
>> cannot upgrade safely from 2.0.0 to 2.1.0, i think.
>>
>> wish i had noticed this sooner
>>
>>
> --
> Maciej Szymkiewicz
>
>

Mime
View raw message