spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julien Champ (JIRA)" <>
Subject [jira] [Created] (SPARK-19451) Long values in Window function
Date Fri, 03 Feb 2017 17:29:51 GMT
Julien Champ created SPARK-19451:

             Summary: Long values in Window function
                 Key: SPARK-19451
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.0.2, 1.6.1
            Reporter: Julien Champ

Hi there,

there seems to be a major limitation in spark window functions and rangeBetween method.

If I have the following code :
    val tw =  Window.orderBy("date")
      .rangeBetween( from , 0)

Everything seems ok, while "from" value is not too large... Even if the rangeBetween() method
supports Long parameters.
But.... If i set "-2160000000L" value to "from" it does not work !

It is probably related to this part of code in the between() method, of the WindowSpec class,
called by rangeBetween()

    val boundaryStart = start match {
      case 0 => CurrentRow
      case Long.MinValue => UnboundedPreceding
      case x if x < 0 => ValuePreceding(-start.toInt)
      case x if x > 0 => ValueFollowing(start.toInt)
( look at this " .toInt " )

Does anybody know it there's a way to solve / patch this behavior ?

Any help will be appreciated


This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message