spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mateusz Buśkiewicz (JIRA) <j...@apache.org>
Subject [jira] [Created] (SPARK-15382) monotonicallyIncreasingId doesn't work when data is upsampled
Date Wed, 18 May 2016 12:24:12 GMT
Mateusz Buśkiewicz created SPARK-15382:
------------------------------------------

             Summary: monotonicallyIncreasingId doesn't work when data is upsampled
                 Key: SPARK-15382
                 URL: https://issues.apache.org/jira/browse/SPARK-15382
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 1.6.1
            Reporter: Mateusz Buśkiewicz


Assigned ids are not unique

{code}
from pyspark.sql import Row
from pyspark.sql.functions import monotonicallyIncreasingId

hiveContext.createDataFrame([Row(a=1), Row(a=2)]).sample(True, 10.0).withColumn('id', monotonicallyIncreasingId()).collect()
{code}

Output:

{code}
[Row(a=1, id=429496729600),
 Row(a=1, id=429496729600),
 Row(a=1, id=429496729600),
 Row(a=1, id=429496729600),
 Row(a=1, id=429496729600),
 Row(a=1, id=429496729600),
 Row(a=1, id=429496729600),
 Row(a=2, id=867583393792),
 Row(a=2, id=867583393792),
 Row(a=2, id=867583393792),
 Row(a=2, id=867583393792),
 Row(a=2, id=867583393792),
 Row(a=2, id=867583393792),
 Row(a=2, id=867583393792),
 Row(a=2, id=867583393792),
 Row(a=2, id=867583393792),
 Row(a=2, id=867583393792),
 Row(a=2, id=867583393792)]
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message