spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Blaž Šnuderl <snud...@gmail.com>
Subject Pyspark 2.1.0 weird behavior with repartition
Date Mon, 30 Jan 2017 19:19:02 GMT
I am loading a simple text file using pyspark. Repartitioning it seems to
produce garbage data.

I got this results using spark 2.1 prebuilt for hadoop 2.7 using pyspark
shell.

>>> sc.textFile("outc").collect()
[u'a', u'b', u'c', u'd', u'e', u'f', u'g', u'h', u'i', u'j', u'k', u'l']
>>> sc.textFile("outc", use_unicode=False).collect()
['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l']

Repartitioning seems to produce garbarge and also only only 2 records here
>>> sc.textFile("outc", use_unicode=False).repartition(10).collect()
['\x80\x02]q\x01(U\x01aU\x01bU\x01cU\x01dU\x01eU\x01fU\x01ge.',
'\x80\x02]q\x01(U\x01hU\x01iU\x01jU\x01kU\x01le.']
>>> sc.textFile("outc", use_unicode=False).repartition(10).count()
2


Without setting use_unicode=False we can't even repartition at all
>>> sc.textFile("outc").repartition(19).collect()
Traceback (most recent call last):                                                       
                                                                                         
                                                                                         
                                                                                         
    
  File "<stdin>", line 1, in <module>
  File
"/home/snuderl/scrappstore/thirdparty/spark-2.1.0-bin-hadoop2.7/python/pyspark/rdd.py",
line 810, in collect
    return list(_load_from_socket(port, self._jrdd_deserializer))
  File
"/home/snuderl/scrappstore/thirdparty/spark-2.1.0-bin-hadoop2.7/python/pyspark/rdd.py",
line 140, in _load_from_socket
    for item in serializer.load_stream(rf):
  File
"/home/snuderl/scrappstore/thirdparty/spark-2.1.0-bin-hadoop2.7/python/pyspark/serializers.py",
line 529, in load_stream
    yield self.loads(stream)
  File
"/home/snuderl/scrappstore/thirdparty/spark-2.1.0-bin-hadoop2.7/python/pyspark/serializers.py",
line 524, in loads
    return s.decode("utf-8") if self.use_unicode else s
  File
"/home/snuderl/scrappstore/virtualenv/lib/python2.7/encodings/utf_8.py",
line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x80 in position 0:
invalid start byte



Input file contents:
a
b
c
d
e
f
g
h
i
j
k
l



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Pyspark-2-1-0-weird-behavior-with-repartition-tp28350.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message