spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From anny9699 <>
Subject sc.textFile can't recognize '\004'
Date Sat, 21 Jun 2014 00:08:11 GMT

I need to parse a file which is separated by a series of separators. I used
SparkContext.textFile and I met two problems:

1) One of the separators is '\004', which could be recognized by python or R
or Hive, however Spark seems can't recognize this one and returns a symbol
looking like '?'. Also this symbol is not a question mark and I don't know
how to parse.

2) Some of the separator are composed of several Chars, like "} =>". If I
use str.split(Array('}', '=>')), it will separate the string but with many
white spaces included in the middle. Is there a good way that I could
separate by String instead of by Array of Chars? 

Thanks a lot!

View this message in context:
Sent from the Apache Spark User List mailing list archive at

View raw message