Hi folks,

Thanks for the help thus far.

I'm trying to track down the source of this error:
  
java.lang.UnsupportedOperationException: org.apache.parquet.column.values.dictionary.PlainValuesDictionary

w hen doing a message.show()

Basically I'm reading in a single Parquet file (to try to narrow things down).

I'm defining the schema in the beginning and loading the parquet with:
   message = spark\
             .read\
             .schema(myMessageSchema)\
             .format("parquet")\
             .option("mergeSchema", "true")\
             .option("badRecordsPath", "/tmp/badRecords/")\
             .load("hdfs:///user/hadoop/feb20/part-00000-c6da95c9-9c40-4623-a5c5-851188e236ff-c000.snappy.parquet")

[I've tried with and without the mergeSchema option]
[ sidenote: I was hoping the badRecordPath would help with the truly bad records, but this seems to do nothing]

I've also tried to cast the potential problematic columns (so Int, Long, Double, etc) with

  message_1 = message\
    .withColumn('price', col('price').cast('double'))\
    .withColumn('price_eur', col('price_eur').cast('double'))\
    .withColumn('cost_usd', col('cost_usd').cast('double'))\
    .withColumn('adapter_status', col('adapter_status').cast('long'))

Yet I get this error and I can't figure out:
(a) whether it's some record WITHIN the parquet file that's causing it and
(b) if it is a single record (or a few records) then how do I find those particular records?

In the previous time I encountered this, there were records that should have had doubles in them (like "price" above) that actually seemed to have null.

I did this to fix that particular problem:

if not 'price' in message.columns:
    message = message.withColumn('price', message.lit('0'))

Any suggestions or help would be MOST welcome. I have also tried using pyarrow to take a look at the Parquet schema and it looks fine. I mean, it doesn't look like the schema in the parquet is the problem - but of course I'm not ruling that out just yet.

Thanks for any suggestions,

Hamish
--
Cloud-Fundis.co.za
Cape Town, South Africa
+27 79 614 4913