spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aram Morcecian <>
Subject SparkDF writing null values when not to database
Date Wed, 22 Sep 2021 14:12:47 GMT
Hi everyone, I'm facing something weird. After doing some transformations
to a SparkDF I print some rows and I see those perfectly, but when I write
that to the database (postgres) one of the transformed columns is written
with all null values in it.
I tried doing it with DynamicFrames (AWS Glue) and it works ok, but since I
need to overwrite the table that I'm writing on, I need to use SparkDF.

I'm attaching the code that I'm using to do this. Could someone help
me with this?

Thank you

View raw message