spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ehbhaskar <>
Subject [Spark SQL] Couldn't save dataframe with null columns to S3.
Date Tue, 06 Nov 2018 01:02:10 GMT
I have a spark job that writes data to S3 as below. \
.write.partitionBy(target_partition_cols_list) \
.format("ORC").save(self.table_location_prefix + self.target_table,

My dataframe some times can have null values for columns. Writing dataframe
with null attributes fails my job stating IllegalArgumentException as below.
Caused by: java.lang.*IllegalArgumentException: Error: type expected at the
position 14 of
but 'null' is found*.

Sample dataframe looks like this:
columns_with_default = "col1, NULL as col2, col2, col4, NULL as col5,
partition_col1, partition_col2"
source_data_df_to_write = self.session.sql(
                 "SELECT %s FROM TEMP_VIEW" % (columns_with_default))

So, is there a way to make spark job to write dataframe with NULL attributes
to S3? 

Sent from:

To unsubscribe e-mail:

View raw message