spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohammed Guller <>
Subject RE: JDBC Dialect for saving DataFrame into Vertica Table
Date Thu, 26 May 2016 22:09:23 GMT
Vertica also provides a Spark connector. It was not GA the last time I looked at it, but available
on the Vertica community site. Have you tried using the Vertica Spark connector instead of
the JDBC driver?

Author: Big Data Analytics with Spark<>

From: Aaron Ilovici []
Sent: Thursday, May 26, 2016 8:08 AM
Subject: JDBC Dialect for saving DataFrame into Vertica Table

I am attempting to write a DataFrame of Rows to Vertica via DataFrameWriter's jdbc function
in the following manner:

dataframe.write().mode(SaveMode.Append).jdbc(url, table, properties);

This works when there are no NULL values in any of the Rows in my DataFrame. However, when
there are rows, I get the following error:

ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 24)
java.sql.SQLFeatureNotSupportedException: [Vertica][JDBC](10220) Driver not capable.
    at com.vertica.exceptions.ExceptionConverter.toSQLException(Unknown Source)
    at com.vertica.jdbc.common.SPreparedStatement.checkTypeSupported(Unknown Source)
    at com.vertica.jdbc.common.SPreparedStatement.setNull(Unknown Source)

This appears to be Spark's attempt to set a null value in a PreparedStatement, but Vertica
does not understand the type upon executing the transaction. I see in JdbcDialects.scala that
there are dialects for MySQL, Postgres, DB2, MsSQLServer, Derby, and Oracle.

1 - Would writing a dialect for Vertica eleviate the issue, by setting a 'NULL' in a type
that Vertica would understand?
2 - What would be the best way to do this without a Spark patch? Scala, Java, make a jar and
call 'JdbcDialects.registerDialect(VerticaDialect)' once created?
3 - Where would one find the proper mapping between Spark DataTypes and Vertica DataTypes?
I don't see 'NULL' handling for any of the dialects, only the base case 'case _ => None'
- is None mapped to the proper NULL type elsewhere?

My environment: Spark 1.6, Vertica Driver 7.2.2, Java 1.7

I would be happy to create a Jira and submit a pull request with the VerticaDialect once I
figure this out.

Thank you for any insight on this,

Software Engineer
Marketing Engineering


4 Copley Place
Boston, MA 02116
(617) 532-6100 x1231<>

View raw message