Hello all, 
we have a hadoop cluster (using yarn) using  s3 as filesystem with s3guard is enabled. 
We are using hadoop 3.2.1 with spark 2.4.5. 

When I try to save a dataframe in parquet format, I get the following exception: 
java.lang.ClassNotFoundException: com.hortonworks.spark.cloud.commit.PathOutputCommitProtocol

My relevant spark configurations are as following: 
"fs.s3a.committer.name": "magic",
"fs.s3a.committer.magic.enabled": true,
"fs.s3a.impl": "org.apache.hadoop.fs.s3a.S3AFileSystem",

While spark streaming fails with the exception above, apache beam succeeds writing parquet files. 
What might be the problem?

Thanks in advance 

"Talkers aren’t good doers. Rest assured that we’re going there to use our hands, not our tongues."
W. Shakespeare