spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rachana Srivastava <rachanasrivas...@yahoo.com.INVALID>
Subject S3-SQS vs Auto Loader With Apache Spark Structured Streaming
Date Sun, 20 Dec 2020 14:40:34 GMT

Problem Statement: I want to read files from S3 write files to s3 using Spark Structured Streaming.
I looked at the reference architecture recommended by Spark team that recommends using S3
-> SNS -> SQS using S3-SQS file source.

Question:
   
   - S3-SQS file source: Is S3-SQS file source available in Apache Spark? Do we need to use
apache Bahir's SQS implementation https://github.com/apache/bahir/tree/master/sql-streaming-sqs
   - Auto Loader: This article recommends that we should use Auto Loader. Is Auto Loader available
from Apache Spark https://docs.databricks.com/spark/latest/structured-streaming/sqs.html





Mime
View raw message