spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rachana Srivastava <>
Subject S3-SQS vs Auto Loader With Apache Spark Structured Streaming
Date Sun, 20 Dec 2020 14:40:34 GMT

Problem Statement: I want to read files from S3 write files to s3 using Spark Structured Streaming.
I looked at the reference architecture recommended by Spark team that recommends using S3
-> SNS -> SQS using S3-SQS file source.

   - S3-SQS file source: Is S3-SQS file source available in Apache Spark? Do we need to use
apache Bahir's SQS implementation
   - Auto Loader: This article recommends that we should use Auto Loader. Is Auto Loader available
from Apache Spark

View raw message