spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Travis Crawford <>
Subject DynamoDB data source questions
Date Wed, 13 Apr 2016 14:45:35 GMT
Hi Spark gurus,

At Medium we're using Spark for an ETL job that scans DynamoDB tables and
loads into Redshift. Currently I use a parallel scanner implementation that
writes files to local disk, then have Spark read them as a DataFrame.

Ideally we could read the DynamoDB table directly as a DataFrame, so I
started putting together a data source at

A few questions:

* What's the best way to incrementally build the RDD[Row] returned by
"buildScan"? Currently I make an RDD[Row] from each page of results, and
union them together. Does this approach seem reasonable? Any suggestions
for a better way?

* Currently my stand-alone scanner creates separate threads for each scan
segment. I could use that same approach and create threads in the Spark
driver, though ideally each scan segment would run in an executor. Any tips
on how to get the segment scanners to run on Spark executors?


View raw message