spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stone Zhong <stone.zh...@gmail.com>
Subject Cross Region Apache Spark Setup
Date Tue, 14 Apr 2020 20:31:26 GMT
Hi,

I am trying to setup a cross region Apache Spark cluster. All my data are
stored in Amazon S3 and well partitioned by region.

For example, I have parquet file at
    S3://mybucket/sales_fact.parquet/us-west
    S3://mybucket/sales_fact.parquet/us-east
    S3://mybucket/sales_fact.parquet/uk

And my cluster have nodes in us-west, us-east and uk region -- basically I
have node in all region that I supported.

When I have code like:

df = spark.read.parquet("S3://mybucket/sales_fact.parquet/*")
print(df.count()) #1
print(df.select("product_id").distinct().count()) #2

For #1, I expect only us-west nodes read data partition in us-west, and
etc, and spark to add 3 regional count and return me a total count. *I do
not expect large cross region data transfer in this case.*
For #2, I expect only us-west nodes read data partition in us-west, and
etc. Each region, do the distinct() locally first, and merge 3 "product_id"
list and do a distinct() again, I am ok with the necessary cross-region
data transfer for merging the distinct product_ids

Can anyone please share the best practice? Is it possible to config the
Apache Spark to work in such a way?

Any idea and help is appreciated!

Thanks,
Stone

Mime
View raw message