I've not played with GCS, and have some noted concerns about test coverage ( https://github.com/GoogleCloudPlatform/bigdata-interop/pull/40 ) , but assuming you are not hitting
any specific problems, it should be a matter of having the input as
gs://bucket/path and dest s3a://bucket-on-s3/path2
Using spark is interoperability communication between two
in my use case i need to take Google store as input to spark and do some
processing and finally needs to store in S3 and my spark engine runs on AWS
Please let me back is there any way for this kind of usecase bu using
directly spark without any middle components and share the info or link if
You'll need the google storage JARs on your classpath, along with those needed for S3n/s3a.
1. little talk on the topic, though I only play with azure and s3
2. some notes; bear in mind that the s3a performance tuning covered relates to things surfacing in Hadoop 2.8, which you probably wont have.
A one line test for s3 installed is can you read the landsat CSV file
this should work from wherever you are if your classpath and credentials are set up