spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Information Technologies>
Subject Looking for a developer to help us with a small ETL project using Spark and Kubernetes
Date Thu, 18 Jul 2019 22:47:04 GMT

We are looking for a developer to help us with a small ETL project using Spark and Kubernetes.
Here are some of the requirements:

1. We need a REST API to run and schedule jobs. We would prefer this done in Node.js but can
be done using Java. The REST API will not be available to the public.
2. We need an easy way to create new jobs in Java without deploying the whole server again.
3. We want jobs deployed/ran using Kubernetes.
4. Must be able to scale to 1000s of ETL jobs.
5. Source for data will be one REST API.
6. Destination for data will be one Couchbase Database cluster. (Couchbase also uses a REST
7. I am not sure how many records will be processed per job.
8. The data is mostly sales related data.

I know there are commercial ETL solutions that do everything I want. We are looking for something
simple and do not need a fancy UI to describe our ETL. We want to use Spark and Java to programmatically
describe out ETL jobs.

Please let me know if you are interested.


Warren Bell


 email and any files transmitted with it are confidential and intended 

solely for the use of the individual or entity to whom they are 

addressed. They may not be disseminated or distributed to persons or 

entities other than the ones intended without the authority of the 
If you have received this email in error or are not the
recipient, you may not use, copy, disseminate or distribute 
it. Delete it 
immediately from your system and notify the sender 
 by email that 
you have done so. This footnote also confirms that this 
email message has 
been scanned for the presence of computer viruses.


Please consider the environment before printing

To unsubscribe e-mail:

View raw message