airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kamil Olszewski <kamil.olszew...@polidea.com>
Subject Generic Transfer Operator
Date Tue, 18 Aug 2020 11:26:32 GMT
Hello all,
in Polidea we have come up with an idea for a generic transfer operator
that would be able to transport data between two destinations of various
types (file, database, storage, etc.) - please find the link with a short
doc with POC
<https://docs.google.com/document/d/1o7Ph7RRNqLWkTbe7xkWjb100eFaK1Apjv27LaqHgNkE/edit?usp=sharing>
where we can discuss the design initially. Once we come to the initial
conclusion I can create an AIP on cWiki - can I ask for permission to do so
(my id is 'kamil.olszewski')? I believe that during the discussion we
should definitely aim for this feature to be released only after Airflow
2.0 is out.

What do you think about this idea? Would you find such an operator helpful
in your pipelines? Maybe you already use a similar solution or know
packages that could be used to implement it?

Best regards,
-- 

Kamil Olszewski
Polidea <https://www.polidea.com> | Software Engineer

M: +48 503 361 783
E: kamil.olszewski@polidea.com

Unique Tech
Check out our projects! <https://www.polidea.com/our-work>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message