airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Van Boxel <a...@vanboxel.be>
Subject Airflow pluggable components : example Logging
Date Fri, 24 Feb 2017 09:47:49 GMT
Hi all,

I'm doing some work to get Google Cloud Logging into Airflow, but without
it actually being implemented into Airflow. For this I used a simple
pluggable discovery system using standard Python setuptools. Here is what
I'm talking about:

In Airflow I reworked all longing config in an AirflowInternalLogging
component. The AirflowInternalLogging is written as a discoverable
compoment and needed to be added to the setup.py:

entry_points={
    'airflow.logging': [
        'AirflowInternalLogging = airflow.defaults:AirflowInternalLogging'
    ]

That's all that is needed to make it discoverable. In the code I just need
to load it via:

def load_airflow_component(group, name):
    from pkg_resources import iter_entry_points
    print "Discovery of '{}' components".format(group)
    for entry_point in iter_entry_points(group=group, name=None):
        if entry_point.name == name:
            component = entry_point
        print("- {}".format(entry_point))
    print "Loading '{}' for '{}'".format(name, group)
    print component
    instance = component.load()
    return instance()

That's it. Now in a completly other project with it's own setup I can add
other implementations (with the same methods) of the component and make
them selectable via the config of that project:

setup(name='airflow_plugin.google_cloud',
      version='0.1.0',
      packages=['airflow_plugin.google_cloud'],
      entry_points={
          'airflow.logging': [
              'GoogleCloudLogging =
airflow_plugin.google_cloud:GoogleCloudLogging'
          ]
      })

I would generatlise this so we could make other parts (like for example
connections) also pluggable.

What do you think. Seems to work fine so far... and doesn't add any
dependencies.

Thoughts?



-- 
  _/
_/ Alex Van Boxel

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message