airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Van Boxel <>
Subject Airflow pluggable components : example Logging
Date Fri, 24 Feb 2017 09:47:49 GMT
Hi all,

I'm doing some work to get Google Cloud Logging into Airflow, but without
it actually being implemented into Airflow. For this I used a simple
pluggable discovery system using standard Python setuptools. Here is what
I'm talking about:

In Airflow I reworked all longing config in an AirflowInternalLogging
component. The AirflowInternalLogging is written as a discoverable
compoment and needed to be added to the

    'airflow.logging': [
        'AirflowInternalLogging = airflow.defaults:AirflowInternalLogging'

That's all that is needed to make it discoverable. In the code I just need
to load it via:

def load_airflow_component(group, name):
    from pkg_resources import iter_entry_points
    print "Discovery of '{}' components".format(group)
    for entry_point in iter_entry_points(group=group, name=None):
        if == name:
            component = entry_point
        print("- {}".format(entry_point))
    print "Loading '{}' for '{}'".format(name, group)
    print component
    instance = component.load()
    return instance()

That's it. Now in a completly other project with it's own setup I can add
other implementations (with the same methods) of the component and make
them selectable via the config of that project:

          'airflow.logging': [
              'GoogleCloudLogging =

I would generatlise this so we could make other parts (like for example
connections) also pluggable.

What do you think. Seems to work fine so far... and doesn't add any


_/ Alex Van Boxel

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message