airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ash Berlin-Taylor <>
Subject 1.10.3beta1 "snapshot" for available testing
Date Mon, 25 Mar 2019 19:10:55 GMT
Hi Everyone,

I'm excited to annoucne that 1.10.3 is almost ready for release.

There are a few more PRs that I'd like to get in to the release, but in the mean time I would
appreciated testing of what is there so far. If you have any changes that you would like to
see in the release then please let me know what they are and I'll try and get them included.

The release branch already contains 199 commits since 1.10.2, and has taken about three weeks
of shouting at Travis and wrangling chains of dependent commits, so I am not sure that it
is without problems. 

In order to distinguish it from an actual (apache) release it is:

1. Marked as beta (python package managers do not install beta versions by default - PEP 440)
2. It is not signed
3. It is not at an official Apache distribution location

It can be installed with:

   pip install 'apache-airflow==1.10.3b1'

(Don't worry, without asking for `--pre` or specifying the version `pip install apache-airflow`
will still get 1.10.2)

Note we don't need the annoying GPL/non-GPL env vars at install time anymore!

Included below is the changelog of this release. I will break this down into the categories
as before in time for rc1 release.


 [AIRFLOW-XXX] Version 1.10.3b1 [Ash Berlin-Taylor]
 [AIRFLOW-3650] Skip running on mysql for the flaky test (#4457) [Tao Feng]
 [AIRFLOW-XXX] Remove old/non-test files that nose ignores (#4930) [Ash Berlin-Taylor]
 [AIRFLOW-4062] Improve docs on install extra package commands (#4966) [Jiajie Zhong]
 [AIRFLOW-3423] Fix mongo hook to work with anonymous access (#4258) [Marcin Szymański]
 [AIRFLOW-3743] Unify different methods of working out AIRFLOW_HOME (#4705) [Ash Berlin-Taylor]
 [AIRFLOW-3982] Fix race condition in CI test (#4968) [Tao Feng]
 [AIRFLOW-3982] Update DagRun state based on its own tasks (#4808) [Dima Kamalov]
 [AIRFLOW-3541] Add Avro logical type conversion to bigquery hook (#4553) [BrechtDeVlieger]
 [AIRFLOW-3737] Kubernetes executor cannot handle long dag/task names (#4636) [Paul Bramhall]
 [AIRFLOW-3945] Stop inserting row when permission views unchanged (#4764) [diederikwp]
 [AIRFLOW-4123] Add Exception handling for _change_state method in K8 Executor (#4941) [andy-g14]
 [AIRFLOW-4106] instrument staving tasks in pool (#4927) [Chao-Han Tsai]
 [AIRFLOW-4122] Deprecate chain function (#4940) [Jan Hicken]
 [AIRFLOW-2568] Azure Container Instances operator (#4121) [Omeed Musavi]
 [AIRFLOW-4107] instrument executor (#4928) [Chao-Han Tsai]
 [AIRFLOW-4002] Option to open debugger on errors in `airflow test`. (#4828) [Joshua Carp]
 [AIRFLOW-3997] Extend Variable.get so it can return None when var not found (#4819) [Kristian
 [AIRFLOW-4009] Fix docstring issue in GCSToBQOperator (#4836) [Ryan Yuan]
 [AIRFLOW-3980] Unify logger (#4804) [Kamil Breguła]
 [AIRFLOW-3771] Minor refactor securityManager (#4594) [Tao Feng]
 [AIRFLOW-987] pass kerberos cli args keytab and principal to (#4238) [Iuliia
 [AIRFLOW-3736] Allow int value in SqoopOperator.extra_import_options(#4906) [kik-kik]
 [AIRFLOW-4076] Correct port type of beeline_default in init_db (#4908) [zhongjiajie]
 [AIRFLOW-4063] Fix exception string in BigQueryHook [2/2] (#4902) [RosterIn]
 [AIRFLOW-4063] Fix exception string in BigQueryHook (#4899) [RosterIn]
 [AIRFLOW-4046] Add validations for poke_interval & timeout for Sensor (#4878) [kani5hk]
 [AIRFLOW-3744] Abandon the use of obsolete aliases of methods (#4568) [Kamil Breguła]
 [AIRFLOW-4033] record stats of task duration (#4858) [Chao-Han Tsai]
 [AIRFLOW-4037] Log response in SimpleHttpOperator even if the response check fails [hixus]
 [AIRFLOW-4044] The documentation of `query_params` in `BigQueryOperator` is wrong.  (#4876)
[Hengfeng Li]
 [AIRFLOW-3865] Add API endpoint to get python code of dag by id (#4687) [raman]
 [AIRFLOW-4015] Make missing API endpoints available in classic mode [Ash Berlin-Taylor]
 [AIRFLOW-3153] Send DAG processing stats to statsd (#4748) [Andrew Stahlman]
 [AIRFLOW-3516] Support to create k8 worker pods in batches (#4434) [raman]
 [AIRFLOW-2966] Catch ApiException in the Kubernetes Executor (#4209) [johnhofman]
 [AIRFLOW-2843] Add flag in ExternalTaskSensor to check if external DAG/task exists (#4547)
 [AIRFLOW-2224] Add support CSV files in MySqlToGoogleCloudStorageOperator (#4738) [Tanay
 [AIRFLOW-XXX] Add missing GCP operators to Docs (#4260) [Kaxil Naik]
 [AIRFLOW-3892] Create Redis pub sub sensor (#4712) [mans2singh]
 [AIRFLOW-3895] GoogleCloudStorageHook/Op create_bucket takes optional resource params (#4717)
[Ole Christian Langfjæran]
 [AIRFLOW-3950] Improve AirflowSecurityManager.update_admin_perm_view (#4774) [Xiaodong]
 [AIRFLOW-4006] Make better use of Set in AirflowSecurityManager (#4833) [Xiaodong]
 [AIRFLOW-4129] Escape HTML in generated tooltips (#4950) [Ash Berlin-Taylor]
 [AIRFLOW-4070] AirflowException -> log.warning for duplicate task dependencies (#4904)
[Newton Le]
 [AIRFLOW-4124] add get_table and get_table_location in aws_glue_hook and tests (#4942) [Bryan
 [AIRFLOW-1262] Adds missing docs for email configuration (#4557) [Felix]
 [AIRFLOW-3917] Specify alternate kube config file/context when running out of cluster (#4859)
 [AIRFLOW-4054] Fix assertEqualIgnoreMultipleSpaces util & add tests (#4886) [J]
 [AIRFLOW-3239] Fix test recovery further (#4074) [Xiaodong]
 [AIRFLOW-4053] Fix KubePodOperator Xcom on Kube 1.13.0 (#4883) [Ash Berlin-Taylor]
 [AIRFLOW-3911] Change Harvesting DAG parsing results to DEBUG log level (#4729) [Nathaniel
 [AIRFLOW-2961] Refactor tests.BackfillJobTest.test_backfill_examples test (#3811) [Fokko
 [AIRFLOW-3606] Fix Flake8 test & fix the Flake8 errors introduced since Flake8 test was
broken (#4415) [Xiaodong]
 AIRFLOW-3543: Fix deletion of DAG with rescheduled tasks (#4646) [Stefan Seelmann]
 [AIRFLOW-3584] Use ORM DAGs for index view. (#4390) [Joshua Carp]
 [AIRFLOW-2548] Output plugin import errors to web UI (#3930) [Jimmy Cao]
 [AIRFLOW-2821] Refine Doc "Plugins" (#3664) [Xiaodong]
 [AIRFLOW-3561] Improve queries (#4368) [Peter van 't Hof]
 [AIRFLOW-3600] Remove dagbag from trigger (#4407) [Peter van 't Hof]
 [AIRFLOW-3713] Updated documentation for GCP optional project_id (#4541) [Jarek Potiuk]
 [AIRFLOW-3458] Deprecation path for moving models.Connection [Ash Berlin-Taylor]
 [AIRFLOW-3458] Move models.Connection into separate file (#4335) [BasPH]
 [AIRFLOW-2767] - Upgrade gunicorn to 19.5.0 to avoid moderate-severity CVE (#4795) [RosterIn]
 [AIRFLOW-3795] provide_context param is now used (#4735) [Sergio Soto]
 [AIRFLOW-4019] Fix AWS Athena Sensor object has no attribute 'mode' (#4844) [Mariko Wakabayashi]
 [AIRFLOW-4012]  - Upgrade tabulate to 0.8.3 (#4838) [OmerJog]
 [AIRFLOW-3758] Fix circular import in WasbTaskHandler (#4601) [Tanay Tummalapalli]
 [AIRFLOW-3701] Add Google Cloud Vision Product Search operators (#4665) [Szymon Przedwojski]
 [AIRFLOW-3706] Fix tooltip max-width by correcting ordering of CSS files (#4947) [Ash Berlin-Taylor]
 [AIRFLOW-4100] Correctly JSON escape data for tree/graph views (#4921) [Ash Berlin-Taylor]
 [AIRFLOW-3636] Fix a test introduced in #4425 (#4446) [Ping Zhang]
 [AIRFLOW-3623] Support download logs by attempts from UI (#4425) [Ping Zhang]
 [AIRFLOW-3977] Add examples of trigger rules in doc (#4805) [Chen Tong]
 [AIRFLOW-2511] Fix improper failed session commit handling causing deadlocks (#4769) [fenglu-g]
 [AIRFLOW-3962] Added graceful handling for creation of dag_run of a dag which doesn't have
any task (#4781) [Vardan Gupta]
 [AIRFLOW-3881] Correct to_csv row number (#4699) [zhongjiajie]
 [AIRFLOW-3875] Simplify SlackWebhookHook code and change docstring (#4696) [zhongjiajie]
 [AIRFLOW-3733] Don't raise NameError in HQL hook to_csv when no rows returned (#4560) [zhongjiajie]
 [AIRFLOW-3734] Fix hql not run when partition is None (#4561) [zhongjiajie]
 [AIRFLOW-2715] Use region setting when launching Dataflow templates (#4139) [Jan Hicken]
 [AIRFLOW-XXX] Fixing the issue in Documentation (#3756) [Santhoshkumar. P]
 [AIRFLOW-3932] Update unit tests and documentation for safe mode flag. (#4760) [Joshua Carp]
 [AIRFLOW-3932] Optionally skip dag discovery heuristic. (#4746) [Joshua Carp]
 [AIRFLOW-3258] K8S executor environment variables section. (#4627) [davlum]
 [AIRFLOW-3766] Add support for kubernetes annotations (#4589) [Stijn De Haes]
 [AIRFLOW-3767] Correct bulk insert function (#4773) [zhongjiajie]
 [AIRFLOW-3741] Add extra config to Oracle hook (#4584) [zhongjiajie]
 [AIRFLOW-XXX] Add Hint at user defined macros (#4885) [marengaz]
 [AIRFLOW-4087] remove sudo in basetaskrunner on_finish (#4916) [Junda Yang]
 [AIRFLOW-3931] set network, subnetwork when launching dataflow template (#4744) [Anthony
 [AIRFLOW-4095] Add template_fields for S3CopyObjectOperator & S3DeleteObjectsOperator
(#4920) [Y.Yaswanth Raj Kumar]
 [AIRFLOW-XXX] Correct schedule_interval in Scheduler docs (#4157) [Vincent Ketelaars]
 [AIRFLOW-3768] Escape search parameter in pagination controls (#4911) [Ash Berlin-Taylor]
 [AIRFLOW-4045] Fix hard-coded URLs in FAB-based UI (#4914) [Ash Berlin-Taylor]
 [AIRFLOW-3123] Use a stack for DAG context management (#3956) [Newton Le]
 [AIRFLOW-3060] DAG context manager fails to exit properly in certain circumstances [Newton
 [AIRFLOW-3924] Fix try number in alert emails (#4741) [Dane Laban]
 [AIRFLOW-1262] Allow configuration of email alert subject and body (#2338) [Alek Storm]
 [AIRFLOW-2798] Remove needless code from [Xiaodong]
 [AIRFLOW-4083] Add tests for link generation utils (#4912) [Ash Berlin-Taylor]
 [AIRFLOW-2190] Send correct HTTP status for base_url not found (#4910) [Ash Berlin-Taylor]
 [AIRFLOW-4015] Add get_dag_runs GET endpoint to "classic" API (#4884) [Ash Berlin-Taylor]
 [AIRFLOW-XXX] Improve airflow-jira script to make RelManager's life easier (#4857) [Ash Berlin-Taylor]
 [AIRFLOW-3731] Constrain mysqlclient to <1.4 (#4558) [Fokko Driesprong]
 [AIRFLOW-3139] include parameters into in SQL operators, if any (#3986) [Xiaodong]
 [AIRFLOW-3174] Refine Docstring for SQL Operators & Hooks (#4043) [Xiaodong]
 [AIRFLOW-3239] Enable existing CI tests (#4131) [Xiaodong]
 [AIRFLOW-1390] Update Alembic to 0.9 (#3935) [Fokko Driesprong]
 [AIRFLOW-3885] Fix race condition in scheduler test (#4737) [Andrew Stahlman]
 [AIRFLOW-3885] ~10x speed-up of SchedulerJobTest suite (#4730) [Andrew Stahlman]
 [AIRFLOW-3780] Fix some incorrect when base_url is used (#4643) [Ian Wong]
 [AIRFLOW-3807] Fix Graph View Highlighting of Tasks (#4653) [Felix]
 [AIRFLOW-XXX] Add missing class references to docs (#4644) [Kamil Breguła]
 [AIRFLOW-2985] Operators for S3 object copying/deleting (#3823) [Xiaodong]
 [AIRFLOW-XXX] Fix typo (#4564) [zhongjiajie]
 [AIRFLOW-2993] s3_to_sftp and sftp_to_s3 operators (#3828) [wmorris75]
 [AIRFLOW-3933] Fix various typos (#4747) [Ryan Yuan]
 [AIRFLOW-XXX] Add a doc about fab security (#4595) [Tao Feng]
 [AIRFLOW-3905] Allow using "parameters" in SqlSensor (#4723) [Xiaodong]
 [AIRFLOW-2761] Parallelize enqueue in celery executor (#4234) [Kevin Yang]
 [AIRFLOW-3009] Import Hashable from to fix Python 3.7 deprecation warning
(#3849) [Francis Lalonde]
 [AIRFLOW-2231] Fix relativedelta DAG schedule_interval (#3174) [brookskd]
 [AIRFLOW-2641] Fix MySqlToHiveTransfer to handle MySQL DECIMAL correctly [OmerJog]
 [AIRFLOW-3751] Option to allow malformed schemas for LDAP authentication (#4574) [Colin]
 [AIRFLOW-2888] Add deprecation path for task_runner config change (#4851) [Ash Berlin-Taylor]
 [AIRFLOW-3540] Respect environment config when looking up config file. (#4340) [Joshua Carp]
 [AIRFLOW-2930] Fix celery excecutor scheduler crash (#3784) [Yingbo Wang]
 [AIRFLOW-2888] Remove shell=True and bash from task launch (#3740) [bolkedebruin]
 [AIRFLOW-2156] Parallelize Celery Executor task state fetching (#3830) [yrqls21]
 [AIRFLOW-3885] ~2.5x speed-up for backfill tests (#4731) [Andrew Stahlman]
 [AIRFLOW-3702] Add backfill option to run backwards (#4676) [Dima Kamalov]
 [AIRFLOW-3885] ~20x speed-up of slowest unit test (#4726) [Andrew Stahlman]
 [AIRFLOW-2508] Handle non string types in Operators templatized fields (#4292) [Géraud]
 [AIRFLOW-3799] Add compose method to GoogleCloudStorageHook (#4641) [Ryan Yuan]
 [AIRFLOW-3792] Fix validation in BQ for useLegacySQL & queryParameters (#4626) [Olivier
 [AIRFLOW-3821] Add replicas logic to GCP SQL example DAG (#4662) [Antoni Smoliński]
 [AIRFLOW-3749] Fix Edit Dag Run page when using RBAC (#4613) [Felix]
 [AIRFLOW-3801] Fix DagBag collect dags invocation to prevent examples to be loaded (#4677)
 [AIRFLOW-XXX] Speed up DagBagTest cases (#3974) [Ash Berlin-Taylor]
 [AIRFLOW-3547] Fixed Jinja templating in SparkSubmitOperator (#4347) [Mathew Wicks]
 [AIRFLOW-3647] Add archives config option to SparkSubmitOperator (#4467) [Penumbra69]
 [AIRFLOW-3802] Updated documentation for HiveServer2Hook (#4647) [mans2singh]
 [AIRFLOW-3817] - Corrected task ids returned by BranchPythonOperator to match the dummy operator
ids (#4659) [mans2singh]
 [AIRFLOW-3782] Clarify docs around celery worker_autoscale in default_airflow.cfg (#4609)
 [AIRFLOW-1945] Add Autoscale config for Celery workers (#3989) [phanindhra]
 [AIRFLOW-3774] Register blueprints with RBAC web app (#4598) [Drew J. Sonne]
 AIRFLOW-3590: Change log message of executor exit status (#4616) [Stefan Seelmann]
 [AIRFLOW-3719] Handle StopIteration in CloudWatch logs retrieval (#4516) [yangaws]
 [AIRFLOW-3591] Fix start date, end date, duration for rescheduled tasks (#4502) [Stefan Seelmann]
 [AIRFLOW-3108] Define get_autocommit method for MsSqlHook (#4525) [Thales Mello]
 [AIRFLOW-3074] Add relevant ECS options to ECS operator. (#3908) [Joshua Carp]
 [AIRFLOW-3218] add support for poking a whole DAG (#4058) [Marcin Szymański]
 [AIRFLOW-3709] Validate `allowed_states` for ExternalTaskSensor (#4536) [Xiaodong]
 [AIRFLOW-3522] Add support for sending Slack attachments (#4332) [Michael Holtzscher]
 [AIRFLOW-3569] Add "Trigger DAG" button in DAG page (/www only) (#4373) [Ash Berlin-Taylor]
 [AIRFLOW-3569] Add "Trigger DAG" button in DAG page (/www_rbac only) (#4373) [Xiaodong]
 [AIRFLOW-3353] Upgrade Redis client (#4834) [Ash Berlin-Taylor]
 [AIRFLOW-3250] Fix for Redis Hook for not authorised connection calls (#4090) [Paweł Graczyk]
 [AIRFLOW-2009] Fix dataflow hook connection-id (#4563) [fenglu-g]
 [AIRFLOW-3044] Dataflow operators accept templated job_name param (#3887) [Jeff Payne]
 [AIRFLOW-3023] Fix docstring datatypes [Kaxil Naik]
 [AIRFLOW-XXX] Fix typo in [Thibault Clement]
 [AIRFLOW-2928] Use uuid4 instead of uuid1 (#3779) [Ken Kawamoto]
 [AIRFLOW-2988] Run specifically python2 for dataflow (#3826) [Jimmy Cao]
 [AIRFLOW-2190] Fix TypeError when returning 404 (#4596) [Andrew Stahlman]
 [AIRFLOW-2876] Update Tenacity to 4.12 (#3723) [Fokko Driesprong]
 [AIRFLOW-XXX] Solve lodash security warning (#4820) [Tao Feng]
 [AIRFLOW-XXX] Pin version of tornado pulled in by Celery. (#4815) [Ash Berlin-Taylor]
 [AIRFLOW-3697] Vendorize nvd3 and slugify (#4513) [Ash Berlin-Taylor]
 [AIRFLOW-3692] Remove ENV variables to avoid GPL (#4506) [bolkedebruin]
 [AIRFLOW-3923] Update flask-admin dependency to 1.5.3 to resolve security vulnerabilities
from safety (#4739) [David Smith]
 [AIRFLOW-3907] Upgrade flask and set cookie security flags. (#4725) [Joshua Carp]
 [AIRFLOW-XXX] Upgrade FAB to 1.12.3 (#4694) [Tao Feng]
 [AIRFLOW-3683] Fix formatting of error message for invalid TriggerRule (#4490) [stefan-wolfsheimer]
 [AIRFLOW-3698] Add documentation for AWS Connection (#4514) [Dana Ma]
 [AIRFLOW-3616][AIRFLOW-1215] Add aliases for schema with underscore (#4523) [Kamil Breguła]
 AIRFLOW-2787 Allow is_backfill to handle NULL DagRun.run_id (#3629) [George Leslie-Waksman]
 [AIRFLOW-XXX] Pin pinodb dependency (#4704) [Tao Feng]
 [AIRFLOW-3780] Fix some incorrect when base_url is used (#4643) [Ian Wong]
 [AIRFLOW-3639] Fix request creation in Jenkins Operator (#4450) [Maria R]
 [AIRFLOW-3779] Don't install enum34 backport when not needed (#4620) [Ash Berlin-Taylor]
 [AIRFLOW-3375] Support returning multiple tasks with BranchPythonOperator (#4215) [BasPH]
 [AIRFLOW-3742] Fix handling of "fallback" for AirflowConfigParsxer.getint/boolean (#4674)
[Tanay Tummalapalli]
 [AIRFLOW-3742] Respect the `fallback` arg in airflow.configuration.get (#4567) [Ash Berlin-Taylor]
 [AIRFLOW-3079] Improve migration scripts to support MSSQL Server (#3964) [Brad Holmes]
 [AIRFLOW-3789] Fix flake8 3.7 errors. (#4617) [Joshua Carp]
 [AIRFLOW-2735] Use equality, not identity, check for detecting AWS Batch failures[] [Craig
 [AIRFLOW-2706] AWS Batch Operator should use top-level job state to determine status [Craig
 [AIRFLOW-3602] Improve ImapHook handling of retrieving no attachments (#4475) [Felix]
 [AIRFLOW-3631] Update flake8 and fix lint. (#4436) [Joshua Carp]
 [AIRFLOW-3315] Add ImapAttachmentSensor (#4161) [Felix]
 [AIRFLOW-2780] Add IMAP Hook to retrieve email attachments (#4119) [Felix]
 [AIRFLOW-3556] Add cross join set dependency function (#4356) [BasPH]
 [AIRFLOW-1978] Add support for additional WinRM parameters [Joshua Iverson]
 [AIRFLOW-2796] Improve utils helpers code coverage (#3637) [Andy Cooper]
 [AIRFLOW-2796] Expand code coverage for utils/ (#3686) [Cameron Moberg]
 [AIRFLOW-XXX] Pin version of Pip in tests to work around pypa/pip#6163 (#4576) [Ash Berlin-Taylor]
 [AIRFLOW-XXX] Fix spark submit hook KeyError (#4578) [zhongjiajie]
 [AIRFLOW-XXX] Reduction of the number of warnings in the documentation (#4585) [Kamil Breguła]
View raw message