airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Driesprong, Fokko" <fo...@driesprong.frl>
Subject Re: 1.9.0 test branch has been cut
Date Mon, 18 Sep 2017 13:04:10 GMT
Hi all,

I'm working on the Airflow logging. Got a few PR's in there, but there
still a few open ends that I would like to have into 1.9:
AIRFLOW-1604 <https://issues.apache.org/jira/browse/AIRFLOW-1604> is mostly
cosmetic and almost finished. I think it is important to get AIRFLOW-1611
<https://issues.apache.org/jira/browse/AIRFLOW-1611> in because this will
greatly simplify the configuration of the logging. In extend of the these,
ticket AIRFLOW-1605 <https://issues.apache.org/jira/browse/AIRFLOW-1605> is
also required as this will correctly initialise the log.

Cheers, Fokko


2017-09-13 23:07 GMT+02:00 Chris Riccomini <criccomini@apache.org>:

> Hey all,
>
> Figured out the issue with the JIRA list at the top of this email. There's
> a bug in the dev/airflow-jira script that limits the results to a max of
> 50. Here's the total list of everything that's slated for 1.9.0:
>
> ISSUE ID     |DESCRIPTION                                       |MERGED
> AIRFLOW-1608 |GCP Dataflow hook missing pending job state       |1
> AIRFLOW-1606 |DAG.sync_to_db is static, but takes a DAG as first|1
> AIRFLOW-1605 |Fix log source of local loggers                   |0
> AIRFLOW-1602 |Use LoggingMixin for the DAG class                |1
> AIRFLOW-1597 |Add GameWisp as Airflow user                      |1
> AIRFLOW-1594 |Installing via pip copies test files into python l|1
> AIRFLOW-1593 |Expose load_string in WasbHook                    |1
> AIRFLOW-1586 |MySQL to GCS to BigQuery fails for tables with dat|1
> AIRFLOW-1584 |Remove the insecure /headers endpoints            |1
> AIRFLOW-1582 |Improve logging structure of Airflow              |1
> AIRFLOW-1580 |Error in string formatter when throwing an excepti|1
> AIRFLOW-1579 |Allow jagged rows in BQ Hook.                     |1
> AIRFLOW-1577 |Add token support to DatabricksHook               |1
> AIRFLOW-1573 |Remove `thrift < 0.10.0` requirement              |1
> AIRFLOW-1568 |Add datastore import/export operator              |1
> AIRFLOW-1567 |Clean up ML Engine operators                      |1
> AIRFLOW-1564 |Default logging filename contains a colon         |1
> AIRFLOW-1556 |BigQueryBaseCursor should support SQL parameters  |1
> AIRFLOW-1546 | add Zymergen to org list in README               |1
> AIRFLOW-1535 |Add support for Dataproc serviceAccountScopes in D|1
> AIRFLOW-1529 |Support quoted newlines in Google BigQuery load jo|1
> AIRFLOW-1525 |Fix minor LICENSE & NOTICE issue                  |0
> AIRFLOW-1522 |Increase size of val column for variable table in |1
> AIRFLOW-1521 |Template fields definition for bigquery_table_dele|1
> AIRFLOW-1507 |Make src, dst and bucket parameters as templated i|1
> AIRFLOW-1505 |Document when Jinja substitution occurs           |1
> AIRFLOW-1504 |Log Cluster Name on Dataproc Operator When Execute|1
> AIRFLOW-1499 s|Eliminate duplicate and unneeded code             |0
> AIRFLOW-1493 |Fix race condition with airflow run               |1
> AIRFLOW-1492 |Add metric for task success/failure               |1
> AIRFLOW-1489 |Docs: Typo in BigQueryCheckOperator               |1
> AIRFLOW-1478 |Chart -> Owner column should be sortable          |1
> AIRFLOW-1476 |Add INSTALL file for source releases              |1
> AIRFLOW-1474 |Add dag_id regex for 'airflow clear' CLI command  |1
> AIRFLOW-1459 |integration rst doc is broken in github view      |1
> AIRFLOW-1438 |Scheduler batch queries should have a limit       |1
> AIRFLOW-1437 |BigQueryTableDeleteOperator should define deletion|1
> AIRFLOW-1402 |Cleanup SafeConfigParser DeprecationWarning       |1
> AIRFLOW-1401 |Standardize GCP project, region, and zone argument|1
> AIRFLOW-1394 |Add quote_character parameter to GoogleCloudStorag|1
> AIRFLOW-1389 |BigQueryOperator should support `createDisposition|1
> AIRFLOW-1384 |Add ARGO/CaDC                                     |1
> AIRFLOW-1359 |Provide GoogleCloudML operator for model evaluatio|1
> AIRFLOW-1352 |Revert bad logging Handler                        |0
> AIRFLOW-1350 |Add "query_uri" parameter for Google DataProc oper|1
> AIRFLOW-1345 |Don't commit on each loop                         |1
> AIRFLOW-1344 |Builds failing on Python 3.5 with AttributeError  |1
> AIRFLOW-1343 |Add airflow default label to the dataproc operator|1
> AIRFLOW-1338 |gcp_dataflow_hook is incompatible with the recent |1
> AIRFLOW-1337 |Customize log format via config file              |1
> AIRFLOW-1335 |Use buffered logger                               |1
> AIRFLOW-1333 |Enable copy function for Google Cloud Storage Hook|1
> AIRFLOW-1324 |Make the Druid operator/hook more general         |1
> AIRFLOW-1301 |Add New Relic to Airflow user list                |1
> AIRFLOW-1299 |Google Dataproc cluster creation operator should s|1
> AIRFLOW-1289 |Don't restrict scheduler threads to CPU cores     |1
> AIRFLOW-1286 |BaseTaskRunner - Exception TypeError: a bytes-like|1
> AIRFLOW-1277 |Forbid creation of a known event with empty fields|1
> AIRFLOW-1276 |Forbid event creation with end_data earlier than s|1
> AIRFLOW-1275 |Fix `airflow pool` command exception              |1
> AIRFLOW-1274 |HttpSensor parameter params is overriding BaseOper|1
> AIRFLOW-1273 |Google Cloud ML Version and Model CRUD Operator   |1
> AIRFLOW-1272 |Google Cloud ML Batch Prediction Operator         |1
> AIRFLOW-1271 |Google Cloud ML Training Operator                 |1
> AIRFLOW-1256 |Add United Airlines as Airflow user               |1
> AIRFLOW-1251 |Add eRevalue as an Airflow user                   |1
> AIRFLOW-1248 |Fix inconsistent configuration name for worker tim|1
> AIRFLOW-1247 |CLI: ignore all dependencies argument ignored     |1
> AIRFLOW-1245 |Fix random failure of test_trigger_dag_for_date un|1
> AIRFLOW-1244 |Forbid creation of a pool with empty name         |1
> AIRFLOW-1242 |BigQueryHook assumes that a valid project_id can't|1
> AIRFLOW-1237 |Fix IN-predicate sqlalchemy warning               |1
> AIRFLOW-1234 |Cover utils.operator_helpers with unit tests      |1
> AIRFLOW-1233 |Cover utils.json with unit tests                  |1
> AIRFLOW-1232 |Remove deprecated readfp warning                  |1
> AIRFLOW-1231 |Use flask_wtf.CSRFProtect instead of flask_wtf.Csr|1
> AIRFLOW-1221 |Fix DatabricksSubmitRunOperator Templating        |1
> AIRFLOW-1217 |Enable logging in Sqoop hook                      |1
> AIRFLOW-1213 |Add hcatalog parameters to the sqoop operator/hook|1
> AIRFLOW-1208 |Speed-up cli tests                                |1
> AIRFLOW-1207 |Enable utils.helpers unit tests                   |1
> AIRFLOW-1203 |Tests failing after oauth upgrade                 |1
> AIRFLOW-1201 |Update deprecated 'nose-parameterized' library to |1
> AIRFLOW-1198 |HDFSOperator to operate HDFS                      |0
> AIRFLOW-1197 |Contrib Spark Submit Hook on_kill error           |1
> AIRFLOW-1193 |Add Checkr to Airflow user list                   |1
> AIRFLOW-1191 |Contrib Spark Submit hook should permit override o|1
> AIRFLOW-1189 |Get pandas DataFrame using BigQueryHook fails     |1
> AIRFLOW-1188 |Add max_bad_records param to GoogleCloudStorageToB|1
> AIRFLOW-1187 |Obsolete package names in documentation           |0
> AIRFLOW-1185 |Incorrect url to PyPi                             |1
> AIRFLOW-1182 |Contrib Spark Submit operator should template fiel|1
> AIRFLOW-1181 |Enable delete and list function for Google Cloud S|1
> AIRFLOW-1179 |Pandas 0.20 broke Google BigQuery hook            |1
> AIRFLOW-1175 |Add Pronto Tools to Airflow user list             |1
> AIRFLOW-1173 |Add Robinhood to list of Airflow users            |1
> AIRFLOW-1165 |airflow webservice crashes on ubuntu16 - python3  |0
> AIRFLOW-1160 |Upadte SparkSubmitOperator parameters             |1
> AIRFLOW-1155 |Add Tails.com to community                        |1
> AIRFLOW-1149 |Allow custom filters to be added to jinja2        |1
> AIRFLOW-1141 |Remove DAG.crawl_for_tasks method                 |1
> AIRFLOW-1140 |DatabricksSubmitRunOperator should template the "j|1
> AIRFLOW-1136 |Invalid parameters are not captured for Sqoop oper|1
> AIRFLOW-1125 |Clarify documentation regarding fernet_key        |1
> AIRFLOW-1122 |Node strokes are too thin for people with color vi|1
> AIRFLOW-1121 |airflow webserver --pid no longer write out pid fi|0
> AIRFLOW-1118 |Add evo.company to Airflow users                  |1
> AIRFLOW-1112 |Log which pool is full in scheduler when pool slot|1
> AIRFLOW-1107 |Add support for ftps non-default port             |1
> AIRFLOW-1106 |Add Groupalia/Letsbonus                           |1
> AIRFLOW-1095 |ldap_auth memberOf should come from configuration |1
> AIRFLOW-1094 |Invalid unit tests under `contrib/`               |1
> AIRFLOW-1091 |As a release manager I want to be able to compare |1
> AIRFLOW-1090 |Add HBO                                           |1
> AIRFLOW-1089 |Add Spark application arguments to SparkSubmitOper|1
> AIRFLOW-1081 |Task duration page is slow                        |1
> AIRFLOW-1075 |Cleanup security docs                             |1
> AIRFLOW-1065 |Add functionality for Azure Blob Storage          |1
> AIRFLOW-1059 |Reset_state_for_orphaned_task should operate in ba|1
> AIRFLOW-1058 |Improvements for SparkSubmitOperator              |0
> AIRFLOW-1055 |airflow/jobs.py:create_dag_run() exception for @on|0
> AIRFLOW-1051 |Add a test for resetdb to CliTests                |1
> AIRFLOW-1047 |Airflow logs vulnerable to XSS                    |1
> AIRFLOW-1045 |Make log level configurable via airflow.cfg       |1
> AIRFLOW-1043 |Documentation issues for operators                |1
> AIRFLOW-1041 |DockerOperator replaces its xcom_push method with |1
> AIRFLOW-1040 |Fix typos in comments/docstrings in models.py     |1
> AIRFLOW-1036 |Exponential backoff should use randomization      |1
> AIRFLOW-1035 |Exponential backoff retry logic should use 2 as ba|1
> AIRFLOW-1034 |Make it possible to connect to S3 in sigv4 regions|1
> AIRFLOW-1030 |HttpHook error when creating HttpSensor           |0
> AIRFLOW-1028 |Databricks Operator for Airflow                   |1
> AIRFLOW-1024 |Handle CeleryExecutor errors gracefully           |1
> AIRFLOW-1019 |active_dagruns shouldn't include paused DAGs      |0
> AIRFLOW-1018 |Scheduler DAG processes can not log to stdout     |0
> AIRFLOW-1016 |Allow HTTP HEAD request method on HTTPSensor      |1
> AIRFLOW-1015 |TreeView displayed over task instances            |0
> AIRFLOW-1013 |airflow/jobs.py:manage_slas() exception for @once |0
> AIRFLOW-1010 |Add a convenience script for signing              |1
> AIRFLOW-1009 |Remove SQLOperator from Concepts page             |1
> AIRFLOW-1007 |Jinja sandbox is vulnerable to RCE                |1
> AIRFLOW-1006 |Move configuration templates to separate files    |1
> AIRFLOW-1005 |Speed up Airflow startup time                     |1
> AIRFLOW-999  |Support for Redis database                        |1
> AIRFLOW-997  |Change setup.cfg to point to Apache instead of Max|1
> AIRFLOW-995  |Update Github PR template                         |1
> AIRFLOW-994  |Add MiNODES to the AIRFLOW Active Users List      |1
> AIRFLOW-991  |Mark_success while a task is running leads to fail|0
> AIRFLOW-990  |DockerOperator fails when logging unicode string  |1
> AIRFLOW-985  |Extend the sqoop operator/hook with additional par|1
> AIRFLOW-984  |Subdags unrecognized when subclassing SubDagOperat|1
> AIRFLOW-979  |Add GovTech GDS                                   |1
> AIRFLOW-976  |Mark success running task causes it to fail       |0
> AIRFLOW-969  |Catch bad python_callable argument at DAG construc|1
> AIRFLOW-963  |Some code examples are not rendered in the airflow|1
> AIRFLOW-960  |Add support for .editorconfig                     |1
> AIRFLOW-959  |.gitignore file is disorganized and incomplete    |1
> AIRFLOW-958  |Improve tooltip readability                       |1
> AIRFLOW-950  |Missing AWS integrations on documentation::integra|1
> AIRFLOW-947  |Make PrestoHook surface better messages when the P|1
> AIRFLOW-945  |Revert psycopg2 workaround when psycopg2 2.7.1 is |0
> AIRFLOW-943  |Add Digital First Media to the Airflow users list |1
> AIRFLOW-942  |Add mytaxi to Airflow Users                       |1
> AIRFLOW-935  |Impossible to use plugin executors                |1
> AIRFLOW-917  |Incorrectly formatted failure status message      |1
> AIRFLOW-916  |Fix ConfigParser deprecation warning              |1
> AIRFLOW-914  |Refactor BackfillJobTest.test_backfill_examples to|0
> AIRFLOW-913  |Refactor tests.CoreTest.test_scheduler_job to real|0
> AIRFLOW-912  |Refactor tests and build matrix                   |0
> AIRFLOW-911  |Add colouring and profiling info on tests         |1
> AIRFLOW-903  |Add configuration setting for default DAG view.   |1
> AIRFLOW-896  |BigQueryOperator fails to execute with certain inp|1
> AIRFLOW-889  |Minor error in the docstrings for BaseOperator.   |1
> AIRFLOW-888  |Operators should not push XComs by default        |0
> AIRFLOW-887  |Add compatibility with future v0.16               |1
> AIRFLOW-886  |Pass Operator result to post_execute hook         |1
> AIRFLOW-885  |Add Change.org to the list of Airflow users       |1
> AIRFLOW-882  |Code example in docs has unnecessary DAG>>Operator|1
> AIRFLOW-881  |Create SubDagOperator within DAG context manager w|1
> AIRFLOW-880  |Fix remote log functionality inconsistencies for W|1
> AIRFLOW-877  |GoogleCloudStorageDownloadOperator: template_ext c|1
> AIRFLOW-875  |Allow HttpSensor params to be templated           |1
> AIRFLOW-871  |multiple places use logging.warn() instead of warn|1
> AIRFLOW-866  |Add FTPSensor                                     |1
> AIRFLOW-863  |Example DAG start dates should be recent to avoid |1
> AIRFLOW-862  |Add DaskExecutor                                  |1
> AIRFLOW-860  |Circular module dependency prevents loading of cus|0
> AIRFLOW-854  |Add Open Knowledge International to Airflow users |1
> AIRFLOW-842  |scheduler.clean_dirty raises warning: SAWarning: T|1
> AIRFLOW-840  |Python3 encoding issue in Kerberos                |1
> AIRFLOW-836  |The paused and queryview endpoints are vulnerable |1
> AIRFLOW-831  |Fix broken unit tests                             |1
> AIRFLOW-830  |Plugin manager should log to debug, not info      |0
> AIRFLOW-829  |Reduce verbosity of successful Travis unit tests  |0
> AIRFLOW-828  |Add maximum size for XComs                        |0
> AIRFLOW-826  |Add Zendesk Hook                                  |1
> AIRFLOW-825  |Add Dataflow semantics                            |0
> AIRFLOW-823  |Make task instance details available via API      |1
> AIRFLOW-822  |Close the connection before throwing exception in |1
> AIRFLOW-821  |Scheduler dagbag importing not Py3 compatible     |1
> AIRFLOW-809  |SqlAlchemy is_ ColumnOperator Causing Errors in MS|0
> AIRFLOW-802  |Integration of spark-submit                       |0
> AIRFLOW-788  |Context unexpectedly added to hive conf           |0
> AIRFLOW-781  |Allow DataFlowJavaOperator to accept jar file stor|1
> AIRFLOW-770  |HDFS hooks should support alternative ways of gett|1
> AIRFLOW-756  |Refactor ssh_hook and ssh_operator                |0
> AIRFLOW-751  |SFTP file transfer functionality                  |1
> AIRFLOW-725  |Make merge tool use OS' keyring for password stora|1
> AIRFLOW-706  |Configuration shell commands are not split properl|1
> AIRFLOW-705  |airflow.configuration.run_command output does not |0
> AIRFLOW-681  |homepage doc link should pointing to apache's repo|1
> AIRFLOW-654  |SSL for AMQP w/ Celery(Executor)                  |1
> AIRFLOW-645  |HttpHook ignores https                            |1
> AIRFLOW-365  |Code view in subdag trigger exception             |1
> AIRFLOW-300  |Add Google Pubsub hook and operator               |1
>
> Cheers,
> Chris
>
> On Wed, Sep 13, 2017 at 2:01 PM, Chris Riccomini <criccomini@apache.org>
> wrote:
>
> > Note: Everything on master up to the point I cut the branch is included
> in
> > 1.9.0. I'm not sure what's up with the `dev/airflow-jira compare 1.9.0`
> > script.. I'm looking into that now.
> >
> > On Wed, Sep 13, 2017 at 1:55 PM, Alex Guziel <alex.guziel@airbnb.com.
> > invalid> wrote:
> >
> >> Nevermind, I misunderstood what you meant. (I thought you meant you were
> >> only including things with a fix version of 1.9.0, when you meant master
> >> cut + 1.9.0 fix versions)
> >>
> >> On Wed, Sep 13, 2017 at 1:19 PM, Alex Guziel <alex.guziel@airbnb.com>
> >> wrote:
> >>
> >> > Shouldn't we include everything on master?
> >> >
> >> > On Wed, Sep 13, 2017 at 12:45 PM, Chris Riccomini <
> >> criccomini@apache.org>
> >> > wrote:
> >> >
> >> >> Hey all,
> >> >>
> >> >> I've cut a 1.9.0 test branch.
> >> >>
> >> >> https://github.com/apache/incubator-airflow/tree/v1-9-test
> >> >>
> >> >> Here are the tickets that are being tracked on 1.9.0.
> >> >>
> >> >> ISSUE ID     |DESCRIPTION
>  |MERGED
> >> >> AIRFLOW-1608 |GCP Dataflow hook missing pending job state       |1
> >> >> AIRFLOW-1606 |DAG.sync_to_db is static, but takes a DAG as first|1
> >> >> AIRFLOW-1605 |Fix log source of local loggers                   |0
> >> >> AIRFLOW-1602 |Use LoggingMixin for the DAG class                |1
> >> >> AIRFLOW-1597 |Add GameWisp as Airflow user                      |1
> >> >> AIRFLOW-1594 |Installing via pip copies test files into python l|1
> >> >> AIRFLOW-1593 |Expose load_string in WasbHook                    |1
> >> >> AIRFLOW-1586 |MySQL to GCS to BigQuery fails for tables with dat|1
> >> >> AIRFLOW-1584 |Remove the insecure /headers endpoints            |1
> >> >> AIRFLOW-1582 |Improve logging structure of Airflow              |1
> >> >> AIRFLOW-1580 |Error in string formatter when throwing an excepti|1
> >> >> AIRFLOW-1579 |Allow jagged rows in BQ Hook.                     |1
> >> >> AIRFLOW-1577 |Add token support to DatabricksHook               |1
> >> >> AIRFLOW-1573 |Remove `thrift < 0.10.0` requirement             
|1
> >> >> AIRFLOW-1568 |Add datastore import/export operator              |1
> >> >> AIRFLOW-1567 |Clean up ML Engine operators                      |1
> >> >> AIRFLOW-1564 |Default logging filename contains a colon         |1
> >> >> AIRFLOW-1556 |BigQueryBaseCursor should support SQL parameters  |1
> >> >> AIRFLOW-1546 | add Zymergen to org list in README               |1
> >> >> AIRFLOW-1535 |Add support for Dataproc serviceAccountScopes in D|1
> >> >> AIRFLOW-1529 |Support quoted newlines in Google BigQuery load jo|1
> >> >> AIRFLOW-1522 |Increase size of val column for variable table in |1
> >> >> AIRFLOW-1521 |Template fields definition for bigquery_table_dele|1
> >> >> AIRFLOW-1507 |Make src, dst and bucket parameters as templated i|1
> >> >> AIRFLOW-1505 |Document when Jinja substitution occurs           |1
> >> >> AIRFLOW-1504 |Log Cluster Name on Dataproc Operator When Execute|1
> >> >> AIRFLOW-1499ss|Eliminate duplicate and unneeded code             |0
> >> >> AIRFLOW-1493 |Fix race condition with airflow run               |1
> >> >> AIRFLOW-1492 |Add metric for task success/failure               |1
> >> >> AIRFLOW-1489 |Docs: Typo in BigQueryCheckOperator               |1
> >> >> AIRFLOW-1478 |Chart -> Owner column should be sortable         
|1
> >> >> AIRFLOW-1476 |Add INSTALL file for source releases              |1
> >> >> AIRFLOW-1474 |Add dag_id regex for 'airflow clear' CLI command  |1
> >> >> AIRFLOW-1459 |integration rst doc is broken in github view      |1
> >> >> AIRFLOW-1438 |Scheduler batch queries should have a limit       |1
> >> >> AIRFLOW-1437 |BigQueryTableDeleteOperator should define deletion|1
> >> >> AIRFLOW-1402 |Cleanup SafeConfigParser DeprecationWarning       |1
> >> >> AIRFLOW-1401 |Standardize GCP project, region, and zone argument|1
> >> >> AIRFLOW-1394 |Add quote_character parameter to GoogleCloudStorag|1
> >> >> AIRFLOW-1389 |BigQueryOperator should support `createDisposition|1
> >> >> AIRFLOW-1384 |Add ARGO/CaDC                                     |1
> >> >> AIRFLOW-1359 |Provide GoogleCloudML operator for model evaluatio|1
> >> >> AIRFLOW-1352 |Revert bad logging Handler                        |0
> >> >> AIRFLOW-1350 |Add "query_uri" parameter for Google DataProc oper|1
> >> >> AIRFLOW-1345 |Don't commit on each loop                         |1
> >> >> AIRFLOW-1344 |Builds failing on Python 3.5 with AttributeError  |1
> >> >> AIRFLOW-1343 |Add airflow default label to the dataproc operator|1
> >> >> AIRFLOW-1338 |gcp_dataflow_hook is incompatible with the recent |1
> >> >> AIRFLOW-1337 |Customize log format via config file              |1
> >> >> AIRFLOW-1335 |Use buffered logger                               |1
> >> >>
> >> >> If you have stuff you want to get in, please set it with a fix
> version
> >> of
> >> >> 1.9.0.
> >> >>
> >> >> Please begin testing, stabilizing, and reporting bugs now! :)
> >> >>
> >> >> Cheers,
> >> >> Chris
> >> >>
> >> >
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message