forked from apache/airflow
-
Notifications
You must be signed in to change notification settings - Fork 8
/
CHANGELOG.txt
4189 lines (4060 loc) · 239 KB
/
CHANGELOG.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Airflow 1.10.14, 2020-12-10
----------------------------
Bug Fixes
"""""""""
- BugFix: Tasks with ``depends_on_past`` or ``task_concurrency`` are stuck (#12663)
- Fix issue with empty Resources in executor_config (#12633)
- Fix: Deprecated config ``force_log_out_after`` was not used (#12661)
- Fix empty asctime field in JSON formatted logs (#10515)
- [AIRFLOW-2809] Fix security issue regarding Flask SECRET_KEY (#3651)
- [AIRFLOW-2884] Fix Flask SECRET_KEY security issue in www_rbac (#3729)
- [AIRFLOW-2886] Generate random Flask SECRET_KEY in default config (#3738)
- Add missing comma in setup.py (#12790)
- Bugfix: Unable to import Airflow plugins on Python 3.8 (#12859)
- Fix setup.py missing comma in ``setup_requires`` (#12880)
- Don't emit first_task_scheduling_delay metric for only-once dags (#12835)
Improvements
""""""""""""
- Update setup.py to get non-conflicting set of dependencies (#12636)
- Rename ``[scheduler] max_threads`` to ``[scheduler] parsing_processes`` (#12605)
- Add metric for scheduling delay between first run task & expected start time (#9544)
- Add new-style 2.0 command names for Airflow 1.10.x (#12725)
- Add Kubernetes cleanup-pods CLI command for Helm Chart (#11802)
- Don't let webserver run with dangerous config (#12747)
- Replace pkg_resources with importlib.metadata to avoid VersionConflict errors (#12694)
Doc only changes
""""""""""""""""
- Clarified information about supported Databases
Airflow 1.10.13, 2020-11-24
----------------------------
New Features
""""""""""""
- Add "already checked" to failed pods in K8sPodOperator (#11368)
- Pass SQLAlchemy engine options to FAB based UI (#11395)
- [AIRFLOW-4438] Add Gzip compression to S3_hook (#8571)
- Add permission "extra_links" for Viewer role and above (#10719)
- Add generate_yaml command to easily test KubernetesExecutor before deploying pods (#10677)
- Add Secrets backend for Microsoft Azure Key Vault (#10898)
Bug Fixes
"""""""""
- SkipMixin: Handle empty branches (#11120)
- [AIRFLOW-5274] dag loading duration metric name too long (#5890)
- Handle no Dagrun in DagrunIdDep (#8389) (#11343)
- Fix Kubernetes Executor logs for long dag names (#10942)
- Add on_kill support for the KubernetesPodOperator (#10666)
- KubernetesPodOperator template fix (#10963)
- Fix displaying of add serialized_dag table migration
- Fix Start Date tooltip on DAGs page (#10637)
- URL encode execution date in the Last Run link (#10595)
- Fixes issue with affinity backcompat in Airflow 1.10
- Fix KubernetesExecutor import in views.py
- Fix issues with Gantt View (#12419)
- Fix Entrypoint and _CMD config variables (#12411)
- Fix operator field update for SerializedBaseOperator (#10924)
- Limited cryptography to < 3.2 for python 2.7
- Install cattr on Python 3.7 - Fix docs build on RTD (#12045)
- Limit version of marshmallow-sqlalchemy
- Pin `kubernetes` to a max version of 11.0.0 (#11974)
- Use snakebite-py3 for HDFS dependency for Python3 (#12340)
- Removes snakebite kerberos dependency (#10865)
- Fix failing dependencies for FAB and Celery (#10828)
- Fix pod_mutation_hook for 1.10.13 (#10850)
- Fix formatting of Host information
- Fix Logout Google Auth issue in Non-RBAC UI (#11890)
- Add missing imports to app.py (#10650)
- Show Generic Error for Charts & Query View in old UI (#12495)
- TimeSensor should respect the default_timezone config (#9699)
- TimeSensor should respect DAG timezone (#9882)
- Unify user session lifetime configuration (#11970)
- Handle outdated webserver session timeout gracefully. (#12332)
Improvements
""""""""""""
- Add XCom.deserialize_value to Airflow 1.10.13 (#12328)
- Mount airflow.cfg to pod_template_file (#12311)
- All k8s object must comply with JSON Schema (#12003)
- Validate airflow chart values.yaml & values.schema.json (#11990)
- Pod template file uses custom custom env variable (#11480)
- Bump attrs and cattrs dependencies (#11969)
- Bump attrs to > 20.0 (#11799)
- [AIRFLOW-3607] Only query DB once per DAG run for TriggerRuleDep (#4751)
- Rename task with duplicate task_id
- Manage Flask AppBuilder Tables using Alembic Migrations (#12352)
- ``airflow test`` only works for tasks in 1.10, not whole dags (#11191)
- Improve warning messaging for duplicate task_ids in a DAG (#11126)
- Pins moto to 1.3.14 (#10986)
- DbApiHook: Support kwargs in get_pandas_df (#9730)
- Make grace_period_seconds option on K8sPodOperator (#10727)
- Fix syntax error in Dockerfile 'maintainer' Label (#10899)
- The entrypoints in Docker Image should be owned by Airflow (#10853)
- Make dockerfiles Google Shell Guide Compliant (#10734)
- clean-logs script for Dockerfile: trim logs before sleep (#10685)
- When sending tasks to celery from a sub-process, reset signal handlers (#11278)
- SkipMixin: Add missing session.commit() and test (#10421)
- Webserver: Further Sanitize values passed to origin param (#12459)
- Security upgrade lodash from 4.17.19 to 4.17.20 (#11095)
- Log instead of raise an Error for unregistered OperatorLinks (#11959)
- Mask Password in Log table when using the CLI (#11468)
- [AIRFLOW-3607] Optimize dep checking when depends on past set and concurrency limit
- Execute job cancel HTTPRequest in Dataproc Hook (#10361)
- Use rst lexer to format airflow upgrade check output (#11259)
- Remove deprecation warning from contrib/kubernetes/pod.py
- adding body as templated field for CloudSqlImportOperator (#10510)
- Change log level for User's session to DEBUG (#12414)
Deprecations
""""""""""""
- Deprecate importing Hooks from plugin-created module (#12133)
- Deprecate adding Operators and Sensors via plugins (#12069)
Doc only changes
""""""""""""""""
- [Doc] Correct description for macro task_instance_key_str (#11062)
- Checks if all the libraries in setup.py are listed in installation.rst file (#12023)
- Revise "Project Focus" copy (#12011)
- Move Project focus and Principles higher in the README (#11973)
- Remove archived link from README.md (#11945)
- Update download url for Airflow Version (#11800)
- Add Project URLs for PyPI page (#11801)
- Move Backport Providers docs to our docsite (#11136)
- Refactor rebase copy (#11030)
- Add missing images for kubernetes executor docs (#11083)
- Fix identation in executor_config example (#10467)
- Enhanced the Kubernetes Executor doc (#10433)
- Refactor content to a markdown table (#10863)
- Rename "Beyond the Horizon" section and refactor content (#10802)
- Refactor official source section to use bullets (#10801)
- Add section for official source code (#10678)
- Add redbubble link to Airflow merchandise (#10359)
- README Doc: Link to Airflow directory in ASF Directory (#11137)
- Fix the default value for VaultBackend's config_path (#12518)
Airflow 1.10.12, 2020-08-25
----------------------------
New Features
""""""""""""
- Add DateTimeSensor (#9697)
- Add ClusterPolicyViolation support to airflow local settings (#10282)
- Get Airflow configs with sensitive data from Secret Backends (#9645)
- [AIRFLOW-4734] Upsert functionality for PostgresHook.insert_rows() (#8625)
- Allow defining custom XCom class (#8560)
Bug Fixes
"""""""""
- Add pre 1.10.11 Kubernetes Paths back with Deprecation Warning (#10067)
- Fixes PodMutationHook for backwards compatibility (#9903)
- Fix bug in executor_config when defining resources (#9935)
- Respect DAG Serialization setting when running sync_perm (#10321)
- Show correct duration on graph view for running task (#8311) (#8675)
- Fix regression in SQLThresholdCheckOperator (#9312)
- [AIRFLOW-6931] Fixed migrations to find all dependencies for MSSQL (#9891)
- Avoid sharing session with RenderedTaskInstanceFields write and delete (#9993)
- Fix clear future recursive when ExternalTaskMarker is used (#9515)
- Handle IntegrityError while creating TIs (#10136)
- Fix airflow-webserver startup errors when using Kerberos Auth (#10047)
- Fixes treatment of open slots in scheduler (#9316) (#9505)
- Fix KubernetesPodOperator reattachment (#10230)
- Fix more PodMutationHook issues for backwards compatibility (#10084)
- [AIRFLOW-5391] Do not re-run skipped tasks when they are cleared (#7276)
- Fix task_instance_mutation_hook (#9910)
- Fixes failing formatting of DAG file containing {} in docstring (#9779)
- Fix is_terminal_support_colors function (#9734)
- Fix PythonVirtualenvOperator when using ``provide_context=True`` (#8256)
- Fix issue with mounting volumes from secrets (#10366)
- BugFix: K8s Executor Multinamespace mode is evaluated to true by default (#10410)
- Make KubernetesExecutor recognize kubernetes_labels (#10412)
- Fix broken Kubernetes PodRuntimeInfoEnv (#10478)
- Sync FAB Permissions for all base views (#12162)
Improvements
""""""""""""
- Use Hash of Serialized DAG to determine DAG is changed or not (#10227)
- Update Serialized DAGs in Webserver when DAGs are Updated (#9851)
- Do not Update Serialized DAGs in DB if DAG did not change (#9850)
- Add __repr__ to SerializedDagModel (#9862)
- Update JS packages to latest versions (#9811) (#9921)
- UI Graph View: Focus upstream / downstream task dependencies on mouseover (#9303)
- Allow ``image`` in ``KubernetesPodOperator`` to be templated (#10068)
- [AIRFLOW-6843] Add delete_option_kwargs to delete_namespaced_pod (#7523)
- Improve process terminating in scheduler_job (#8064)
- Replace deprecated base classes used in bigquery_check_operator (#10272)
- [AIRFLOW-5897] Allow setting -1 as pool slots value in webserver (#6550)
- Limit all google-cloud api to <2.0.0 (#10317)
- [AIRFLOW-6706] Lazy load operator extra links (#7327) (#10318)
- Add Snowflake support to SQL operator and sensor (#9843)
- Makes multi-namespace mode optional (#9570)
- Pin Pyarrow < 1.0
- Pin pymongo version to <3.11.0
- Pin google-cloud-container to <2 (#9901)
- Dockerfile: Remove package.json and yarn.lock from the prod image (#9814)
- Dockerfile: The group of embedded DAGs should be root to be OpenShift compatible (#9794)
- Update upper limit of flask-swagger, gunicorn & jinja2 (#9684)
- Webserver: Sanitize values passed to origin param (#10334)
- Sort connection type list in add/edit page alphabetically (#8692)
Doc only changes
""""""""""""""""
- Add new committers: Ry Walker & Leah Cole to project.rst (#9892)
- Add Qingping Hou to committers list (#9725)
- Updated link to official documentation (#9629)
- Create a short-link for Airflow Slack Invites (#10034)
- Fix docstrings in BigQueryGetDataOperator (#10042)
- Set language on code-block on docs/howto/email-config.rst (#10238)
- Remove duplicate line from 1.10.10 CHANGELOG (#10289)
- Improve heading on Email Configuration page (#10175)
- Fix link for the Jinja Project in docs/tutorial.rst (#10245)
- Create separate section for Cron Presets (#10247)
- Add Syntax Highlights to code-blocks in docs/best-practices.rst (#10258)
- Fix docstrings in BigQueryGetDataOperator (#10042)
- Fix typo in Task Lifecycle section (#9867)
- Make Secret Backend docs clearer about Variable & Connection View (#8913)
Airflow 1.10.11, 2020-07-10
-----------------------------
New Features
""""""""""""
- Add task instance mutation hook (#8852)
- Allow changing Task States Colors (#9520)
- Add support for AWS Secrets Manager as Secrets Backend (#8186)
- Add airflow info command to the CLI (#8704)
- Add Local Filesystem Secret Backend (#8596)
- Add Airflow config CLI command (#8694)
- Add Support for Python 3.8 (#8836)(#8823)
- Allow K8S worker pod to be configured from JSON/YAML file (#6230)
- Add quarterly to crontab presets (#6873)
- Add support for ephemeral storage on KubernetesPodOperator (#6337)
- Add AirflowFailException to fail without any retry (#7133)
- Add SQL Branch Operator (#8942)
Bug Fixes
"""""""""
- Use NULL as dag.description default value (#7593)
- BugFix: DAG trigger via UI error in RBAC UI (#8411)
- Fix logging issue when running tasks (#9363)
- Fix JSON encoding error in DockerOperator (#8287)
- Fix alembic crash due to typing import (#6547)
- Correctly restore upstream_task_ids when deserializing Operators (#8775)
- Correctly store non-default Nones in serialized tasks/dags (#8772)
- Correctly deserialize dagrun_timeout field on DAGs (#8735)
- Fix tree view if config contains " (#9250)
- Fix Dag Run UI execution date with timezone cannot be saved issue (#8902)
- Fix Migration for MSSQL (#8385)
- RBAC ui: Fix missing Y-axis labels with units in plots (#8252)
- RBAC ui: Fix missing task runs being rendered as circles instead (#8253)
- Fix: DagRuns page renders the state column with artifacts in old UI (#9612)
- Fix task and dag stats on home page (#8865)
- Fix the trigger_dag api in the case of nested subdags (#8081)
- UX Fix: Prevent undesired text selection with DAG title selection in Chrome (#8912)
- Fix connection add/edit for spark (#8685)
- Fix retries causing constraint violation on MySQL with DAG Serialization (#9336)
- [AIRFLOW-4472] Use json.dumps/loads for templating lineage data (#5253)
- Restrict google-cloud-texttospeach to <v2 (#9137)
- Fix pickling failure when spawning processes (#8671)
- Pin Version of Azure Cosmos to <4 (#8956)
- Azure storage 0.37.0 is not installable any more (#8833)
- Fix modal_backdrop z-index in the UI (#7313)
- Fix Extra Links in Gannt View (#8308)
- Bug fix for EmrAddStepOperator init with cluster_name error (#9235)
- Fix KubernetesPodOperator pod name length validation (#8829)
- Fix non updating DAG code by checking against last modification time (#8266)
- BugFix: Unpausing a DAG with catchup=False creates an extra DAG run (#8776)
Improvements
""""""""""""
- Improve add_dag_code_table migration (#8176)
- Persistent display/filtering of DAG status (#8106)
- Set unique logger names (#7330)
- Update the version of cattrs from 0.9 to 1.0 to support Python 3.8 (#7100)
- Reduce response payload size of /dag_stats and /task_stats (#8655)
- Add authenticator parameter to snowflake_hook (#8642)
- Show "Task Reschedule" table in Airflow Webserver (#9521)
- Change worker_refresh_interval fallback to default of 30 (#9588)
- Use pfromat instead of str to render arguments in WebUI (#9587)
- Simplify DagFileProcessorManager (#7521)
- Reload gunicorn when plugins has beeen changed (#8997)
- Fix the default value for store_dag_code (#9554)
- Add support for fetching logs from running pods (#8626)
- Persist start/end date and duration for DummyOperator Task Instance (#8663)
- Ensure "started"/"ended" in tooltips are not shown if job not started (#8667)
- Add context to execution_date_fn in ExternalTaskSensor (#8702)
- Avoid color info in response of ``/dag_stats`` & ``/task_stats`` (#8742)
- Make loading plugins from entrypoint fault-tolerant (#8732)
- Refactor Kubernetes worker config (#7114)
- Add default ``conf`` parameter to Spark JDBC Hook (#8787)
- Allow passing backend_kwargs to AWS SSM client (#8802)
- Filter dags by clicking on tag (#8897)
- Support k8s auth method in Vault Secrets provider (#8640)
- Monitor pods by labels instead of names (#6377)
- Optimize count query on /home (#8729)
- Fix json string escape in tree view (#8551)
- Add TaskInstance state to TI Tooltip to be colour-blind friendlier (#8910)
- Add a tip to trigger DAG screen (#9049)
- Use Markup for htmlcontent for landing_times (#9242)
- Pinning max pandas version to 2.0 (lesser than) to allow pandas 1.0 (#7954)
- Update example webserver_config.py to show correct CSRF config (#8944)
- Fix displaying Executor Class Name in "Base Job" table (#8679)
- Use existing DagBag for 'dag_details' & 'trigger' Endpoints (#8501)
- Flush pending Sentry exceptions before exiting (#7232)
- Display DAG run conf in the list view (#6794)
- Fix performance degradation when updating dagrun state (#8435)
- Don't use the ``|safe`` filter in code, it's risky (#9180)
- Validate only task commands are run by executors (#9178)
- Show Deprecation warning on duplicate Task ids (#8728)
- Move DAG._schedule_interval logic out of ``DAG.__init__`` (#8225)
- Make retrieving Paused Dag ids a separate method (#7587)
- Bulk fetch paused_dag_ids (#7476)
- Add a configurable DAGs volume mount path for Kubernetes (#8147)
- Add schedulername option for KubernetesPodOperator (#6088)
- Support running git sync container as root (#6312)
- Add extra options for Slack Webhook operator and Slack hook (#9409)
- Monkey patch greenlet celery pools (#8559)
- Decrypt secrets from SystemsManagerParameterStoreBackend (#9214)
- Prevent clickable sorting on non sortable columns in TI view (#8681)
- Make hive macros py3 compatible (#8598)
- Fix SVG tooltip positioning with custom scripting (#8269)
- Avoid unnecessary sleep to maintain local task job heart rate (#6553)
- Include some missing RBAC roles on User and Viewer roles (#9133)
- Show Dag's Markdown docs on Tree View (#9448)
- Improved compatibility with Python 3.5+ - Convert signal.SIGTERM to int (#9207)
- Add 'main' param to template_fields in DataprocSubmitPySparkJobOperator (#9154)
- Make it possible to silence warnings from Airflow (#9208)
- Remove redundant count query in BaseOperator.clear() (#9362)
- Fix DB migration message (#8988)
- Fix awkward log info in dbapi_hook (#8482)
- Fix Celery default to no longer allow pickle (#7205)
- Further validation that only task commands are run by executors (#9240)
- Remove vendored nvd3 and slugify libraries (#9136)
- Enable configurable git sync depth (#9094)
- Reduce the required resources for the Kubernetes's sidecar (#6062)
- Refactor K8S codebase with k8s API models (#5481)
- Move k8sexecutor out of contrib to closer match master (#8904)
- Allow filtering using "event" and "owner" in "Log" view (#4881)
- Add Yandex.Cloud custom connection to 1.10 (#8791)
- Add table-hover css class to DAGs table (#5033)
- Show un/pause errors in dags view. (#7669)
- Restructure database queries on /home (#4872)
- Add Cross Site Scripting defence (#6913)
- Make Gantt tooltip the same as Tree and Graph view (#8220)
- Add config to only delete worker pod on task failure (#7507)(#8312)
- Remove duplicate error message on chart connection failure (#8476)
- Remove default value spark_binary (#8508)
- Expose Airflow Webserver Port in Production Docker Image (#8228)
- Commit after each alembic migration (#4797)
- KubernetesPodOperator fixes and test (#6524)
- Docker Image: Add ADDITIONAL_AIRFLOW_EXTRAS (#9032)
- Docker Image: Add ADDITIONAL_PYTHON_DEPS (#9031)
- Remove httplib2 from Google requirements (#9194)
- Merging multiple sql operators (#9124)
- Adds hive as extra in pyhive dependency (#9075)
- Change default auth for experimental backend to deny_all (#9611)
- Restrict changing XCom values from the Webserver (#9614)
- Add __repr__ for DagTag so tags display properly in /dagmodel/show (#8719)
- Functionality to shuffle HMS connections used by HiveMetastoreHook facilitating load balancing (#9280)
- Expose SQLAlchemy's connect_args and make it configurable (#6478)
Doc only changes
""""""""""""""""
- Add docs on using DAGRun.conf (#9578)
- Enforce code-block directives in doc (#9443)
- Carefully parse warning messages when building documentation (#8693)
- Make KubernetesPodOperator clear in docs (#8444)
- Improve language in Pod Mutation Hook docs (#8445)
- Fix formatting of Pool docs in concepts.rst (#8443)
- Make doc clearer about Airflow Variables using Environment Variables (#8427)
- Fix pools doc for LocalExecutor (#7643)
- Add Local and Sequential Executors to Doc (#8084)
- Add documentation for CLI command airflow dags test (#8251)
- Fix typo in DAG Serialization documentation (#8317)
- Add scheduler in production section (#7351)
- Add a structural dag validation example (#6727)
- Adding Task re-run documentation (#6295)
- Fix outdated doc on settings.policy (#7532)
- Add docs about reload_on_plugin_change opiton (#9575)
- Add copy button to Code Blocks in Airflow Docs (#9450)
- Update commands in docs for v1.10+ (#9585)
- Add more info on dry-run CLI option (#9582)
- Document default timeout value for SSHOperator (#8744)
- Fix docs on creating CustomOperator (#8678)
- Enhanced documentation around Cluster Policy (#8661)
- Use sphinx syntax in concepts.rst (#7729)
- Update README to remove Py 3.8 limitation for Master (#9451)
- Add note about using dag_run.conf in BashOperator (#9143)
- Improve tutorial - Include all imports statements (#8670)
- Added more precise Python requirements to README.md (#8455)
- Fix Airflow Stable version in README.md (#9360)
- Update AWS connection example to show how to set from env var (#9191)
- Fix list formatting of plugins doc. (#8873)
- Add 'Version Added' on Secrets Backend docs (#8264)
- Simplify language re roll-your-own secrets backend (#8257)
- Add installation description for repeatable PyPi installation (#8513)
- Add note extra links only render on when using RBAC webserver (#8788)
- Remove unused airflow import from docs (#9274)
- Add PR/issue note in Contribution Workflow Example (#9177)
- Don't use the term "whitelist" - language matters (#9174)
- Add docs to change Colors on the Webserver (#9607)
- Change 'initiate' to 'initialize' in installation.rst (#9619)
- Replace old Variables View Screenshot with new (#9620)
- Replace old SubDag zoom screenshot with new (#9621)
- Update docs about the change to default auth for experimental API (#9617)
Airflow 1.10.10, 2020-04-09
---------------------------
New Features
""""""""""""
- [AIRFLOW-7048] Allow user to chose timezone to use in UI (#8046)
- Add Production Docker image support (#7832)
- Get Airflow Variables from Environment Variables (#7923)
- Get Airflow Variables from Hashicorp Vault (#7944)
- Get Airflow Variables from AWS Systems Manager Parameter Store (#7945)
- Get Airflow Variables from GCP Secrets Manager (#7946)
- [AIRFLOW-5705] Add secrets backend and support for AWS SSM / Get Airflow Connections from AWS Parameter Store(#6376)
- [AIRFLOW-7104] Add Secret backend for GCP Secrets Manager / Get Airflow Connections from GCP Secrets Manager (#7795)
- [AIRFLOW-7076] Add support for HashiCorp Vault as Secrets Backend / Get Airflow Connections from Hashicorp Vault (#7741)
- [AIRFLOW-6685] Add ThresholdCheckOperator (#7353)
- [AIRFLOW-7080] Add API endpoint to return a DAG's paused state (#7737)
Bug Fixes
"""""""""
- BugFix: Show task_id in the Graph View tooltip (#7859)
- [AIRFLOW-6730] Use total_seconds instead of seconds (#7363)
- [AIRFLOW-6167] Escape column name in create table in hive (#6741)
- [AIRFLOW-6628] DAG auto-complete now suggests from all acessible DAGs (#7251)
- [AIRFLOW-7113] Fix gantt render error (#7913)
- [AIRFLOW-6399] Add _access control to validate deserialized DAGs (#7896)
- [AIRFLOW-6399] Serialization: DAG access_control field should be decorated field in DAG serialization (#7879)
- [AIRFLOW-4453] Make behavior of `none_failed` consistent with documentation (#7464)
- [AIRFLOW-4363] Fix JSON encoding error (#7628)
- [AIRFLOW-6683] Run REST API tests when DAGs are serialized (#7352)
- [AIRFLOW-6704] Copy common TaskInstance attributes from Task (#7324)
- [AIRFLOW-6734] Use configured base_template instead of hard-coding (#7367)
- [AIRFLOW-7098] Simple salesforce release 1.0.0 breaks the build (#7775)
- [AIRFLOW-6062] Executor would only delete workers in its own namespace (#7123)
- [AIRFLOW-7074] Add Permissions to view SubDAGs (#7752)
- [AIRFLOW-7025] Fix SparkSqlHook.run_query to handle its parameter properly (#7677)
- [AIRFLOW-6855] Escape project_dataset_table in SQL query in gcs to bq operator (#7475)
- [AIRFLOW-6949] Respect explicit conf to SparkSubmitOperator (#7575)
- [AIRFLOW-6588] write_stdout and json_format are boolean (#7199)
- [AIRFLOW-3439] Decode logs with 'utf-8' (#4474)
- [AIRFLOW-6878] Fix misconfigured default value for kube_client_request_args
- [AIRFLOW-5167] Update dependencies for GCP packages (#7116)
- [AIRFLOW-6821] Success callback not called when task marked as success from UI (#7447)
- [AIRFLOW-6740] Remove Undocumented, deprecated, dysfunctional PROXY_FIX_NUM_PROXIES (#7359)
- [AIRFLOW-6728] Change various DAG info methods to POST (#7364)
- [AIRFLOW-6997] Make sure worker pods initcontainers obtain env vars from config (#7663)
- [AIRFLOW-7062] Fix pydruid release breaking the build (#7720)
- [AIRFLOW-6040] ReadTimoutError in KubernetesExecutor should not raise exception (#7616)
- [AIRFLOW-6943] Fix utf-8 encodeed description in DAG in Python 2 (#7567)
- [AIRFLOW-6892] Fix broken non-wheel releases (#7514)
- [AIRFLOW-6789] BugFix: Fix Default Worker concurrency (#7494)
- [AIRFLOW-6840] Bump up version of future (#7471)
- [AIRFLOW-5705] Fix bugs in AWS SSM Secrets Backend (#7745)
- [AIRFLOW-5705] Fix bug in Secrets Backend (#7742)
- Fix CloudSecretsManagerBackend invalid connections_prefix (#7861)
- [AIRFLOW-7045] BugFix: DebugExecutor fails to change task state. (#8073)
- BugFix: Datetimepicker is stuck on the UI (#8092)
- [AIRFLOW-5277] Gantt chart respects per-user the Timezone UI setting (#8096)
- Fix timezones displayed in Task Instance tooltip (#8103)
- BugFix: Fix writing & deleting Dag Code for Serialized DAGs (#8151)
- Make the default TI pool slots '1' (#8153)
- Fix 500 error in Security screens (#8165)
- Fix Viewing Dag Code for Stateless Webserver (#8159)
- Fix issue with SQLAlchemy 1.3.16 (#8230)
Improvements
""""""""""""
- Use same tooltip for Graph and Tree views for TaskInstances (#8043)
- Allow DateTimePicker in Webserver to actually pick times too (#8034)
- [AIRFLOW-5590] Add run_id to trigger DAG run API response (#6256)
- [AIRFLOW-6695] Can now pass dagrun conf when triggering dags via UI (#7312)
- [AIRFLOW-5336] Add ability to make updating FAB perms on webserver in it optional (#5940)
- [AIRFLOW-1467] Allow tasks to use more than one pool slot (#7160)
- [AIRFLOW-6987] Avoid creating default connections (#7629)
- [AIRFLOW-4175] S3Hook load_file should support ACL policy parameter (#7733)
- [AIRFLOW-4438] Add Gzip compression to S3_hook (#7680)
- Allow setting Airflow Variable values to empty string (#8021)
- Dont schedule dummy tasks (#7880)
- Prevent sequential scan of task instance table (#8014)
- [AIRFLOW-7017] Respect default dag view in trigger dag origin (#7667)
- [AIRFLOW-6837] Limit description length of a Dag on HomePage (#7457)
- [AIRFLOW-6989] Display Rendered template_fields without accessing Dag files (#7633)
- [AIRFLOW-5944] Rendering templated_fields without accessing DAG files (#6788)
- [AIRFLOW-5946] DAG Serialization: Store source code in db (#7217)
- [AIRFLOW-7079] Remove redundant code for storing template_fields (#7750)
- [AIRFLOW-7024] Add the verbose parameter support to SparkSqlOperator (#7676)
- [AIRFLOW-6733] Extend not replace template (#7366)
- [AIRFLOW-7001] Further fix for the MySQL 5.7 UtcDateTime (#7655)
- [AIRFLOW-6014] Handle pods which are preempted & deleted by kubernetes but not restarted (#6606)
- [AIRFLOW-6950] Remove refresh_executor_config from ti.refresh_from_db (#7577)
- [AIRFLOW-7016] Sort dag tags in the UI (#7661)
- [AIRFLOW-6762] Fix link to "Suggest changes on this page" (#7387)
- [AIRFLOW-6948] Remove ASCII Airflow from version command (#7572)
- [AIRFLOW-6767] Correct name for default Athena workgroup (#7394)
- [AIRFLOW-6905] Update pin.svg with new pinwheel (#7524)
- [AIRFLOW-6801] Make use of ImportError.timestamp (#7425)
- [AIRFLOW-6830] Add Subject/MessageAttributes to SNS hook and operator (#7451)
- [AIRFLOW-6630] Resolve handlebars advisory (#7284)
- [AIRFLOW-6945] MySQL 5.7 is used in v1-10-test as an option
- [AIRFLOW-6871] Optimize tree view for large DAGs (#7492)
- [AIRFLOW-7063] Fix dag.clear() slowness caused by count (#7723)
- [AIRFLOW-7023] Remove duplicated package definitions in setup.py (#7675)
- [AIRFLOW-7001] Time zone removed from MySQL TIMSTAMP field inserts
- [AIRFLOW-7105] Unify Secrets Backend method interfaces (#7830)
- Make BaseSecretsBackend.build_path generic (#7948)
- Allow hvac pakage installation using 'hashicorp' extra (#7915)
- Standardize SecretBackend class names (#7846)
- [AIRFLOW-5705] Make AwsSsmSecretsBackend consistent with VaultBackend (#7753)
- [AIRFLOW-7045] Update SQL query to delete RenderedTaskInstanceFields (#8051)
- Handle DST better in Task Instance tool tips (#8104)
Misc/Internal
"""""""""""""
- Fix Flaky TriggerDAG UI test (#8022)
- Remove unnecessary messages in CI (#7951)
- Fixes too high parallelism in CI (#7947)
- Install version is not persistent in breeze (#7914)
- Fixed automated check for image rebuild (#7912)
- Move Dockerfile to Dockerfile.ci (#7829)
- Generate requirements are now sorted (#8040)
- Change name of the common environment initialization function (#7805)
- Requirements now depend on python version (#7841)
- Bring back reset db explicitly called at CI entry (#7798)
- Fixes unclean installation of Airflow 1.10 (#7796)
- [AIRFLOW-7029] Use separate docker image for running license check (#7678)
- [AIRFLOW-5842] Swtch to Debian buster image as a base (#7647)
- [AIRFLOW-5828] Move build logic out from hooks/build (#7618)
- [AIRFLOW-6839] Even more mypy speed improvements (#7460)
- [AIRFLOW-6820] split breeze into functions (#7433)
- [AIRFLOW-7097] Install gcloud beta componensts in CI image (#7772)
- [AIRFLOW-7018] fixing travis's job name escaping problem (#7668)
- [AIRFLOW-7054] Breeze has an option now to reset db at entry (#7710)
- [AIRFLOW-7005] Added exec command to Breeze (#7649)
- [AIRFLOW-7015] Detect Dockerhub repo/user when building on Dockerhub (#7673)
- [AIRFLOW-6727] Fix minor bugs in Release Management scripts (#7355)
- [AIRFLOW-7013] Automated check if Breeze image needs to be pulled (#7656)
- [AIRFLOW-7010] Skip in-container checks for Dockerhub builds (#7652)
- [AIRFLOW-7011] Pin JPype release to allow to build 1.10 images
- [AIRFLOW-7006] Fix missing +e in Breeze script (#7648)
- [AIRFLOW-6979] Fix breeze test-target specific test param issue (#7614)
- [AIRFLOW-6932] Add restart-environment command to Breeze
- [AIRFLOW-6919] Make Breeze DAG-test friedly (#7539)
- [AIRFLOW-6838] Introduce real subcommands for Breeze (#7515)
- [AIRFLOW-6763] Make systems tests ready for backport tests (#7389)
- [AIRFLOW-6866] Fix wrong export for Mac on Breeze (#7485)
- [AIRFLOW-6842] Skip fixing ownership on Mac (#7469)
- [AIRFLOW-6841] Fixed unbounded variable on Mac (#7465)
- [AIRFLOW-7067] Pinned version of Apache Airflow (#7730)
- [AIRFLOW-7058] Add support for different DB versions (#7717)
- [AIRFLOW-7002] Get rid of yaml "parser" in bash (#7646)
- [AIRFLOW-6972] Shorter frequently used commands in Breeze (#7608)
Doc only changes
""""""""""""""""
- Fix typo for store_serialized_dags config (#7952)
- Fix broken link in README.md (#7893)
- Separate supported Postgres versions with comma (#7892)
- Fix grammar in setup.py (#7877)
- Add Jiajie Zhong to committers list (#8047)
- Update Security doc for 1.10.* for auth backends (#8072)
- Fix Example in config_templates for Secrets Backend (#8074)
- Add backticks in IMAGES.rst command description (#8075)
- Change version_added for store_dag_code config (#8076)
- [AIRFLOW-XXXX] Remove duplicate docs
- [AIRFLOW-XXXX] Remove the defunct limitation of Dag Serialization (#7716)
- [AIRFLOW-XXXX] Add prerequisite tasks for all GCP operators guide (#6049)
- [AIRFLOW-XXXX] Simplify AWS/Azure/Databricks operators listing (#6047)
- [AIRFLOW-XXXX] Add external reference to all GCP operator guide (#6048)
- [AIRFLOW-XXXX] Simplify GCP operators listing
- [AIRFLOW-XXXX] Simplify Qubole operators listing
- [AIRFLOW-XXXX] Add autogenerated TOC (#6038)
- [AIRFLOW-XXXX] Create "Using the CLI" page (#5823)
- [AIRFLOW-XXXX] Group references in one section (#5776)
- [AIRFLOW-XXXX] Fix analytics doc (#5885)
- [AIRFLOW-XXXX] Add S3 Logging section (#6039)
- [AIRFLOW-XXXX] Move Azure Logging section above operators (#6040)
- [AIRFLOW-XXXX] Update temp link to a fixed link (#7715)
- [AIRFLOW-XXXX] Add Updating.md section for 1.10.9 (#7385)
- [AIRFLOW-XXXX] Remove duplication in BaseOperator docstring (#7321)
- [AIRFLOW-XXXX] Update tests info in CONTRIBUTING.rst (#7466)
- [AIRFLOW-XXXX] Small BREEZE.rst update (#7487)
- [AIRFLOW-XXXX] Add instructions for logging to localstack S3 (#7461)
- [AIRFLOW-XXXX] Remove travis config warnings (#7467)
- [AIRFLOW-XXXX] Add communication chapter to contributing (#7204)
- [AIRFLOW-XXXX] Add known issue - example_dags/__init__.py (#7444)
- [AIRFLOW-XXXX] Fix breeze build-docs (#7445)
- [AIRFLOW-XXXX] Less verbose docker builds
- [AIRFLOW-XXXX] Speed up mypy runs (#7421)
- [AIRFLOW-XXXX] Fix location of kubernetes tests (#7373)
- [AIRFLOW-XXXX] Remove quotes from domains in Google Oauth (#4226)
- [AIRFLOW-XXXX] Add explicit info about JIRAs for code-related PRs (#7318)
- [AIRFLOW-XXXX] Fix typo commiter => committer (#7392)
- [AIRFLOW-XXXX] Remove duplicated paragraph in docs (#7662)
- Fix reference to KubernetesPodOperator (#8100)
- Update the tree view of dag on Concepts Last Run Only (#8268)
Airflow 1.10.9, 2020-02-07
--------------------------
Bug Fixes
"""""""""
- [AIRFLOW-6751] Pin Werkzeug (dependency of a number of our dependencies) to < 1.0.0 (#7377)
Airflow 1.10.8, 2020-02-07
--------------------------
New Features
""""""""""""
- [AIRFLOW-4026] Add filter by DAG tags (#6489)
- [AIRFLOW-6613] Center dag on graph view load (#7238)
- [AIRFLOW-5843] Add conf option to Add DAG Run view (#7281)
- [AIRFLOW-4495] Allow externally triggered dags to run for future exec dates (#7038)
Improvements
""""""""""""
- [AIRFLOW-6438] Filter DAGs returned by blocked (#7019)
- [AIRFLOW-6666] Resolve js-yaml advisory (#7283)
- [AIRFLOW-6632] Bump dagre-d3 to resolve lodash CVE advisory (#7280)
- [AIRFLOW-6667] Resolve serialize-javascript advisory (#7282)
- [AIRFLOW-6451] self._print_stat() in dag_processing.py should be skippable (#7134)
- [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner (#7090)
- [AIRFLOW-6319] Add support for AWS Athena workgroups (#6871)
- [AIRFLOW-6677] Remove deprecation warning from SQLAlchmey (#7289)
- [AIRFLOW-6428] Fix import path for airflow.utils.dates.days_ago in Example DAGs (#7007)
- [AIRFLOW-6595] Use TaskNotFound exception instead of AirflowException (#7210)
- [AIRFLOW-6620] Mock celery in worker cli test (#7243)
- [AIRFLOW-6608] Change logging level for Bash & PyOperator Env exports
- [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker (#6633)
- [AIRFLOW-6359] Make Spark status_poll_interval explicit (#6978)
- [AIRFLOW-6359] spark_submit_hook.py status polling interval config (#6909)
- [AIRFLOW-6316] Use exampleinclude directives in tutorial.rst (#6868)
- [AIRFLOW-6519] Make TI logs constants in Webserver configurable (#7113)
- [AIRFLOW-6327] http_hook: Accept json= parameter for payload (#6886)
- [AIRFLOW-6261] flower_basic_auth eligible to _cmd (#6825)
- [AIRFLOW-6238] Filter dags returned by dag_stats
- [AIRFLOW-5616] Switch PrestoHook from pyhive to prestosql-client to s
- [AIRFLOW-6611] Add proxy_fix configs to default_airflow.cfg (#7236)
- [AIRFLOW-6557] Add test for newly added fields in BaseOperator (#7162)
- [AIRFLOW-6584] Pin cassandra driver (#7194)
- [AIRFLOW-6537] Fix backticks in rst files (#7140)
- [AIRFLOW-4428] Error if exec_date before default_args.start_date in trigger_dag (#6948)
- [AIRFLOW-6330] Show cli help when param blank or typo (#6883)
- [AIRFLOW-4113] Unpin boto3 (#6884)
- [AIRFLOW-6181] Add DebugExecutor (#6740)
- [AIRFLOW-6504] Allow specifying configmap for Airflow Local Setting (#7097)
- [AIRFLOW-6436] Cleanup for Airflow configs doc generator code (#7036)
- [AIRFLOW-6436] Add x_frame_enabled config in config.yml (#7024)
- [AIRFLOW-6436] Create & Automate docs on Airflow Configs (#7015)
- [AIRFLOW-6527] Make send_task_to_executor timeout configurable (#7143)
- [AIRFLOW-6272] Switch from npm to yarnpkg for managing front-end dependencies (#6844)
- [AIRFLOW-6350] Security - spark submit operator logging+exceptions should mask passwords
- [AIRFLOW-6358] Log details of failed task (#6908)
- [AIRFLOW-5149] Skip SLA checks config (#6923)
- [AIRFLOW-6057] Update template_fields of the PythonSensor (#6656)
- [AIRFLOW-4445] Mushroom cloud errors too verbose (#6952)
- [AIRFLOW-6394] Simplify github PR template (#6955)
- [AIRFLOW-5385] spark hook does not work on spark 2.3/2.4 (#6976)
Bug Fixes
"""""""""
- [AIRFLOW-6345] Ensure arguments to ProxyFix are integers (#6901)
- [AIRFLOW-6576] Fix scheduler crash caused by deleted task with sla misses (#7187)
- [AIRFLOW-6686] Fix syntax error constructing list of process ids (#7298)
- [AIRFLOW-6683] REST API respects store_serialized_dag setting (#7296)
- [AIRFLOW-6553] Add upstream_failed in instance state filter to WebUI (#7159)
- [AIRFLOW-6357] Highlight nodes in Graph UI if task id contains dots (#6904)
- [AIRFLOW-3349] Use None instead of False as value for encoding in StreamLogWriter (#7329)
- [AIRFLOW-6627] Email with incorrect DAG not delivered (#7250)
- [AIRFLOW-6637] Fix airflow test command in 1.10.x
- [AIRFLOW-6636] Avoid exceptions when printing task instance
- [AIRFLOW-6522] Clear task log file before starting to fix duplication in S3TaskHandler (#7120)
- [AIRFLOW-5501] Make default `in_cluster` value in KubernetesPodOperator respect config (#6124)
- [AIRFLOW-6514] Use RUNNING_DEPS to check run from UI (#6367)
- [AIRFLOW-6381] Remove styling based on DAG id from DAGs page (#6985)
- [AIRFLOW-6434] Add return statement back to DockerOperator.execute (#7013)
- [AIRFLOW-2516] Fix mysql deadlocks (#6988)
- [AIRFLOW-6528] Disable flake8 W503 line break before binary operator (#7124)
- [AIRFLOW-6517] Make merge_dicts function recursive (#7111)
- [AIRFLOW-5621] Failure callback is not triggered when marked Failed on UI (#7025)
- [AIRFLOW-6353] Security - ui - add click jacking defence (#6995)
- [AIRFLOW-6348] Security - cli.py is currently printing logs with password (#6915)
- [AIRFLOW-6323] Remove non-ascii letters from default config (#6878)
- [AIRFLOW-6506] Fix do_xcom_push defaulting to True in KubernetesPodOperator (#7122)
- [AIRFLOW-6516] BugFix: airflow.cfg does not exist in Volume Mounts (#7109)
- [AIRFLOW-6427] Fix broken example_qubole_operator dag (#7005)
- [AIRFLOW-6385] BugFix: SlackAPIPostOperator fails when blocks not set (#7022)
- [AIRFLOW-6347] BugFix: Can't get task logs when serialization is enabled (#7092)
- [AIRFLOW-XXXX] Fix downgrade of db migration 0e2a74e0fc9f (#6859)
- [AIRFLOW-6366] Fix migrations for MS SQL Server (#6920)
- [AIRFLOW-5406] Allow spark without kubernetes (#6921)
- [AIRFLOW-6229] SparkSubmitOperator polls forever if status json can't… (#6918)
- [AIRFLOW-6352] Security - ui - add login timeout (#6912)
- [AIRFLOW-6397] Ensure sub_process attribute exists before trying to kill it (#6958)
- [AIRFLOW-6400] Fix pytest not working on Windows (#6964)
- [AIRFLOW-6418] Remove SystemTest.skip decorator (#6991)
- [AIRFLOW-6425] Serialization: Add missing DAG parameters to Json Schema (#7002)
Misc/Internal
"""""""""""""
- [AIRFLOW-6467] Use self.dag i/o creating a new one (#7067)
- [AIRFLOW-6490] Improve time delta comparison in local task job tests (#7083)
- [AIRFLOW-5814] Implementing Presto hook tests (#6491)
- [AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing (#6516)
- [AIRFLOW-XXXX] Move airflow-config-yaml pre-commit before pylint (#7108)
- [AIRFLOW-XXXX] Improve clarity of confirm message (#7110)
- [AIRFLOW-6662] install dumb init (#7300)
- [AIRFLOW-6705] One less chatty message at breeze initialisation (#7326)
- [AIRFLOW-6705] Less chatty integration/backend checks (#7325)
- [AIRFLOW-6662] Switch to --init docker flag for signal propagation (#7278)
- [AIRFLOW-6661] Fail after 50 failing tests (#7277)
- [AIRFLOW-6607] Get rid of old local scripts for Breeze (#7225)
- [AIRFLOW-6589] BAT tests run in pre-commit on bash script changes (#7203)
- [AIRFLOW-6592] Doc build is moved to test phase (#7208)
- [AIRFLOW-6641] Better diagnostics for kubernetes flaky tests (#7261)
- [AIRFLOW-6642] Make local task job test less flaky (#7262)
- [AIRFLOW-6643] Fix flakiness of kerberos tests
- [AIRFLOW-6638] Remove flakiness test from test_serialized_db remove
- [AIRFLOW-6701] Rat is downloaded from stable backup/mirrors (#7323)
- [AIRFLOW-6702] Dumping kind logs to file.io. (#7319)
- [AIRFLOW-6491] Improve handling of Breeze parameters (#7084)
- [AIRFLOW-6470] Avoid pipe to file when do curl (#7063)
- [AIRFLOW-6471] Add pytest-instafail plugin (#7064)
- [AIRFLOW-6462] Limit exported variables in Dockerfile/Breeze (#7057)
- [AIRFLOW-6465] Add bash autocomplete for airflow in Breeze (#7060)
- [AIRFLOW-6464] Add cloud providers CLI tools in Breeze (#7059)
- [AIRFLOW-6461] Remove silent flags in Dockerfile (#7052)
- [AIRFLOW-6459] Increase verbosity of pytest (#7049)
- [AIRFLOW-6370] Skip Cassandra tests if cluster is not up (#6926)
- [AIRFLOW-6511] Remove BATS docker containers (#7103)
- [AIRFLOW-6475] Remove duplication of volume mount specs in Breeze.. (#7065)
- [AIRFLOW-6489] Add BATS support for Bash unit testing (#7081)
- [AIRFLOW-6387] print details of success/skipped task (#6956)
- [AIRFLOW-6568] Add Emacs related files to .gitignore (#7175)
- [AIRFLOW-6575] Entropy source for CI tests is changed to unblocking (#7185)
- [AIRFLOW-6496] Separate integrations in tests (#7091)
- [AIRFLOW-6634] Set PYTHONPATH in interactive Breeze
- [AIRFLOW-6564] Additional diagnostics information on CI check failure (#7172)
- [AIRFLOW-6383] Add no trailing-whitespace pre-commit hook (#6941)
Doc only changes
""""""""""""""""
- [AIRFLOW-XXXX] Consistency fixes in new documentation (#7207)
- [AIRFLOW-XXXX] Improve grammar and structure in FAQ doc (#7291)
- [AIRFLOW-XXXX] Fix email configuration link in CONTRIBUTING.rst (#7311)
- [AIRFLOW-XXXX] Update docs with new BranchPythonOperator behaviour (#4682)
- [AIRFLOW-XXXX] Fix Typo in scripts/ci/ci_run_airflow_testing.sh (#7235)
- [AIRFLOW-XXXX] Screenshot showing disk space configuration for OSX (#7226)
- [AIRFLOW-XXXX] Add mentoring information to contributing docs (#7202)
- [AIRFLOW-XXXX] Add rebase info to contributing (#7201)
- [AIRFLOW-XXXX] Increase verbosity of static checks in CI (#7200)
- [AIRFLOW-XXXX] Adds branching strategy to documentation (#7193)
- [AIRFLOW-XXXX] Move email configuration from the concept page (#7189)
- [AIRFLOW-XXXX] Update task lifecycle diagram (#7161)
- [AIRFLOW-XXXX] Fix reference in concepts doc (#7135)
- [AIRFLOW-XXXX] Clear debug docs (#7104)
- [AIRFLOW-XXXX] Fix typos and broken links in development docs (#7086)
- [AIRFLOW-XXXX] Clarify wait_for_downstream and execution_date (#6999)
- [AIRFLOW-XXXX] Add `airflow dags show` command guide (#7014)
- [AIRFLOW-XXXX] Update operation chaining documentation (#7018)
- [AIRFLOW-XXXX] Add `.autoenv_leave.zsh` to .gitignore (#6986)
- [AIRFLOW-XXXX] Fix development packages installation instructions (#6942)
- [AIRFLOW-XXXX] Update committers list (#7212)
- [AIRFLOW-XXXX] Move UPDATING changes into correct versions (#7166)
- [AIRFLOW-XXXX] Add Documentation for check_slas flag (#6974)
- [AIRFLOW-XXXX] Fix gcp keyfile_dict typo (#6962)
- [AIRFLOW-XXXX] Add tips for writing a note in UPDATIND.md (#6960)
- [AIRFLOW-XXXX] Add note warning that bash>4.0 is required for docs build script (#6947)
- [AIRFLOW-XXXX] Add autoenv to gitignore (#6946)
- [AIRFLOW-XXXX] Fix GCSTaskHandler Comment Typo (#6928)
- [AIRFLOW-XXXX] Fix broken DAG Serialization Link (#6891)
- [AIRFLOW-XXXX] Add versions_added field to configs
Airflow 1.10.7, 2019-12-24
--------------------------
New Features
""""""""""""
- [AIRFLOW-5088] [AIP-24] Persisting serialized DAG in DB for webserver scalability (#5992)
- [AIRFLOW-6083] Adding ability to pass custom configuration to AWS Lambda client. (#6678)
- [AIRFLOW-5117] Automatically refresh EKS API tokens when needed (#5731)
- [AIRFLOW-5118] Add ability to specify optional components in DataprocClusterCreateOperator (#5821)
- [AIRFLOW-5681] Allow specification of a tag or hash for the git_sync init container (#6350)
- [AIRFLOW-6025] Add label to uniquely identify creator of Pod (#6621)
- [AIRFLOW-4843] Allow orchestration via Docker Swarm (SwarmOperator) (#5489)
- [AIRFLOW-5751] add get_uri method to Connection (#6426)
- [AIRFLOW-6056] Allow EmrAddStepsOperator to accept job_flow_name as alternative to job_flow_id (#6655)
- [AIRFLOW-2694] Declare permissions in DAG definition (#4642)
- [AIRFLOW-4940] Add DynamoDB to S3 operator (#5663)
- [AIRFLOW-4161] BigQuery to Mysql Operator (#5711)
- [AIRFLOW-6041] Add user agent to the Discovery API client (#6636)
- [AIRFLOW-6089] Reorder setup.py dependencies and add ci (#6681)
- [AIRFLOW-5921] Add bulk_load_custom to MySqlHook (#6575)
- [AIRFLOW-5854] Add support for `tty` parameter in Docker related operators (#6542)
- [AIRFLOW-4758] Add GcsToGDriveOperator operator (#5822)
Improvements
""""""""""""
- [AIRFLOW-3656] Show doc link for the current installed version (#6690)
- [AIRFLOW-5665] Add path_exists method to SFTPHook (#6344)
- [AIRFLOW-5729] Make InputDataConfig optional in Sagemaker's training config (#6398)
- [AIRFLOW-5045] Add ability to create Google Dataproc cluster with custom image from a different project (#5752)
- [AIRFLOW-6132] Allow to pass in tags for the AzureContainerInstancesOperator (#6694)
- [AIRFLOW-5945] Make inbuilt OperatorLinks work when using Serialization (#6715)
- [AIRFLOW-5947] Make the json backend pluggable for DAG Serialization (#6630)
- [AIRFLOW-6239] Filter dags return by last_dagruns (to only select visible dags, not all dags) (#6804)
- [AIRFLOW-6095] Filter dags returned by task_stats (to only select visible dags, not all dags) (#6684)
- [AIRFLOW-4482] Add execution_date to "trigger DagRun" API response (#5260)
- [AIRFLOW-1076] Add get method for template variable accessor (#6793)
- [AIRFLOW-5194] Add error handler to action log (#5883)
- [AIRFLOW-5936] Allow explicit get_pty in SSHOperator (#6586)
- [AIRFLOW-5474] Add Basic auth to Druid hook (#6095)
- [AIRFLOW-5726] Allow custom filename in RedshiftToS3Transfer (#6396)
- [AIRFLOW-5834] Option to skip serve_logs process with ``airflow worker`` (#6709)
- [AIRFLOW-5583] Extend the 'DAG Details' page to display the start_date / end_date (#6235)
- [AIRFLOW-6250] Ensure on_failure_callback always has a populated context (#6812)
- [AIRFLOW-6222] http hook logs response body for any failure (#6779)
- [AIRFLOW-6260] Drive _cmd config option by env var (``AIRFLOW__CORE__SQL_ALCHEMY_CONN_CMD`` for example) (#6801)
- [AIRFLOW-6168] Allow proxy_fix middleware of webserver to be configurable (#6723)
- [AIRFLOW-5931] Use os.fork when appropriate to speed up task execution. (#6627)
- [AIRFLOW-4145] Allow RBAC roles permissions, ViewMenu to be over-ridable (#4960)
- [AIRFLOW-5928] Hive hooks load_file short circuit (#6582)
- [AIRFLOW-5313] Add params support for awsbatch_operator (#5900)
- [AIRFLOW-2227] Add delete method to Variable class (#4963)
- [AIRFLOW-5082] Add subject in AwsSnsHook (#5694)
- [AIRFLOW-5715] Make email, owner context available (#6385)
- [AIRFLOW-5345] Allow SqlSensor's hook to be customized by subclasses (#5946)
- [AIRFLOW-5417] Fix DB disconnects during webserver startup (#6023)
- [AIRFLOW-5730] Enable get_pandas_df on PinotDbApiHook (#6399)
- [AIRFLOW-3235] Add list function in AzureDataLakeHook (#4070)
- [AIRFLOW-5442] implementing get_pandas_df method for druid broker hook (#6057)
- [AIRFLOW-5883] Improve count() queries in a few places (#6532)
- [AIRFLOW-5811] Add metric for externally killed task count (#6466)
- [AIRFLOW-5758] Support the custom cursor classes for the PostgreSQL hook (#6432)
- [AIRFLOW-5766] Use httpbin.org in http_default (#6438)
- [AIRFLOW-5798] Set default ExternalTaskSensor.external_task_id (#6431)
- [AIRFLOW-5643] Reduce duplicated logic in S3Hook (#6313)
- [AIRFLOW-5562] Skip grant single DAG permissions for Admin role. (#6199)
- [AIRFLOW-6192] Stop creating Hook from SFTPSensor.__init__ (#6748)
- [AIRFLOW-5749][AIRFLOW-4162] Support the "blocks" component for the Slack operators (#6418)
- [AIRFLOW-5693] Support the "blocks" component for the Slack messages (#6364)
- [AIRFLOW-5714] Collect SLA miss emails only from tasks missed SLA (#6384)
- [AIRFLOW-5049] Add validation for src_fmt_configs in bigquery hook (#5671)
- [AIRFLOW-6177] Log DAG processors timeout event at error level, not info (#6731)
- [AIRFLOW-6180] Improve kerberos init in pytest conftest (#6735)
- [AIRFLOW-6159] Change logging level of the heartbeat message to DEBUG (#6716)
- [AIRFLOW-6144] Improve the log message of airflow scheduler (#6710)
- [AIRFLOW-6099] Add host name to task runner log (#6688)
- [AIRFLOW-6045] Error on failed execution of compile_assets (#6640)
- [AIRFLOW-5144] Add confirmation on delete button click (#6745)
- [AIRFLOW-6099] Add host name to task runner log (#6688)
- [AIRFLOW-5915] Add support for the new documentation theme (#6563)
- [AIRFLOW-5897] Allow setting -1 as pool slots value in webserver (#6550)
- [AIRFLOW-5888] Use psycopg2-binary for postgres operations (#6533)
- [AIRFLOW-5870] Allow -1 for pool size and optimise pool query (#6520)
- [AIRFLOW-6297] Add Airflow website link to UI docs (#6849)
Bug Fixes
"""""""""
- [AIRFLOW-XXX] Bump Jira version to fix issue with async
- [AIRFLOW-XXX] Add encoding to fix Cyrillic output when reading back task logs (#6631)
- [AIRFLOW-5304] Fix extra links in BigQueryOperator with multiple queries (#5906)
- [AIRFLOW-6268] Prevent (expensive) ajax calls on home page when no dags visible (#6839)
- [AIRFLOW-6259] Reset page to 1 with each new search for dags (#6828)
- [AIRFLOW-6185] SQLAlchemy Connection model schema not aligned with Alembic schema (#6754)
- [AIRFLOW-3632] Only replace microseconds if execution_date is None in trigger_dag REST API (#6380)
- [AIRFLOW-5458] Bump Flask-AppBuilder to 2.2.0 (for Python >= 3.6) (#6607)
- [AIRFLOW-5072] gcs_hook should download files once (#5685)
- [AIRFLOW-5744] Environment variables not correctly set in Spark submit operator (#6796)
- [AIRFLOW-3189] Remove schema from DbHook.get_uri response if None (#6833)
- [AIRFLOW-6195] Fixed TaskInstance attrs not correct on UI (#6758)
- [AIRFLOW-5889] Make polling for AWS Batch job status more resillient (#6765)
- [AIRFLOW-6043] Fix bug in UI when "filtering by root" to display section of dag (#6638)
- [AIRFLOW-6033] Fix UI Crash at "Landing Times" when task_id is changed (#6635)
- [AIRFLOW-3745] Fix viewer not able to view dag details (#4569)
- [AIRFLOW-6175] Fixes bug when tasks get stuck in "scheduled" state (#6732)
- [AIRFLOW-5463] Make Variable.set when replacing an atomic operation (#6807)
- [AIRFLOW-5582] Add get_autocommit to JdbcHook (#6232)
- [AIRFLOW-5867] Fix webserver unit_test_mode data type (#6517)
- [AIRFLOW-5819] Update AWSBatchOperator default value (#6473)
- [AIRFLOW-5709] Fix regression in setting custom operator resources. (#6331)
- [AIRFLOW-5658] Fix broken navigation links (#6374)
- [AIRFLOW-5727] SqoopHook: Build --connect parameter only if port/schema are defined (#6397)
- [AIRFLOW-5695] use RUNNING_DEPS to check run from UI (#6367)
- [AIRFLOW-6254] obscure conn extra in logs (#6817)
- [AIRFLOW-4824] Add charset handling for SqlAlchemy engine for MySqlHook (#6816)
- [AIRFLOW-6091] Add flushing in execute method for BigQueryCursor (#6683)
- [AIRFLOW-6256] Ensure Jobs table is cleared when resetting DB (#6818)
- [AIRFLOW-5224] Add encoding parameter to GoogleCloudStorageToBigQuery (#6297)
- [AIRFLOW-5179] Remove top level __init__.py (#5818)
- [AIRFLOW-5660] Attempt to find the task in DB from Kubernetes pod labels (#6340)
- [AIRFLOW-6241] Fix typo in airflow/gcp/operator/dataflow.py (#6806)
- [AIRFLOW-6171] Apply .airflowignore to correct subdirectories (#6784)
- [AIRFLOW-6018] Display task instance in table during backfilling (#6612)
- [AIRFLOW-6189] Reduce the maximum test duration to 8 minutes (#6744)
- [AIRFLOW-6141] Remove ReadyToRescheduleDep if sensor mode == poke (#6704)
- [AIRFLOW-6054] Add a command that starts the database consoles (#6653)
- [AIRFLOW-6047] Simplify the logging configuration template (#6644)
- [AIRFLOW-6017] Exclude PULL_REQUEST_TEMPLATE.md from RAT check (#6611)
- [AIRFLOW-4560] Fix Tez queue parameter name in mapred_queue (#5315)
- [AIRFLOW-2143] Fix TaskTries graph counts off-by-1 (#6526)
- [AIRFLOW-5873] KubernetesPodOperator fixes and test (#6523)
- [AIRFLOW-5869] BugFix: Some Deserialized tasks have no start_date (#6519)
- [AIRFLOW-4020] Remove DAG edit permissions from Viewer role (#4845)
- [AIRFLOW-6263] Fix broken WinRM integration (#6832)
- [AIRFLOW-5836] Pin azure-storage-blob version to <12 (#6486)
- [AIRFLOW-4488] Fix typo for non-RBAC UI in max_active_runs_per_dag (#6778)
- [AIRFLOW-5942] Pin PyMSSQL to <3.0 (#6592)
- [AIRFLOW-5451] SparkSubmitHook don't set default namespace (#6072)
- [AIRFLOW-6271] Printing log files read during load_test_config (#6842)
- [AIRFLOW-6308] Unpin Kombu for Python 3
Misc/Internal
"""""""""""""
- [AIRFLOW-6009] Switch off travis_wait for regular tests (#6600)
- [AIRFLOW-6226] Always reset warnings in tests
- [AIRFLOW-XXX] Remove cyclic imports and pylint hacks in Serialization (#6601)
- [AIRFLOW-XXX] Bump npm from 6.4.1 to 6.13.4 in /airflow/www (#6815)
- [AIRFLOW-XXX] Remove executable permission from file
- [AIRFLOW-XXX] Group AWS & Azure dependencies (old ``[emr]`` etc. extra still work)
- [AIRFLOW-5487] Fix unused warning var (#6111)
- [AIRFLOW-5925] Relax funcsigs and psutil version requirements (#6580)
- [AIRFLOW-5740] Fix Transient failure in Slack test (#6407)
- [AIRFLOW-6058] Running tests with pytest (#6472)
- [AIRFLOW-6066] Added pre-commit checks for accidental debug stmts (#6662)
- [AIRFLOW-6060] Improve conf_vars context manager (#6658)
- [AIRFLOW-6044] Standardize the Code Structure in kube_pod_operator.py (#6639)
- [AIRFLOW-4940] Simplify tests of DynamoDBToS3Operator (#6836)
- [AIRFLOW-XXX] Update airflow-jira release management script (#6772)
- [AIRFLOW-XXX] Add simple guidelines to unit test writing (#6846)
- [AIRFLOW-6309] Fix stable build on Travis
Doc only changes
""""""""""""""""
- [AIRFLOW-6211] Doc how to use conda for local virtualenv (#6766)
- [AIRFLOW-5855] Fix broken reference in custom operator doc (#6508)
- [AIRFLOW-5875] Fix typo in example_qubole_operator.py (#6525)
- [AIRFLOW-5702] Fix common docstring issues (#6372)
- [AIRFLOW-5640] Document and test ``email`` parameters of BaseOperator (#6315)
- [AIRFLOW-XXX] Improve description OpenFaaS Hook (#6187)
- [AIRFLOW-XXX] GSoD: How to make DAGs production ready (#6515)
- [AIRFLOW-XXX] Use full command in examples (#5973)
- [AIRFLOW-XXX] Update docs to accurately describe the precedence of remote and local logs (#5607)
- [AIRFLOW-XXX] Fix example "extras" field in mysql connect doc (#5285)
- [AIRFLOW-XXX] Fix wrong inline code highlighting in docs (#5309)
- [AIRFLOW-XXX] Group executors in one section (#5834)
- [AIRFLOW-XXX] Add task lifecycle diagram to documentation (#6762)
- [AIRFLOW-XXX] Highlight code blocks (#6243)
- [AIRFLOW-XXX] Documents about task_concurrency and pool (#5262)
- [AIRFLOW-XXX] Fix incorrect docstring parameter (#6649)
- [AIRFLOW-XXX] Add link to XCom section in concepts.rst (#6791)
- [AIRFLOW-XXX] Update kubernetes doc with correct path (#6774)
- [AIRFLOW-XXX] Add information how to configure pytest runner (#6736)
- [AIRFLOW-XXX] more GSOD improvements (#6585)
- [AIRFLOW-XXX] Clarified a grammatically incorrect sentence (#6667)
- [AIRFLOW-XXX] Add notice for Mesos Executor deprecation in docs (#6712)
- [AIRFLOW-XXX] Update list of pre-commits (#6603)
- [AIRFLOW-XXX] Updates to Breeze documentation from GSOD (#6285)
- [AIRFLOW-XXX] Clarify daylight savings time behavior (#6324)
- [AIRFLOW-XXX] GSoD: Adding 'Create a custom operator' doc (#6348)
- [AIRFLOW-XXX] Add resources & links to CONTRIBUTING.rst (#6405)
- [AIRFLOW-XXX] Update chat channel details from gitter to slack (#4149)
- [AIRFLOW-XXX] Add logo info to readme (#6349)
- [AIRFLOW-XXX] Fixed case problem with CONTRIBUTING.rst (#6329)
- [AIRFLOW-XXX] Google Season of Docs updates to CONTRIBUTING doc (#6283)
Airflow 1.10.6, 2019-10-28
--------------------------
New Features