Configuration & Environment Overrides

The cluster definition also supports overriding configuration properties and environment variables, either per role or per role group, where the more specific override (role group) has precedence over the less specific one (role).

Overriding certain properties which are set by operator (such as the HTTP port) can interfere with the operator and can lead to problems. Additionally, for Airflow it is recommended that each component has the same configuration: not all components use each setting, but some things - such as external end-points - need to be consistent for things to work as expected.

Configuration Properties

Airflow exposes an environment variable for every Airflow configuration setting, a list of which can be found in the Configuration Reference.

As Airflow can be configured with python code too, arbitrary code can be added to the webserver_config.py. You can use either EXPERIMENTAL_FILE_HEADER to add code to the top or EXPERIMENTAL_FILE_FOOTER to add to the bottom.

This is an experimental feature.

webservers:
  configOverrides:
    webserver_config.py:
      CSV_EXPORT: "{'encoding': 'utf-8'}"
      EXPERIMENTAL_FILE_HEADER: |
        from modules.my_module import my_class
      EXPERIMENTAL_FILE_FOOTER: |
        import logging
        from airflow.security import AirflowSecurityManager

        class myCustomSecurityManger(AirflowSecurityManager):
          def __init__():
            init()

        CUSTOM_SECURITY_MANAGER = myCustomSecurityManger
  roleGroups:
    default:
      config: {}

Although Kubernetes can override these settings in one of two ways (Configuration overrides, or Environment Variable overrides), the affect is the same and currently only the latter is implemented. This is described in the following section.

Environment Variables

These can be set - or overwritten - at either the role level:

webservers:
  envOverrides:
    AIRFLOW__WEBSERVER__AUTO_REFRESH_INTERVAL: "8"
  roleGroups:
    default:
      replicas: 1

Or per role group:

webservers:
  roleGroups:
    default:
      envOverrides:
        AIRFLOW__WEBSERVER__AUTO_REFRESH_INTERVAL: "8"
      replicas: 1

In both examples above we are replacing the default value of the UI DAG refresh (3s) with 8s. Note that all override property values must be strings.