[FIXED] apache-airflow-backport-providers-google fails to install with apache-airflow(1.10.12)

Issue

We use apache-airflow 1.10.12 for our current system. We need some classes from providers available in airflow 2.*. The airflow docs recommend to use these classes in airflow 1.10 using the backport package. This will allow us to test the new classes without jumping onto airflow 2. But when I install the backport dependency after the core airflow, I get the following error: "Failed to add packages, reverting the pyproject.toml to its original content" after throwing a bunch of warnings and errors.

Have you faced this before and were you able to resolve it in anyway?

To recreate the steps, here's what I did:

1. poetry init
2. poetry env use python3
3. poetry shell
4. poetry add apache-airflow="1.10.12"
5. poetry add apache-airflow-backport-providers-google (this is where tons of errors get thrown with the message mentioned before).

Solution

First of all Airflow 1.10.* has reached end-of-life as of June 17th 2021. It's not supported any more by the community and it will not receive any more fixes - not even critical security fixes. In fact there were already security fixes in Airflow 2 that have been fixed but they are not fixed in 1.10 and will never be, so you should upgrade to Airflow 2 as soon as possible otherwise you put your business at risk. You can find out more about it at our discussion panel at Airflow Summit two weeks ago: Keep your Airflow Secure.

Not upgrading to Airflow 2 now is a very bad idea (like any other outdated software), especially taking into account all the ransomware attacks and generally supply-chain attacks.

Secondly, you should not use poetry to install Airflow, not even the latest version. It is not supported. Airflow has complex dependencies and if you want to install airflow in reproducible way, you need to use pip and constraints mechanisms.

Poetry has very opinionated and simplistic view on the dependency resolution and it does not have constraint support and you will not be able to use constraints. In the latest documentation we specifically mention that you are not advised to use poetry to install airflow: https://airflow.apache.org/docs/apache-airflow/stable/installation.html#installation-tools

On the other hand pip has made a terrific progress recently with resolver and general resolution capabilities and avoiding/resolving conflicts.

The official way of installing airflow with 1.10.12 is here. Just follow it: https://airflow.apache.org/docs/apache-airflow/1.10.12/installation.html

Also it might happen than latest version of backport providers will generate other dependency problems because it stopped being released already more than 6 months ago. In this case you might try:

• pip install pip==20.2.4 before you install airflow - you might avoid some conflicts that were resolved after we released 1.10.12 and the backports.
• try some of the earlier version of the backport providers by specifying ==<version when you install it. You can see all the versions here: https://pypi.org/project/apache-airflow-backport-providers-google/#history

Also note that backport providers stopped being released more than 6 months ago and there were MANY fixes and improvements in airflow providers google since which makes it even more important to migrate to Airflow 2. The longer you delay the inevitable, the more problems you will have (on top of the aforementioned security problems)

Answered By - Jarek Potiuk