Make Docs builds fallback in case external docs sources are missing

[potiuk]

Every now and then our docs builds start to fail because of external dependency (latest example here #14985). And while we are doing caching now of that information, it does not help when the initial retrieval fails. This information does not change often but with the number of dependencies we have it will continue to fail regularly simply because many of those depenencies are not very reliable - they are just a web page hosted somewhere. They are nowhere near the stabilty of even PyPI or Apt sources and we have no mirroring in case of problem.

Maybe we could

a) see if we can use some kind of mirroring scheme (do those sites have mirrrors ? )
b) if not, simply write a simple script that will dump the cached content for those to S3, refresh it in the CI scheduled (nightly) master builds ad have a fallback mechanism to download that from there in case of any problems in CI?

[potiuk]

Looks like our sphinx can handle it, but mirroring seems to be a feature of intersphinx mapping:

New in 1.3.

Alternative files can be specified for each inventory. One can give a tuple for the second inventory tuple item as shown in the 
following example. This will read the inventory iterating through the (second) tuple items until the first successful fetch. The
 primary use case for this to specify mirror sites for server downtime of the primary inventory:


intersphinx_mapping = {'python': ('https://docs.python.org/3',
                                  (None, 'python-inv.txt'))}

I think it is just a matter of:

a) automatically fetching the inventories from time to time
b) putting them on our S3 bucket with "web" property
c) adding automated mirroring from the S3 bucket.
d) we can even automatically update those S3 bucket inventories in our nightly scheduled build.

Should be easy to do and can save us a LOT of hassle.

[uranusjr]

Mirroring sounds like a good idea. What should happen if the nightly S3 bucket update fails?

[kaxil]

Mirroring sounds like a good idea. What should happen if the nightly S3 bucket update fails?

I think we somehow need to ignore the inventories if possible or building those providers if inventories are absolutely required. Would you like to take a stab at this @uranusjr :) ?

[uranusjr]

Sure! Could you provide some pointers where I should start looking for the code/configuration for the nightly builds?

[uranusjr]

I think I’ve it mostly figured out. Since we are already downloading the inventory locally ourselves into docs/_inventory_cache, we don’t really need to use the tuple form mentioned above. Instead, we can leave conf.py unchanged, but instead change fetch_inventories.py to check the mirror if the primary source fails during _inventory_cache population.

As mentioned above, the mirror inventory can be populated periodically. So a can will be added to ci.yml to fetch the inventory and populate them to S3.

Does that sound like a plan? I’ll try to come up with something.

[uranusjr]

First stab in #15024.

[ashb]

I think the build job in CI should always use the S3 cache - it is much more predictable and under our control that way.

And if we have the mirror as a separate workflow, then it can be triggered manually if we want to update it sooner than the nightly job.

[potiuk]

Looks great! @uranusjr !

I have another thought in the meantime. Why not using caching of GitHub Actions: https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows - the inventory files are not big (2.3 M):

[jarek:~/code/airflow/docs] master+ ± du -h --summarize _inventory_cache
2.3M	_inventory_cache

So we could simply cache it and use already existing inventory cache mechanism this way (we are already using the _inventory_cache as fallback)? That will probably be way simpler than setting up S3 and managing it periodically?

Also we could improve it to automatically add the "exact" version of each library we use in each inventory URL, that would make it much better - currently we are taking "latest" or "stable" but in fact this is wrong - we should look at our constraints file and use the exact version of the library we use - this should be rather simple to retrieve from constraints file. That would also make caching much more efficient - we could invalidate cache whenever any of the libraries change and get it rebuilt. And this works really nicely if we use restore-keys: part of caching already present in GithubAction.

Some pointers:

• we could take current library versions from : https://raw.githubusercontent.com/apache/airflow/constraints-master/constraints-3.8.txt
• we could simply download the constraint file and it could be used to calculate cache validity:
  with this key: cache-inventory-${{ hashFiles('constrant-3.8.txt') }}
  and this restore-keys: cache-inventory-
  the restore-keys work in the way that if the hash of such constraint file changes, it will download the "latest available" cache matching the prefix and use it as a base (but then after it is rebuilt, it will upload a new version of cache with the changed hash for next job to use it.
• and we could also use the constraints derive the right URLs when building inventory URLs to download the inventory from

WDYT? Happy to help and brainstorm on it, but this would be much simpler operationally (no separate S3 folder, no access needed etc. ) and more "correct" in terms of the documentation generated.

[uranusjr]

Using GitHub Action’s cache sounds like a cool idea. The only concern I have is the cache can be pretty difficult to purge if something goes wrong (maybe just me not knowing a good way to do it). Maybe we should have a “cache busting” part to the key (e.g. cache-inventory-${{ hashFiles('constrant-3.8.txt') }}-${{ cacheKey }} where cacheKey is a variable in the Action’s config) so we can force a cache miss when we need to.

Generally constraints (and requirements) file syntax is considered pip internals, but the files in this case are entirely managed by Airflow anyway, so that’s not really a concern.

We’ll probably need check if each third-party has a versioned documentation as well (not a given). And if one doesn’t, just default to latest?

[uranusjr]

I just thought of another thing, does Action’s cache set file creation date etc. correctly? Because if it does not, we won’t be able to fetch in new documentation updates (say the 1.4 version of a documentation releases a fix without changing the URL) since the files we retrieve from the cache will always be newer.

[potiuk]

Using GitHub Action’s cache sounds like a cool idea. The only concern I have is the cache can be pretty difficult to purge if something goes wrong (maybe just me not knowing a good way to do it). Maybe we should have a “cache busting” part to the key (e.g. cache-inventory-${{ hashFiles('constrant-3.8.txt') }}-${{ cacheKey }} where cacheKey is a variable in the Action’s config) so we can force a cache miss when we need to.

In this case we will simply change the cache prefix. We've done that several times with other caches. We can always turn it into:

key: cache-inventory-v2-${{ hashFiles('constraints-3.8.txt') }}
restore-key: cache-inventory-v2-

by changing the .yml file. No need for special variable for that it can simply be changed in ci.yml. And I think this cache will be self-healing. As I understand it the inventory we are using now simply downloads URL and only if it fails, it reads it from the cache @mik-laj might confirm it maybe?

I just thought of another thing, does Action’s cache set file creation date etc. correctly? Because if it does not, we won’t be able to fetch in new documentation updates (say the 1.4 version of a documentation releases a fix without changing the URL) since the files we retrieve from the cache will always be newer.

Highly doubt it. Even if they do, this is not a big problem for us. First of all the cache will always be secondary choice - so even if we have it from the cache, the "URL one" will take precedence (Or this is at least I think how it works ). And secondly the cache will get "eventually consistent". Every time we change constraints (should happen regularly - every day or so) the cache will get refreshed because while we will have it as secondary choice, the main one (URL pull) will still get the latest version and override and then new version of the cache will be produced.