Make our CI faster

[gabrieldemarmiesse]

Currently our CI take ~30 min to run. After some analysis, here is the critical path and the big steps:

• Building the wheel on linux: 25m
  • pull the docker image: 3m
  • install TF and dependencies: 2m
  • build CPU and GPU ops: 8m (building CPU ops only takes 2m in another build, so we can assume 6m for building the GPU ops)
  • Run all tests: 6m, I should re-enable pytest-xdist here.
  • Rebuild all ops for the wheel: 5m (because the bazel cache was cleaned before)
• Test the release wheel on Windows: 4m10s
• Upload to pypi, but not really: 25s.

I'll work toward making this faster, help and ideas are welcome too of course.

[bhack]

Can you point on the CI scripts sources?
Can we take any speed-up with https://github.com/actions/cache?

[bhack]

See also this thread https://github.community/t5/GitHub-Actions/Cache-docker-images/td-p/44978

[gabrieldemarmiesse]

Caching may help, but it may also make things slower. It's usually hard to know until we try.

[bhack]

I've not built the linux wheel Dockerfile with the progress option but I suppose that in 25m the main problem is that /root/.cache/bazel is an empty volume when you run the Action vs local build right?

[bhack]

I've asked if we there is a way to cache with action/cache the mount of buildkit RUN --mount=type=cache at actions/cache#260.

[bhack]

In the meantime I think that

--cache-from=type=registry,ref=$CACHE_TAG
--cache-to=type=registry,ref=$CACHE_TAG,mode=max

could work if we have a registry to push. See https://medium.com/titansoft-engineering/docker-build-cache-sharing-on-multi-hosts-with-buildkit-and-buildx-eb8f7005918e