ACR - proxy cache for external docker registries
We would like to use ACR for hosting images created on public cloud, but also as a single endpoint registry for pulling images that are hosted on-prem or from external registries like dockerhub, gcr. Other OSS registry products like Nexus OSS provide this feature to act like a proxy or ‘pull-through cache’ for Docker Registries and helm repositories. This proxy should also support token based authentication to integrate with registries that require auth.
This would be useful for enterprises moving from on-prem to cloud to have a means to expose on-prem hosted images on cloud.
The challenges with a mirror are outlined in the co-authored OCI Blob post regarding consuming public content mentioned below https://opencontainers.org/posts/blog/2020-10-30-consuming-public-content/). In addition to mirrors reflecting the good and bad, the latency in resolution of content, the auth issue identified is another huge challenge.
Freegate notes: It would nice to be able to specify the repositories that we want to proxy.
This is another key feature we want to incorporate. Perhaps, not just filter by repo, but how many past images should be imported?
David: …The gated import workflow would not be ideal. -- Attempting to use ACR as a "mirror-registry" does not appear to work even when images are copied from Docker hub to ACR.
Can you elaborate a bit why this doesn’t work? Are you referring to the default registry issue? If you refer to node:9, importing to an ACR would require you to prepend the image with myregistry.azurecr.io/node:9. We recognize the challenge here and we’re exploring how we can change the way registries are referenced. See more here: Is It Time to Change How We Reference Container Images?
In the short term, including the domain also assures you can work with VNets, like ACR’s Private Link support (https://aka.ms/acr/privatelink)
Paul: What I really want is for us to be able to combine imports with a task which keeps that import fresh, so that we can count on our builds being snappy, reliable, and based on an up-to-date upstream image.
You’re correct, acr import (https://aka.ms/acr/import) is a one-off, and we do need some perf and security work to improve the performance and reduce the permissions required (contributor) to run it. However, if you look closely at the gated-import docs (https://aka.ms/acr/tasks/gated-import), you’ll notice we do use ACR Tasks (https://aka.ms/acr/tasks), with base image triggers to initiate the build, In addition to tracking the source registry (mirror source), Tasks can also be triggered by git commits and cron jobs (https://docs.microsoft.com/en-us/azure/container-registry/container-registry-tasks-scheduled#cron-expressions as triggers. https://aka.ms/acr/tasks/scheduling
We fully recognize the low level configuration of ACR Tasks/Gated Import workflows needs more productivity. I call it the (pointy-clicky) level of productivity. We’re currently investigating what this could look like. Imagine trying to establish a CI/CD system without any pointy-clicky tooling.
Imagine you could establish a gated import with a few clicks by specifying the source registry, optional auth needed to pull from the registry (including docker hub to avoid throttling, or private registries/repos). By default, the content is imported on change. However, you can add unit and functional tests as you need. So, you get the easy part, easily, and have the option to provide more “gates”.
By using a gated workflow, you can:
- Control the content you need in your secure supply chain
- Replicated to any region with ACR Geo-replication (https://aka.ms/acr/geo-replication)
- Replicate within a region with ACR Availability Zone support (https://aka.ms/acr/az)
- Place your content in a VNet (https://aka.ms/acr/privatelink), restricting access to the public registry from within the VNet
- Not be subject to internet outages
- Benefit from acr-on prem support (https://aka.ms/acr/connected-registry)
- Double encrypt at rest with Customer Managed Keys (https://aka.ms/acr/cmk)
- Scan the content you depend upon with your security scanning solution, like Azure Security Center, Palto Alto, Snyk, Aqua, …
Paul Gear commented
I agree with David Lewis that the gated workflow is non-ideal. Or more specifically, there seems little value in using ACR for a non-gated workflow, as 'az acr import' is a one-off event rather than something triggered by an upstream SHA update. What I really want is for us to be able to combine imports with a task which keeps that import fresh, so that we can count on our builds being snappy, reliable, and based on an up-to-date upstream image.
David Lewis commented
Now that docker hub has rate limiting, this feature is crucial for us. The gated import workflow would not be ideal. -- Attempting to use ACR as a "mirror-registry" does not appear to work even when images are copied from Docker hub to ACR.
While we're still considering a proxy cache, please review this post: https://opencontainers.org/posts/blog/2020-10-30-consuming-public-content/
We've provided an example for using ACR Tasks to create a gated-workflow: https://aka.ms/acr/tasks/gated-import
We are investigating tooling the gated-import workflow to be an easy cli, and/or a pointy clicky UI in the Azure Portal or possibly VS Code.
The question we have is; if we could only do one, a mirror or tool the gated workflow would you prefer?
Here's the tooling gated-import workflow user-voice item: https://feedback.azure.com/forums/903958-azure-container-registry/suggestions/41859736-tool-gated-import-workflows
Dmitry Makovey commented
Also posted in Visual Studio developer comminity:
Chris van der Pennen commented
Now that Docker hub are rate limiting pulls, this would be great to have.
Javid Salmanov commented
Very important and needed feature
Yes it will a great feature. So we could provide to our customer an unique registry that hosts all Docker images and Helm charts that we use. It would nice to be able to specify the repositories that we want to proxy.
REPAKA, RAO commented
this is a good feature to have.
This is a critical feature for us. Jay H.
Mallik Medarametla commented
Good idea. Please prioritize and get this feature available soon.
Gangaram Godipelly commented
This seems to be a good one to be included, I am voting for Thaniga's idea!!!
Suresh Manikonda commented
This will be a value added feature to the product and will help the enterprise teams.