This page may contain information related to upcoming products, features and functionality.
It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes.
Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.
Problem to solve
With the GitLab Dependency Proxy, you can proxy and cache container images hosted on Docker Hub, so that you can reduce your reliance on external dependencies and improve your build times.
The user interface for the Dependency Proxy is available at the group level and provides a copyable URL for using the proxy and lists the number of blobs currently stored in the cache. However, it does not currently display which image tags have been fetched, whether or not they are in the cache and how often they are used. This makes it difficult for the user to understand how effective this feature is and to validate that the correct image tags are being used.
As users begin to adopt the Dependency Proxy, the UI makes it clear that the feature is working as expected and helps the user to understand which images have been cached and when.
Proposal
As a Developer, when I navigate to my group's Dependency Proxy, I need to see which image tags have been fetched from Docker Hub and when so that I can validate that the correct image was used and troubleshoot when something has gone wrong.
As a Developer, when I am looking at which image tags have been cached using the Dependency Proxy, I'd like to see meta data for the image, so I can understand how it was built.
When a user lands on the Dependency Proxy main UI, they can click on the npmjs.org(default) remote registry to view the details. That detail view will include a list of packages pulled from the default. From there, we could display which packages have been pulled, which of those are cached (and for how long), and a warning if any of those show security warnings.
(beyond the MVC) As an Admin, When I am trying to understand how my team uses the Dependency Proxy, I need to know how often image tags are fetched from Docker Hub, how often they are pulled from the cache and the ratio of those two metrics.
Further details
As discussed in #241639 (comment 413250100), currently only the blobs are cached when an image is pulled, not the manifest. This will likely need to be resolved so that we can display meta data in the UI.
As discussed in this issue #215393 the design of the dependency proxy UI should also include an ability to flag possible security vulnerabilities being pulled in through the cache. So when an image tag is requested from Docker Hub, we also check against the known vulnerabilities database and include a warning in the UI.
Metadata
Name
Tags
Copyable install commands
Downloaded date
Last used
Size
OS/Arch
Testing and Availability
We currently have an existing end-to-end test that verifies the blob count on the Dependency Proxy. It is ideal that the test is updated to verify the image cached instead as it is available explicitly in the UI. To verify this test result we need to run package-and-qa on the MRs.
Tim Rizzichanged title from Design the Dependency Proxy to display info about image tags to The Dependency Proxy now displays container image tags and metadata
changed title from Design the Dependency Proxy to display info about image tags to The Dependency Proxy now displays container image tags and metadata
Tim Rizzichanged the descriptionCompare with previous version
changed the description
Tim Rizzichanged title from The Dependency Proxy now displays container image tags and metadata to The Dependency Proxy now displays cached container image tags
changed title from The Dependency Proxy now displays container image tags and metadata to The Dependency Proxy now displays cached container image tags
@nmezzopera@katiemacoy Now that the dependency proxy has GraphQL queries (#322839 (closed)) what do you think about updating the UI? We created designs for this a while ago, but it could be worth validating/updating as needed.
If you agree, I'd love to schedule this issue for milestone 14.4.
@trizzi just so I understand was #322839 (closed) exposing the data for things such as which image tags have been fetched from Dockerhub so now we have the data to display in the front end? Have those designs above been solution tested before?
@trizzi@nmezzopera
I archived the older designs and uploaded 2 options on this issue for my interpretation of the above comment 🔼 or you can view on Figma
This design is basically adapting patterns I've seen elsewhere so please feel to correct anything that isn't looking right.
I guess for MVC we could also remove certain things from the design, such as how the image was cached
@nmezzopera@trizzi one thing I'm not sure about is do we display all tags on this table or only the image? I am leaning towards all tags for the MVC because if we only show images there is no detail view (yet) for the user to click into to see which tags of that image
also @nmezzopera I'm just confirming if we have a bulk deletion pattern to follow (also related to issues 227715 and 336934, Figma here) so noting that the deletion UI from the attached design could change - I would guess any deletion would probably be post-MVC anyway and the first iteration of this would be a read-only table
if we have a bulk deletion pattern to follow (also related to issues 227715 and 336934
I'm not sure if a bulk deletion pattern will work in this case. Right now we really only have the option to purge the cache or set a time to live policy. It would be great to have a button to clear the cache on the page though. Although now I'm adding scope.
one thing I'm not sure about is do we display all tags on this table or only the image?
I'm open, but maybe for the MVC we start with just the images and then as we expand in the future, we can add a details page with the tags, manifest, and blobs info. Docker Hub typically presents a list of images and then you have to click into the image to see additional details. For example:
I'm not sure if a bulk deletion pattern will work in this case...
Makes sense. I will remove that functionality from the design for clarity
maybe for the MVC we start with just the images and then as we expand in the future...
I see your point. My assumption is that two people could be collaborating on a project and be using different tags of an image. If we only display the image name, how will I know which different tags are in the dependency proxy? What tag will we base the Cached xx days ago on?
If we only display the image name, how will I know which different tags are in the dependency proxy? What tag will we base the Cached xx days ago on?
That's a good point. I suppose we'd have to leave that information out for the MVC. What do you think @nmezzopera, can we include tag details in the MVC?
In the current mockups, I see cached at which I assume is the cache entry creation timestamp.
The read_at timestamp plays an important role in the dependency proxy policies. The duration [read_at, Time.zone.now] defines if a cache entry should still be kept or not.
In other words:
created_at: when the cache entry was created
read_at: last time that the cache entry was accessed (pulled)
This one is used in cleanup policies
The reasoning is that I want to avoid user feedback like:
I don't get it. My images have been created 100 days ago but the policy didn't remove them. This is not working
The answer to that is
The images were not removed because they were pulled less than 90 days ago = they are not expired from the cleanup policy point of view.
Last question for a future iteration:
The dependency proxy manages 2 types for data: blobs and manifests. It caches both. For example, at the time of this writing, if you pull alpine:latest through the dependency proxy, 3 objects are put in cache: 1 manifest and 2 blobs.
Should we display both? I'm not sure if this is useful. As a comparison, in the container registry pages, we don't display manifests and blobs, we present a single entity that is the container image tag. We might want to keep that for consistency.
Thanks for the feedback @10io! That's a really good point about read_at and created_at. I wonder if created_at is useful at all? What do you think @katiemacoy?
@10io thanks for your detailed explanation and feedback 🙇
My assumption of what customers want to know is primarily "is item xyz in the cache?" or "what's in the cache?". A secondary thing they might be curious about is how "fresh" an item is which based on your comment is read_at.
I would think putting the read_at value would be useful for the customer, but we'll need to adjust the language. cc @ngaskill
Image
Suggestions for changes
- Pulled 2 hours ago - Used 2 hours ago - Accessed 2 hours ago ...Something else?
@trizzi
David's comment makes me think we should incorporate some questions about this into an upcoming research project. Perhaps the post-MVP Dependency Proxy UI solution validation.
Questions I'd like to explore:
how relevant showing how "fresh" an item is in the cache?
what are reasons they are viewing the Dependency Proxy UI?
what information is missing? (e.g. manifests and blobs..)
what is their understanding of how the TTL works? I keep forgetting that the 90 day time resets when accessed.
Good idea @katiemacoy, I opened #343048 (closed) and added it to the problem validation backlog. I'm sure once we build this feature, we'll receive plenty of feedback from the community.
Be aware that the read_at attribute will get bumped each time the image is pulled. In other words:
created_at. It is set when the cache entry is created.
read_at. This one will get bumped each time the cache entry is used (for example when you pull an image).
If in the UI we have cached X hours ago and X depends on read_at, the UI will move each time the image is pulled. This could be confusing.
To me, when we say cached at X or cached X hours ago, this explicitly refers to the created_at attribute. Now my comment above is about: could have something for the read_at attribute so that users have everything in the UI to understand why a given image is kept or deleted by a cleanup policy.
@10io sorry I missed this! In future could you please @ me so I get notified?
Is your concern that the user will think we are pulling from Docker more often than we actually are? Because cached x hours ago sounds like we pulled it and re-cached it? Thus impacting their Docker rate limits?
We could show the following in the UI:
Cached at
Read at
How long an tag has left in the cache
Making a decision of what to show requires us to know why the user is visiting the UI, which we will explore in upcoming research.
My hypothesises:
People visiting the UI will be double-checking that something they recently published is in the cache.
People visiting the UI will be browsing what's in there.
The best way to order the list of tags in the UI is by most recent activity.
As long as users are not having problems with rate limiting or slow builds they don't need to how long the tag has left in the cache
Option 1: We don't know enough about why our users view the UI so we remove the field(s) from the UI for now
Option 2: We show the most recent activity in this field conditionally.
If the tag was just published, the UI says "Cached at" with the created_at attribute.
If the tag was just accessed the UI says "Accessed at" or "Read at" using the read_at attribute
Option 3: we show both the created_at and read_at in the UI.
I'm making a lot of guesses here so please chime in. cc @nmezzopera@trizzi
@katiemacoy Probably option 3 is the least confusing? But I think since this issue is workflowready for development and actively developed in this milestone we should move this discussion to a follow issue. WDYT?
In this issue I would like to apply a new set of checks to determine the issue readiness from the frontend point of view. This issue is ready to be worked but I want to start here regardless.
This issue has a clear, updated and confirmed design associated
The GraphQL API or the REST API contains all the data necessary to implement the UI
The GraphQL API or the REST API contains all the necessary mutation/actions to implement the UX
This issue copy is finalized or is clear that it will be finalized in the implementation phase
The impact on QA and Feature test of this issue is clear
This issue is not dependent to any other issue in the milestone, or the dependency is explicitly communicated and noted in the issue description.
This is great @nmezzopera, thanks for helping us to improve!
I'm doing a similar check for all our milestone issues, can you please confirm all topics are covered?
weight considered for development work
if this will require review from stable counterpart, a) they are aware b) they have capacity in the planned milestone c) if necessary work from stable counterpart, weight is adjusted
@michelletorres I think we are covering everything! I wonder if it would be better to explode point 2 in several points, to make it even more explicit. WDYT?
Summarizing here the conversation that I had with @katiemacoy in our 1:1:
Since the search and sorting can't be achieved because the API does not support it we will remove the search and sorting UI moment and add a title and a bit of margin to separate the list and the intro part.
No new design is needed, we will do the final adjustments in the MR phase.