The mockups below are showing two iterations. The first iteration would be to create the basic ability to upload files and have them added to pipelines. In the first iteration, each file would have a configurable name and permissions. In the second iteration, we would expand the capabilities for known file types and enrich them with additional metadata.
I've started an implementation discussion in this issue as well.
First Iteration
Second Iteration
I'm looking for feedback on this proposal and the naming, so please add any comments you have below, thanks!
thanks @darbyfrey - would you consider an API for upload and delete as a smaller MVC potentially and if it gets traction build out the UI to upload and manage those files?
Hi @jheimbuck_gl, yes I had been thinking that the MVC would be an API - I've got a few notes in here #346290 (closed). I suppose part of the purpose of this issue would be to make sure the UX vision would align with how things are being built on the backend. I also thought some mockups might be handy to visualize how the feature would work.
Thanks @darbyfrey sorry for jumping right into the weeds ;)
I could see this living under Security as well since SSH keys and certs are likely to be put into a place like this. Going forward tracking of usage for audits and restriction on use of pipelines or approval rules could be tied in as well eg; you can't use this secure file unless the MR / job is approved by this role maybe?
Overall this sounds like a great step forward. I'd check with @kbychu and @cbalane about any customer feedback they hear from a release perspective and how these types of files could be helpful there as well.
The usage tracking/auditing and pipeline restrictions are interesting, I hadn't considered those cases yet so I'll need to give them some more thought.
@darbyfrey I wonder if we can use our existing generic package feature for this? We already have an API for publishing generic files and we also have a status column, perhaps that can be expanded to include a secure parameter so they only show up in this view?
Thanks @trizzi! I dug into the generic_packages API a bit and I think the models are different enough that it would make sense to keep them separate for a couple of reasons:
We would want Secure Files to be encrypted at rest, which would add some conditional complexity to the generic file uploader.
The generic packages API makes sense in the context of packages, but I think it's a bit too much complexity for individual files. I've got a draft MR that I started to sketch out a simple API for secure files that I think could easily be expanded to support future iterations.
Happy to discuss further if there's something I'm missing though. Thanks again for the help @trizzi
Hi @jreporter and @rkenney2, any feedback or input would be appreciated. Even high-level opinions like naming (does Secure Files make sense as a name?), or positioning (it makes sense to me that this is a CI feature, but are there stages it might align with?)
Also, any other use cases outside of mobile you could point me to would be helpful as well.
Adding @supadhyaya for any PMM perspectives as well.
Thanks, @darbyfrey! I would say this definitely has applications outside of mobile. For example, people have asked to support protected variables as files, and I think this may have some applications for storing credentials in repos in a secure way. WDYT?
@darbyfrey@jreporter Any chance of consolidating approaches / solutions / product enhancements for: (1) Secure File storage, (2) build artifacts, (3) container images (also called "artifacts" outside of GitLab), (4) package binaries, (5) etc. ?? I'm happy to take the lead rounding up DRIs ...
ah I see... In that case, I recommend rolling this out in a canary or blue environment as an MVC proof of concept. I don't think this is a great fit right now for the testing Engineering team to take on, but I DO think it's a good idea. I'm hesitant to accidentally create more artifact product and engineering silos too. So I recommend trying this out with a limited Alpha User group, then reconvening in a cross-section-stage-group meeting to sync on artifacts in general and see where this would fit into the overall product design.
Yeah, Darby is what we call a Single Engineer Group who owns feature scoping and building as well as release. This will likely be used our Verify users on GitLab.com and we would want to build usage tracking to be able to interview the users using the offering later.
As far as deployment approach, we typically deploy behind a feature flag for Self-managed installations while keeping it usable by SaaS. Curating an alpha might be too labor intensive for this feature set.
Once this is released across self-manage and SaaS while being enabled by default I could see the maintenance of the feature running in Build Artifacts.
That's exactly the approach needed here. Good deal. My only callout is that by opening this up fully on SaaS, we will get bombarded with general comments and then accidentally bake in a POC approach that is not in sync with other artifact handling.
We do already have a UI for container images and packages. My understanding of this MVC is that it's focused on binary configuration files as part of their CI pipelines. I don't think we should recreate the container registry/package registry UI here.
(1) What are we using behind the scenes to manage secure credentials that are referenced within workflows? Vault? GitLab-acquired tool? Other approach?
(2) What processes and tools are we using to manage SSO / AD / LDAP ??
(3) What processes are we currently using to encrypt secure files at rest and in transit?
(4) What are we currently doing in terms of automated backups for files in general? Copy to fallback S3 in alt region? Other approach?
(5) Can we build a modular POC using Serverless Golang so that the process starts up fast and is very cheap?
These are good questions @rkenney2 and most can be found in the docs, I’d recommend searching and seeing if you can find some of these answers, since I am sure @darbyfrey will also have to search.
Best approach is to let a dedicated MVC proof of concept engineering team roll this out first, taking into consideration the above questions. Then circle back in a month.
Wanted to post a quick update for those who are following along here - I've been working on the implementation for this and have the first MR up for review. The first iteration is API only, the UI will come together after the API is in place.
For those following along, the MVC of this feature (API only) is now running on gitlab.com behind a feature flag. If you'd like to take it for a spin, please let me know https://docs.gitlab.com/ee/ci/secure_files/