If we can identify the needs and match these up with solutions to be developed into the product or other solutions, we can move forward in removing the persistent access accounts.
Criteria for capability
Functionality
Potential Solution
delete attachments in response to customer requests
@lyle@lbot@shaunmccann@vparsons It would be great to have your collaboration in defining what would be required in the product and/or in other process to enable us to remove persistent rails console accounts for your team members. Is there someone that we can consider a Support DRI for identifying the needs and acceptance criteria?
One thing I'd like us to consider is what the new process would be for console escalations. We can open a separate issue to discuss if needed.
We already try in most cases to point customers to bug issues, but there are times when actions time out and something needs to be deleted in a short turn around time (1-2 days), including groups/projects/user accounts.
We occasionally also do work on behalf of dev/security. We'll need to update our handbook/issue templates to redirect them to the process.
@lyle + @cynthia I looked into the attachment manager solution (gitlab-org/gitlab#16229) which would be great but has so many implications and might be a really heavy lift (currently investigating).
How about an instance admin form where you can post attachment URL's which then would be deleted after basic checks? Would that be already sufficient as a first step?
@cynthia Thanks for the quick answer. Totally agree, but that might take a couple of releases to get there, based on the complexity of making that change. But that would be at least a fast mitigation for the rails console access.
@timzallmann thanks for the consideration. I foresee some of the other issues to take time as well, so if building at least a MVC for self-service management of uploads attachment could be planned for in the next few releases, then I'd prefer we focus on that rather than spending efforts building out both features.
Steve/Lyle can correct me if I'm wrong, but I believe the intent is to ensure we identify and plan for the eventual removal of rails console access for Support, but that the timeline has not been determined and we have some flexibility (meaning, there is no hard deadline).
the intent is to ensure we identify and plan for the eventual removal of rails console access for Support, but that the timeline has not been determined and we have some flexibility (meaning, there is no hard deadline).
The only thing I'd clarify is write access. Read-only Rails access through Teleport would still be available.
@timzallmann what should be the next step? Do we already have an issue open for: instance admin form where you can post attachment URL's which then would be deleted after basic checks. Also, do we want the ~"group::authentication and authorization" team to work on this?
@lyle If we were to develop an API (project/group Owner access) to remove the attachment from a project providing the direct link - would this be a good step towards eliminating that from the console
As a follow up we will probably need to add an audit event for this
If we had an API to remove an attachment scoped to a Owner, that would completely eliminate the need for us to intervene via the console. You have all of my thumbs-ups on this one.
I'll let @lyle comment on impact more. I believe it may be difficult to say how much that improves the need for console since we would need console of any one of the issues listed in the description.
@ogolowinski we don't currently tag our console issues with the specific issue, so there's no great way, but doing a keyword search on the issues shows we used to get them every couple of months, but haven't gotten one recently
Orit Golowinskichanged title from Enable removal of rails console access for Support to Enable removal of rails console access for Support for GitLab.com
changed title from Enable removal of rails console access for Support to Enable removal of rails console access for Support for GitLab.com
Just making sure it is clear that this issue is only for SaaS and not for SM users since we have offered some workaround for SM users using ails console
SM users have admin access and rails console access, and so have a great deal more control over what data is stored/presented to their users. This issue is not specifically about uploads though; it's a more general discussion about what SEs currently do with write access to the GitLab.com Rails console. This is just one example.
Please be tolerant with this comment, as I have joined the Customer Support - Console Escalations team recently and I may not have all the required information.
Still allow me to share a "fresh" view.
The problems that Console Escallations team is dealing with on daily basis have (or maybe even have not yet) been addressed as a separate issues but while waiting for issue's fix, we have to action the problems immediately (due to various reasons).
(Problems that have been resolved or have a suitable workaround already don't end up in our queue.)
With the short release life-cycle GitLab is running, there will always be new issues that require R/W actions in the console in order to resolve problems our customers are facing.
I don't see any way how we could prevent that.
I do understand the segregation of duties principle and securing access to prod environments to be critical.
Yet we still have to be able to support our customers even for newly appearing problems, don't we?
(If we revoke R/W access from Console Escalations team completely, who would handle those? Infra/SREs?)
Wouldn't it be wiser, to define secure enough rules for granting temporary R/W access instead of revoking R/W completely?
(Similar process is already happening for R/O access through teleport.)
@sloyd - we've had some movement in our team and I want to establish some guidelines around the max number of team members support will provision with this access as I've had a couple of requests float by my desk.
We still do need RW access to service certain requests, but I'm more than happy to follow y'all's lead on how accomplish that. It seems like at least for now though the best strategy is to have clear and strict guidelines for engineers who want to work console escalations.
@lyle thank you for your continued engagement to ensure we're limiting exposure and reducing risk. While I'm very interested in the results, we should engage more directly with the Reliability team to advise on methods. @alanrichards looking for your guidance here, potentially involving @afappiano and team or others.