Add prohibited filename exemptions

more details and examples can be found here

https://stackoverflow.com/questions/54240398/figuring-out-re2-equiv-for-negative-lookback-in-gitlab

added customer label

Note the reason we introduced RE2 due to https://gitlab.com/gitlab-org/gitlab-ce/issues/24570. It looks like RE2 doesn't plan on supporting negative lookahead because of https://github.com/google/re2/issues/156#issuecomment-354109175.

/cc: @nick.thomas

@stanhu yes, lack of negative lookahead/lookback are basically why we're using RE2. It's simply not safe to provide untrusted regexps to the ruby standard regexp library.

I'd suggest that as we come across use cases that can't be expressed in the more-limited regexp functionality, we could consider splitting them into their own push rule section. "Forbid these paths/extensions" or "Only permit these paths/extensions" is something we could support without requiring people to write regexps like the one in this issue's description, for instance, which I'm having a very hard time unpicking.

WDYT @jramsay ?

great, in the mean time I have an audit compliance issue. what do you suggest I do? cuz right now the leading option is to roll back gitlab.

Heres the explanation of what we do.

^ = start of line
(?!(?:.*\/)?python\/shim(?:\/|$)) = allow anything in a python/shim folder, anything in it is allowed
(?!(?:.*\/)?src\/test\/resources(?:\/|$)) = allow anything in the src/test/resources folder, anything in it is allowed
(?!.*gradle-wrapper\.jar$) = allow filenames gradle-wrapper.jar, no matter where its at
.*\.?:jar|exe|a|ar|cpio|shar|iso|LBR|lbr|mar|sbx|tar|bz2|F|gz|lz|lzma|lzo|rz|sfark|sz|xz|z|Z|7z|s7z|ace|afa|alz|apk|arc|arj|b1|ba|bh|cab|car|cfs|cpt|dar|dd|dgc|dmg|ear|gca|ha|hki|ice|kgb|lzh|lha|lzx|pak|partimg|paq6|paq7|paq8|pea|pim|pit|qda|rar|rk|sda|sea|sen|sfx|shk|sit|sitx|sqx|tar\.gz|tgz|tar\.Z|tar\.bz2|tbz2|tar\.lzma|tlz|uc|uc0|uc2|ucn|ur2|ue2|uca|uha|war|wim|xar|xp3|yz1|zip|zipx|zoo|zpaq|zz|dll) 
= flag any other file that has any of these extensions.
$ = end of line

@scphantm one option might be to use a custom pre-receive hook to reject non-compliant pushes, instead of the regexp? These are documented here: https://docs.gitlab.com/ee/administration/custom_hooks.html and can be arbitrarily complex. You can set them globally, or just for specific projects.

I can't recommend downgrading - if you're only coming across this problem now, that suggests you've upgraded from a version that's been out of security support for a long time.

@nick.thomas One of our issues is that we are not allowed direct access to the directories running the Gitlab server. We have a very strict security policy that denies the creation or updating of files on a running application server. So, we are unable to create custom hooks which must be placed on the Gitlab server.

it would be more accurate to say that we upgraded to 10.7 shortly after that was released due to other bugs and had no indication that this stopped working. It was just dumb luck that I was updating the regex on one of our projects this week and realized it wasn't working. Started searching and found every project created since we upgraded to 10.7 has no push rule, and many of those projects are already violating policy. So we are likely going to have to nuke those projects and tell them to start over again (screw the history, just create a new repo).

i was looking for hooks, i couldn't remember where they lived. but yea, this may be a solution.

I was just thinking. We know that the regex thats above still works, because we get calls that the system blocked pushes all the time, thats what prompted me to go and try to modify it. So am I correct in that RE2 is used as a validator on the input, but Ruby is used for the actual execution?

If thats the case, we may be able to use the configuration system we built (when we adjust settings, its not to one project, its to hundreds, so we wrote a system to do it) to insert the regex directly into the database. That may get us by until we come up with a better idea.

I'd suggest that as we come across use cases that can't be expressed in the more-limited regexp functionality, we could consider splitting them into their own push rule section. "Forbid these paths/extensions" or "Only permit these paths/extensions" is something we could support without requiring people to write regexps like the one in this issue's description, for instance, which I'm having a very hard time unpicking.

@nick.thomas we had the same issue with the push rule for commit messages.
That's why I added a new field for it in 11.1 that does exactly what you're suggesting.
So I agree with you that's probably the easiest solution.

Is there any update or solution to this issue? We would like to know how we can resolve this through the UI without modifications to any hooks on the server.

changed the description

The merge request referred to by @haynes is https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5453, which I think takes precedence over allow (if the commit message is both allowed and rejected, we reject wins). @nick.thomas is this correct?

@Neumsy could you confirm I understand your intention: reject all commits containing files with certain extensions, except if they are in a specific directory?

That is the intention of the regex, correct.

Sorry for the late reply, I was on vacation and I am just returning today.

We would not just want a specific directory, but specific files or directory.

For example, we want to deny any JAR files from being committed, unless they are in a specific directory with a specific file name.

Deny: .jar Allow: /gradle/gradlewrapper.jar,*/excemptjars/.jar

This would deny all JAR files except for the gradlewrapper*.jar files under the gradle folder and any JAR file under the excemptjars folder.

added devopscreate repository ~2975006 labels

changed title from RE2 does not allow for negative look back/ahead causes usability issues for prohibited files. to Add prohibited filename exemptions

changed the description

@scphantm @Neumsy I think the above regex could be replicated with:

prohibited filenames: (jar|exe|a|ar|cpio|shar|iso|LBR|lbr|mar|sbx|tar|bz2|F|gz|lz|lzma|lzo|rz|sfark|sz|xz|z|Z|7z|s7z|ace|afa|alz|apk|arc|arj|b1|ba|bh|cab|car|cfs|cpt|dar|dd|dgc|dmg|ear|gca|ha|hki|ice|kgb|lzh|lha|lzx|pak|partimg|paq6|paq7|paq8|pea|pim|pit|qda|rar|rk|sda|sea|sen|sfx|shk|sit|sitx|sqx|tar\.gz|tgz|tar\.Z|tar\.bz2|tbz2|tar\.lzma|tlz|uc|uc0|uc2|ucn|ur2|ue2|uca|uha|war|wim|xar|xp3|yz1|zip|zipx|zoo|zpaq|zz|dll)$
exempted filenames: *(python\/shim\/.*|src\/test\/resources\/.*|gradle-wrapper\.jar)$

Requested by https://gitlab.my.salesforce.com/0016100000Sudgq

changed milestone to %Backlog

added [deprecated] Accepting merge requests label

Just noticed this was in the CE tracker, but push rules are in GitLab Starter

@jramsay I believe those two entries would solve our issue.

mentioned in issue #60286 (moved)

Add prohibited filename exemptions

Problem to solve

Further details

Proposal

Links / references

Designs

Child items ...

Activity