The EntryPoint Image UI is very crufty

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

I am evaluating GitLab Enterprise. I was trying to use the image sqlfluff/sqlfluff in my pipeline. This is much more complex then I would like -- especially when you're selling to devops users. Here is my saga.

The Saga

The Image

The sqlfluff image which I don't control has these two lines,

# Set SQLFluff command as entry point for image.
ENTRYPOINT ["sqlfluff"]
CMD ["--help"]

It's made to be invoked like this,

podman run -it --rm -v "${PWD}:/sql" sqlfluff/sqlfluff lint /sql

The lint sql will normally just get passed to the entrypoint which is sqlfluff. This is part of the documented Docker conventions for EntryPoint.

Enter GitLab, the Ideal

When using this in GitLab, what I wanted to happen is something simple like this,

stages:
  - lint

sqlfluff:
  stage: lint
  image:
    name: sqlfluff/sqlfluff
    args: lint
  allow_failure: false

This would seem to me to convey the information to do exactly what I want.

My User Experience, the Real World

But I don't see this documented anywhere under script (likely documentation bug there). So without a pointer, I decided to use script and get the hands dirty. I don't like having to shoot in the dark, and I'm not sure what's supposed to happen without a doc link referencing this situation from script.

stages:
  - lint

sqlfluff:
  stage: lint
  image:
    name: sqlfluff/sqlfluff
  script:
    - lint
  allow_failure: false

This didn't work (obviously). I got this error

/bin/sh: 0: cannot open sh: No such file

Futile Debugging Attempt: Check image for faults

I try every variation of "script" to get the right invocation here. And no joy. Next I look up the images entrypoint locally and start to wonder if the image is broken or something.. So I try things like this podman inspect sqlfluff/sqlfluff | jq '.[0].Config' Everything looks good.

{
  "User": "5000",
  "Env": [
    "PATH=/app/.venv/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
    "LANG=C.UTF-8",
    "GPG_KEY=E3FF2839C048B25C084DEBE9B26995E310250568",
    "PYTHON_VERSION=3.9.21",
    "PYTHON_SHA256=3126f59592c9b0d798584755f2bf7b081fa1ca35ce7a6fea980108d752a05bb1",
    "VIRTUAL_ENV=/app/.venv"
  ],
  "Entrypoint": [
    "sqlfluff"
  ],
  "Cmd": [
    "--help"
  ],
  "WorkingDir": "/sql",
  "ArgsEscaped": true
}

Futile Debugging Attempt: Understand the runner error?

Then I try to figure out why I'm getting that error. My plan was to try to reproduce that error. So I try to jump into the container, changing the entrypint locally, and checking for /bin/sh.

$ podman run -ti --entrypoint /bin/sh  sqlfluff/sqlfluff -c -- "ls -lah /bin/sh"
lrwxrwxrwx 1 root root 4 Feb  3 00:00 /bin/sh -> dash

It's there... what about $PATH does that resolve sh?

$ podman run -ti --entrypoint /bin/sh  sqlfluff/sqlfluff -c -- "command -v sh"
/bin/sh

That works... Ok, next.

Futile Debugging Attempt: Understand the runner?

So my next option is to dive in the gitlab runner source code. I was searching and I can't figure out where the code is to troubleshoot this error. I see this message at the top of my runner:

Preparing the "docker+machine" executor
Using Docker executor with image sqlfluff/sqlfluff ...
Pulling docker image sqlfluff/sqlfluff ...

Looking up the docs on the executor, I see this

The Docker Machine executor was deprecated in GitLab 17.5 and is scheduled for removal in GitLab 20.0 (May 2027). While we continue to support the Docker Machine executor till GitLab 20.0, we do not plan to add new features. We will address only critical bugs that could prevent CI/CD job execution or affect running costs. If you’re using the Docker Machine executor on Amazon Web Services (AWS) EC2, Microsoft Azure Compute, or Google Compute Engine (GCE), migrate to the GitLab Runner Autoscaler.

Ok, that's not good maybe it's documented under GtLab Runner Autoscaler, but nope!

Side note: It's weird to have potential customers evaluate a product with a deprecated executor which is scheduled to be removed. My guess is that this docker+machine executor doesn't respect entrypoint.

I see in runner_manager.rb it's mapped to 'docker+machine' => :docker_machine,. But that's a dead end because I don't see the key for docker_machine anywhere. So moving on....

Futile Debugging Attempt: Figure out EntryPoint?

So maybe I can force an entrypoint. I look for some kind of pointer on this. I find tons of bugs and people complaining about entrypoint and gitlab. I found some documentation here. Note,

Command or script to execute as the container’s entry point.

When the Docker container is created, the entrypoint is translated to the Docker --entrypoint option. The syntax is similar to the Dockerfile ENTRYPOINT directive, where each shell token is a separate string in the array.

This is beautiful! That would work. Maybe I could just force the entrypoint to an absolute path and it would work.

  image:
    name: sqlfluff/sqlfluff
    entrypoint: ["/app/.venv/bin/sqlfluff"]

Figure out sqlfluff is supposed to resolve to /app/.venv/bin/sqlfluff. Think this will work. Try it. Nope, still getting that /sh error!

Now I'm really confused... What happens if I set it to /dev/null?

stages:
  - lint

sqlfluff:
  stage: lint
  image:
    name: sqlfluff/sqlfluff
    entrypoint: ["/dev/null"]
  script:
    - echo foo
  allow_failure: false

Nothing?!?! Are you kidding me?! At this point let's go back to Google and look for how to use entrypoint.

Succes

Finally, I find a clue here in this documentation page

To override the entrypoint of a Docker image, in the .gitlab-ci.yml file:

For Docker 17.06 and later, set entrypoint to an empty value.

This is also pretty bad. First, I don't want to "override" the entrypoint. The entrypoint is in the image. I want Gitlab to use the entry point in the image...

podman inspect sqlfluff/sqlfluff | jq '.[0].Config.Entrypoint'

But even if I want to override it setting the entrypoint to "" seems like a pretty wonky way to do it, because how then do I set the entrypoint to something I want, like /app/.venv/bin/sqlfluff? The answer is you can't. There is no method to "override" the entrypoint. There is only a method to "ignore" the entrypoint entirely. Perhaps there was a method to override at one point, but that is no more.

Action Items

It would be great to see these things addressed.

  • If ENTRYPOINT is set to anything but an empty string on 17.06 and later, you should fail .gitlab-ci parsing before CI runs. There is no way that can be correct. This would do a lot to save us time.
  • I would deprecate this whole empty string idea. You're making an empty string magical and it's not even documented in the config yaml. Instead consider this syntax,
    image:
      name: foo/bar
      ignore_image_entrypoint: true
    Isn't this better than entrypoint: [""]. If image.ignore_image_entrypoint is true, ensure image.entrypoint is not set.
  • Provide a way to actually override the entrypoint in the image, from the .gitlab-ci.yaml. While. I didn't need this functionality, because my specific image has a functioning shell. What if the image does not have a functioning shell. What if someone provides an image with one binary inside it built on top of scratch and the ENTRYPOINT is wrong? Then what? It seems there really should be an entrypoint for 17.06 and later too.
  • If image.ignore_image_entrypoint is not set (unset should be the default) there should be some way to use the entrypoint in an image providing arguments. I still don't see this documented or available anywhere. Just look at the line given above: podman run -it --rm -v "${PWD}:/sql" sqlfluff/sqlfluff lint /sql If I want to move that into an image, there should be someway to pass lint /sql the same way docker/podman accept it. Without having to override the EntryPoint to a shell, and to call the old EntryPoint manually. But, as always, if you can't provide arguments to the hard-coded EntryPoint in the image (like you can with Docker and podman), just come out and say it,

    "GitLab does not support providing arguments to an entrypoint hard coded in the image. Instead, you must ignore that entrypoint point, and from the shell invoke whatever command with whatever arguments are needed manually"

Edited by 🤖 GitLab Bot 🤖