AWS codeartifact requirements with python jobs
Description
Python AWS codeartifact OIDC variant/integration
Implementation ideas
Hi! using your templates we've come across several cases where in our pipelines we use codeartifact, the integration process is not completely clear with the Python component for the AWS CA registry, in contrast to the docker-ecr and aws-oidc which were intuitive and easy to use!
case 0:
building python packages at test stages that require internal dependencies from our codeartifact pypi registry
our workaround: overriding parts of the component's job to perform:
aws codeartifact login --tool pip --repository ${AWS_CODEARTIFACT_REPOSITORY_ENDPOINT} --domain ${AWS_CODE_ARTIFACT_DOMAIN} --domain-owner ${AWS_CODEARTIFACT_DOMAIN_OWNER} --region ${AWS_DEFAULT_REGION}
which in itself isn't idle I assume since awscli needs to be present.
case 1:
publishing built python package to AWS codeartifact simliar to case 0, here we use either poetry or twine, which means the codeartifact credentials process is a bit different, but the idea stays the same
our workaround: creating our own code-artifact-pypi job and implement the required identification/release
case 2: (might be more related to Docker component, lmk if you want me to transfer over there)
when building an image that require installations from AWS codeartifact, we would usually pass
a CODEARTIFACT_URL to the build args:
Dockerfile ARG CODEARTIFACT_URL=${CODEARTIFACT_URL} RUN pip install --no-cache-dir our-package --extra-index-url ${CODEARTIFACT_URL}
its unclear how should we create/achieve this using the component, since awscli is being used to create the CODEARTIFACT_URL
Thank you very much! wonderful product.
Activity
IMHO: Providing integration with every service is not a goal of TBC template.
But they should be extensible enough to allow to add those features and create additional "variant" template extensions if you want
You seems to have solved cases 0 and 1. The remains questions are:
- What could have been made to make this process easier ?
- Do you want to contribute a variant template with those features ?
For case 2, it is indeed specific to the docker template. One idea might make it easier to add a "before script" which would
exportthis variableEdited by Clement BoisI would love to contribute on this as we use TBC, and specifically these workflows on many parts of our CICD, though being not very experienced in devops, definitely not up to par with the expertise level of TBC. I would require some support in designing the correct approach to the issue at hand.
at the base level of things, looking at the
docker-ecr, perhaps the right approach would be to add apython-{aws,ca,oidc}variant of sorts, that would use theaws-auth-providerservice, similiar to thedocker-ecr, and then add endpoints for the service to provide the required login/variables forcodeartifactthrough theboto3which is already present there (no need to installawsclilike in our current workaround.this would be useful for case's 0-1 but perhaps also for case 2, since the 'docker-ecr' variant already relies on the
aws-auth-provider, though I'm not completely set on whats the best practice way for providing pip configuration for a build stage in a generalized way across multiple build system.another thought I've had in approaching this, is perhaps adding input parameters for providing pips configurations with
index-url|extra-index-urlvariables (as I don't think there is currently a way to do so in the base python template, besides hard coding in req*.txt files)implementing both these ideas I imagine the flow would look something likes this:
include: - component: $CI_SERVER_FQDN/to-be-continuous/python/gitlab-ci-python@7.1.0 - component: $CI_SERVER_FQDN/to-be-continuous/python/gitlab-ci-python-aws-ca@7.1.0 inputs: aws-oidc-role-arn: ... aws-region: ... aws-codeartifact-domain: ... aws-codeartifact-domain-owner: ... aws-codeartifact-repository: ...at the .python-base
gitlab-ci-python-aws-cawould:- identify with aws
- generate the tokens/url required for
codeartifact - set pip config values for
index-urlorextra-index-url
I've also joined the discord if further discussions required
at the base level of things, looking at the
docker-ecr, perhaps the right approach would be to add apython-{aws,ca,oidc}variant of sorts, that would use theaws-auth-providerservice, similiar to thedocker-ecr, and then add endpoints for the service to provide the required login/variables forcodeartifactthrough theboto3which is already present there (no need to installawsclilike in our current workaround.Very good guess, that's exactly the proper way to support this:
- extend
aws-auth-providerwith appropriate endpoints to obtain CodeArtifact creds - create a Python
codeartifactvariant that embedsaws-auth-provideras a service container and does the magic
This TBC design is very versatile as it allows adding features in a very decoupled manner (no need to build a super image with everything).
Don't hesitate to propose a Merge Request, we'll guide you through the process.
See: https://to-be-continuous.gitlab.io/doc/dev/workflow/ (3 pages)
- extend
extend
aws-auth-providerwith appropriate endpoints to obtain CodeArtifact credshttps://gitlab.com/EytanDNetz/aws-auth-provider-aws-codeartifact
would love your comments on this, pretty straight forward implementation.
My apologies for the redundant aws in the project name.
-
That's great. Could you create a Merge Request from your modified branch to the original repo main branch please? That will be much easier to review the changes.
...
Le lun. 21 oct. 2024, 14:37, Eytan Dahan (@EytanDNetz) gitlab@mg.gitlab.com a écrit :
Eytan Dahan https://gitlab.com/EytanDNetz commented on a discussion #85 (comment 2169056284):
extend aws-auth-provider https://gitlab.com/to-be-continuous/tools/aws-auth-provider with appropriate endpoints to obtain CodeArtifact creds
https://gitlab.com/EytanDNetz/aws-auth-provider-aws-codeartifact
would love your comments on this, pretty straight forward implementation.
My apologies for the redundant aws in the project name.
— Reply to this email directly or view it on GitLab #85 (comment 2169056284). You're receiving this email because of your account on gitlab.com. Unsubscribe https://gitlab.com/-/sent_notifications/REDACTED/unsubscribe from this thread · Manage all notifications https://gitlab.com/-/profile/notifications · Help https://gitlab.com/help
mentioned in merge request !126 (merged)
- auth-provider has been updated thanks @pismy!
- python-aws-ca is ready to be reviewed and merged as well.
-
The new version of
aws-auth-providerwith the CodeArtifact APIs is released.
mentioned in commit d97846a4
closed with merge request !126 (merged)
mentioned in merge request !128 (merged)
mentioned in commit 8a65e600
The release is available on GitLab release.
Your semantic-release bot
