GitLab's annual major release is around the corner. Along with a lot of new and exciting features, there will be a few breaking changes. Learn more here.

Commit 6715e48e authored by Stefan Scherfke's avatar Stefan Scherfke
Browse files

Initial import

parents
Pipeline #57186386 skipped
*.egg-info/
*.pyc
*.ropeproject
*~
.DS_Store
.cache/
.coverage
.coverage.*
.idea/
.pytest_cache/
.tox/
MANIFEST
__pycache__/
build/
dist/
docs/_build/
htmlcov/
image: forge.services.own/centos7-ownconda-develop
before_script:
- conda update --quiet --yes --all own-conda-tools
stages:
- build
- test
- upload
- docs
build:
stage: build
script:
- ownconda build conda
artifacts:
expire_in: 7 days
paths:
- artifacts
test:
stage: test
script:
- ownconda test
coverage: '/^TOTAL.*\s+(\d+\%)\s*$/'
lint:
stage: test
script:
- ownconda lint src/* tests
security:
stage: test
script:
- ownconda sec-check -d B310 -d B321 -d B402 -d B604 src/* tests
upload:
stage: upload
only:
- master
except:
- schedules
script:
- ownconda upload
make_docs:
stage: docs
only:
- master
except:
- schedules
script:
- ownconda make-docs
# Blue Oak Model License
Version 1.0.0
## Purpose
This license gives everyone as much permission to work with
this software as possible, while protecting contributors
from liability.
## Acceptance
In order to receive this license, you must agree to its
rules. The rules of this license are both obligations
under that agreement and conditions to your license.
You must not do anything with this software that triggers
a rule that you cannot or will not follow.
## Copyright
Each contributor licenses you to do everything with this
software that would otherwise infringe that contributor's
copyright in it.
## Notices
You must ensure that everyone who gets a copy of
any part of this software from you, with or without
changes, also gets the text of this license or a link to
<https://blueoakcouncil.org/license/1.0.0>.
## Excuse
If anyone notifies you in writing that you have not
complied with [Notices](#notices), you can keep your
license by taking all practical steps to comply within 30
days after the notice. If you do not do so, your license
ends immediately.
## Patent
Each contributor licenses you to do everything with this
software that would otherwise infringe any patent claims
they can license or become able to license.
## Reliability
No contributor can revoke this license.
## No Liability
***As far as the law allows, this software comes as is,
without any warranty or condition, and no contributor
will be liable to anyone for any damages related to this
software or this license, under any kind of legal claim.***
include *.cfg
include *.rst
recursive-include src/own_conda_tools/resources *
recursive-exclude * *.py[co]
recursive-exclude * __pycache__
recursive-include docs *
recursive-exclude docs *.pyc
recursive-exclude docs/_build *
==============
ownconda-tools
==============
.. image:: https://gitlab.com/ownconda/ownconda-tools/badges/master/build.svg
:alt: build status
:target: https://gitlab.com/ownconda/ownconda-tools/commits/master
.. image:: https://gitlab.com/ownconda/ownconda-tools/badges/master/coverage.svg
:alt: coverage report
:target: https://gitlab.com/ownconda/ownconda-tools/commits/master
Tools for creating and maintaining our Conda packages.
Documentation
=============
You can find the documentation at
https://forge.services.own/docs/ownconda/ownconda-tools
$PYTHON -m pip install --no-deps .
#
# NOTE: Package meta data has to be sync'ed manually with setup.py!
#
package:
name: own-conda-tools
version: {{ environ.get('GIT_DESCRIBE_TAG', 0) }}.{{ environ.get('GIT_DESCRIBE_NUMBER', 0) }}
source:
path: ..
build:
number: {{ environ.get('CI_PIPELINE_ID', 0) }}
entry_points:
- ownconda = own_conda_tools.__main__:cli
requirements:
build:
- python
- pip
- setuptools
run:
- python
- attrs
- click >=6.6
- trio
# Recipe and dependency management
- beautifulsoup4
- jinja2
- networkx
- packaging
- pip
- requests
- ruamel.yaml >=0.15
- setuptools
# Testing
- pytest >=3.0
- pytest-cov
- pytest-sugar
# Linting
- astroid
- pylint
# Sec-Check
- bandit
# Docs
- sphinx
- sphinx-rtd-theme
# GitLab
- python-gitlab
# Prune
- pendulum
test:
requires:
- bandit
- pylint
- pytest
- pytest-cov
commands:
- ownconda --help
about:
home: https://gitlab.com/ownconda/ownconda-tools
license: Proprietary
summary: Utilities for creating and maintaining our Conda packages
#!/bin/bash
#
# Bootstrap the ownconda stack inside the current working directory.
#
# This scripts builds all Conda packages for the ownconda tools, conda-build
# and their dependencies and uploads them into the "stable" channel.
#
# With these packages, you can create an ownconda distribution and the Docker
# images.
uid=$(id -u)
gid=$(id -g)
docker run --rm -i -v ~/Projects/ownconda:/ownconda centos:centos7 /bin/bash -s << EOF
# Install build dependencies
yum install -y deltarpm epel-release wget bzip2 sudo
yum upgrade -y
yum groupinstall -y 'Development Tools'
yum install -y centos-release-scl
yum-config-manager --enable rhel-server-rhscl-7-rpms
yum install -y devtoolset-7
yum install -y rh-perl526
source /opt/rh/devtoolset-7/enable
# Install miniconda, conda-build and (via pip) ownconda-tools
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O /tmp/miniconda.sh
bash /tmp/miniconda.sh -p /tmp/mc -b
source /tmp/mc/bin/activate
conda install --yes 'conda-build<3.14'
cd /ownconda
pip install ./ownconda-tools
# Configure miniconda
cat <<IEOF >/tmp/mc/.condarc
channels:
- file:///tmp/mc/conda-bld
default_channels:
- file:///tmp/mc/conda-bld
add_pip_as_python_dependency: False
anaconda_upload: False
conda-build:
filename_hashing: false
IEOF
# Build a list with all packages we need to build
ownconda dep-graph --out=txt --print-path --implicit --requirements conda-build external-recipes > build.txt
ownconda dep-graph --out=txt --print-path --implicit --requirements own-conda-tools external-recipes ownconda-tools >> build.txt
sort -u build.txt > build.txt.sorted
mv -f build.txt.sorted build.txt
# Build packages and make resulting artifacts be owned by the current user
cat build.txt | xargs ownconda build --quiet
rm build.txt
chown -R $uid:$gid artifacts
# Upload packages
ownconda upload
EOF
import time
from gitlab.exceptions import GitlabCreateError
from own_conda_tools.click_util import Action
def task(project):
branches = {b.name for b in project.branches.list()}
assert 'master' in branches
with Action('Running pipeline for "master"') as action:
pipeline = project.pipelines.create({'ref': 'master'})
while True:
time.sleep(15)
action.progress()
pipeline.refresh()
if pipeline.finished_at is not None:
if pipeline.status != 'success':
raise RuntimeError('Pipeline failed')
break
#!/bin/env python
#
# Clone all listed projects and run their pipeline.
#
# By default, the script when a pipeline fails. Pass "-i" to ignore errors
# and run all pipelines.
import functools
import json
import os
import pathlib
import shutil
import subprocess
import sys
import click
ROOT_DIR = pathlib.Path.home() /'Projects' / 'gitlab-testing'
CONDA_ROOT = ROOT_DIR / '.conda'
INSTALLER_URL = 'https://forge.services.own/conda/stable/ownconda-dev.sh'
INSTALLER_LOC = ROOT_DIR / 'ownconda-dev.sh'
PROJECTS_PREFIX = ROOT_DIR
CONDA_CHANNELS = 'staging'
# CONDA_CHANNELS = 'testing,staging'
# CONDA_CHANNELS = 'experimental,staging'
# You can get a list of projects via "ownconda gitlab print-project"
PROJECTS = """
ownconda/ownconda-tools
""".strip().splitlines()
PROJECTS = [p for p in (row.partition('#')[0].strip() for row in PROJECTS) if p]
@click.command()
@click.option(
'--ignore-errors',
'-i',
is_flag=True,
default=False,
help='Do not exit when an error occurs. Print the number of errors at the end.',
)
@click.option(
'--skip-setup',
'-s',
is_flag=True,
default=False,
help='Skip clean setup and use existing Conda env',
)
@click.option(
'--branch',
'-b',
default='develop',
show_default=True,
help='Git branch to check out',
)
@click.option(
'--ci-opts',
default='--quiet --with-tests',
show_default=True,
)
@click.option(
'--channels',
'-c',
default=CONDA_CHANNELS,
show_default=True,
help='Additional Conda channels to use',
)
def cli(ignore_errors, skip_setup, branch, ci_opts, channels):
"""Clone all listed projects and run their pipeline."""
# Delete CONDA_ variables and update PATH
env = getenv(channels)
if not skip_setup:
setup(env)
run = functools.partial(subprocess.run, env=env, shell=True)
conda_rev = json.loads(
run('conda list --revisions --json', capture_output=True, text=True).stdout
)[-1]['rev']
failed_pipelines = []
for project in PROJECTS:
# Run pipeline
rc = run(
f'ownconda ci {ci_opts} '
f'--git-branch={branch} --git-update {project} {PROJECTS_PREFIX}'
).returncode
# Clean-up
# "ownconda ci" automatically deletes the artifacts
run(f'rm -rf {CONDA_ROOT / "conda-bld"}')
run(f'conda install --quiet --yes --rev {conda_rev}', capture_output=True)
if rc != 0:
if ignore_errors:
failed_pipelines.append(project)
else:
sys.exit(rc)
if failed_pipelines:
error = functools.partial(click.secho, fg='red', bold=True, err=True)
error(f'{len(failed_pipelines)}/{len(PROJECTS)} projects failed:')
for p in failed_pipelines:
error(f'- {p}')
def getenv(channels):
env = {k: v for k, v in os.environ.copy().items() if not k.startswith('CONDA_')}
env['PATH'] = ':'.join([
str(CONDA_ROOT / 'bin'),
str(CONDA_ROOT / 'condabin'),
'/usr/lib64/ccache',
'/usr/local/bin',
'/usr/bin',
'/usr/local/sbin',
'/usr/sbin',
])
env['CONDA_CHANNELS'] = channels
return env
def setup(env):
if ROOT_DIR.is_dir():
shutil.rmtree(ROOT_DIR)
ROOT_DIR.mkdir()
run = functools.partial(subprocess.run, env=env, shell=True, check=True)
run(f'wget {INSTALLER_URL} -O {INSTALLER_LOC}')
run(f'bash {INSTALLER_LOC} -p {CONDA_ROOT} -b')
run(f'conda update --quiet --yes --all own-conda-tools')
if __name__ == '__main__':
cli() # pylint: disable=no-value-for-parameter
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS = -n
SPHINXBUILD = sphinx-build
SPHINXPROJ = ownconda-tools
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
Put static files (images, ...) in here.
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!-- Created with Inkscape (http://www.inkscape.org/) -->
<svg
xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:cc="http://creativecommons.org/ns#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns="http://www.w3.org/2000/svg"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
width="512"
height="512"
viewBox="0 0 512.00001 512.00001"
id="svg2"
version="1.1"
inkscape:version="0.92.2 (5c3e80d, 2017-08-06)"
sodipodi:docname="ownconda-tools.svg">
<defs
id="defs4" />
<sodipodi:namedview
id="base"
pagecolor="#ffffff"
bordercolor="#666666"
borderopacity="1.0"
inkscape:pageopacity="0.0"
inkscape:pageshadow="2"
inkscape:zoom="0.65393567"
inkscape:cx="130.93023"
inkscape:cy="-92.242904"
inkscape:document-units="px"
inkscape:current-layer="layer1"
showgrid="false"
units="px"
inkscape:snap-bbox="true"
inkscape:bbox-nodes="true"
showborder="true"
inkscape:window-width="1920"
inkscape:window-height="1145"
inkscape:window-x="0"
inkscape:window-y="0"
inkscape:window-maximized="1">
<inkscape:grid
type="xygrid"
id="grid4136"
spacingx="1"
spacingy="1"
empspacing="8" />
</sodipodi:namedview>
<metadata
id="metadata7">
<rdf:RDF>
<cc:Work
rdf:about="">
<dc:format>image/svg+xml</dc:format>
<dc:type
rdf:resource="http://purl.org/dc/dcmitype/StillImage" />
<dc:title />
</cc:Work>
</rdf:RDF>
</metadata>
<g
inkscape:label="Ebene 1"
inkscape:groupmode="layer"
id="layer1"
transform="translate(0,-540.36216)">
<g
id="g4238">
<path
inkscape:connector-curvature="0"
style="fill:#ca6e1f;fill-opacity:1"
d="m 55.590758,867.20493 c -8.459463,16.90566 -15.3076,34.61636 -20.947242,52.72956 18.933084,5.23271 37.866168,8.85535 57.202084,10.46541 0.402832,-15.29559 1.611327,-29.78616 3.625484,-44.27673 -13.696273,-5.2327 -26.989715,-11.67295 -39.880326,-18.91824 z"
id="path4236" />
<path
inkscape:connector-curvature="0"
style="fill:#613c1f;fill-opacity:1"
d="m 132.93442,672.38733 c -11.27928,0 -22.15574,0.80503 -33.43502,2.41509 1.61133,11.27044 3.22265,22.13837 6.04247,33.40881 7.6538,-12.88051 16.91893,-24.55346 27.39255,-35.01887 l 0,-0.80503 z"
id="path4234" />
<path
inkscape:connector-curvature="0"
style="fill:#df781f;fill-opacity:1"
d="m 42.297316,937.64519 c 14.501937,22.13836 32.629358,41.86163 53.173768,58.36478 C 93.456927,978.70179 91.8456,962.19864 91.8456,945.6955 74.926674,944.08544 58.410579,941.26783 42.297316,937.64519 Z"
id="path4232" />
<path
inkscape:connector-curvature="0"
style="fill:#37281f;fill-opacity:1"
d="m 173.21758,554.0477 c -20.14158,6.84277 -38.67183,16.10063 -56.39642,27.37107 13.29344,2.4151 26.18405,5.23271 39.07466,9.25787 4.83398,-12.47799 10.87645,-24.55346 17.32176,-36.62894 z"
id="path4230" />
<path
inkscape:connector-curvature="0"
style="fill:#f48200;fill-opacity:1"
d="m 256.20088,540.36217 c -10.07078,0 -19.73874,0.80503 -29.4067,1.61006 13.69627,9.66038 26.58688,19.72327 39.07466,30.99371 l 10.07079,9.25786 -10.07079,10.0629 c -8.45946,8.05031 -16.51609,16.50314 -24.16989,25.761 l 0,0.40252 c 0,0 -1.61133,1.61006 -4.02832,4.42767 6.04248,-0.80503 12.08495,-0.80503 18.53025,-0.80503 96.27675,0 174.42608,78.08805 174.42608,174.28931 0,96.20126 -78.14933,174.2893 -174.42608,174.2893 -33.43502,0 -64.85588,-9.25786 -91.44276,-25.761 -13.29345,1.61006 -26.18406,2.41509 -39.4775,2.41509 -6.04247,0 -12.08495,0 -18.53025,-0.40251 0.40283,19.72327 2.41699,40.25157 5.63964,61.58495 41.08882,27.7735 90.23427,43.8742 143.40804,43.8742 141.39389,0 256.20089,-114.71701 256.20089,-256.00003 0.40283,-141.28302 -114.40417,-256 -255.79806,-256 z"
id="path4141" />
<path
inkscape:connector-curvature="0"
style="fill:#231f20;fill-opacity:1"
d="m 221.9602,600.73953 c 5.23681,-6.03774 10.47362,-12.07547 15.71043,-17.71069 -12.08495,-10.0629 -24.57272,-19.72327 -37.86617,-27.77359 -8.05663,13.68554 -15.3076,28.1761 -21.35007,42.66667 11.68212,4.42767 22.9614,9.25786 33.83785,14.49056 5.23681,-6.44025 8.8623,-10.4654 9.66796,-11.67295 z"
id="path4226" />
<path
inkscape:connector-curvature="0"
style="fill:#76461f;fill-opacity:1"
d="m 78.149327,677.62003 c -18.933085,3.22013 -37.463337,8.45283 -55.590759,14.89308 7.250969,18.51573 16.516095,36.62893 26.989716,53.53459 12.084948,-8.45283 24.572727,-16.50314 37.463337,-23.34591 -4.028316,-14.89308 -6.848137,-29.78616 -8.862294,-45.08176 z"
id="path4224" />
<path
inkscape:connector-curvature="0"
style="fill:#a05a1f;fill-opacity:1"
d="M 44.714305,774.2238 C 31.018032,785.49424 18.127421,797.56972 6.0424737,811.25525 19.335916,822.92821 33.435021,833.39361 48.33979,843.05399 55.187926,830.97852 62.841726,818.90305 70.898358,807.63261 62.036063,797.1672 53.173769,785.89676 44.714305,774.2238 Z"
id="path4222" />
<path
inkscape:connector-curvature="0"
style="fill:#df781f;fill-opacity:1"
d="m 109.97302,891.75839 c -1.61132,12.88051 -2.81982,26.16353 -3.22265,39.84906 12.89061,0.80503 25.78122,0.80503 38.67183,0 -13.69627,-11.27044 -25.78122,-24.55346 -35.44918,-39.84906 z"
id="path4220" />
<path
inkscape:connector-curvature="0"
style="fill:#4c321f;fill-opacity:1"
d="m 147.43636,613.21752 c -16.11326,-4.42768 -32.62936,-8.05032 -49.145454,-10.0629 -1.611327,16.90566 -1.611327,33.40881 -0.402832,49.91195 13.293446,-1.61006 26.184056,-2.41509 39.477496,-2.41509 2.41699,-12.88051 6.04247,-25.35849 10.07079,-37.43396 z"
id="path4218" />
<path
inkscape:connector-curvature="0"
style="fill:#37281f;fill-opacity:1"
d="m 170.39776,620.4628 c -3.22265,9.66038 -6.04248,19.32075 -8.45946,28.98113 11.27928,-7.24528 23.36423,-13.28302 35.85201,-17.71069 -8.8623,-4.02516 -18.12743,-7.6478 -27.39255,-11.27044 z"
id="path4216" />
<path
inkscape:connector-curvature="0"
style="fill:#b5641f;fill-opacity:1"
d="m 2.0141579,830.576 c 3.2226526,24.95598 10.0707891,48.70441 20.1415791,70.44026 5.23681,-14.49057 11.279284,-28.98114 18.127421,-42.66667 C 26.989716,849.89676 14.501937,840.6389 2.0141579,830.576 Z"
id="path4214" />
<path
inkscape:connector-curvature="0"
style="fill:#613c1f;fill-opacity:1"
d="m 75.732337,614.82758 c -15.3076,15.2956 -28.601042,32.20126 -39.880326,50.71698 13.293442,-4.02516 26.586884,-7.24528 40.283158,-9.66038 -0.805664,-13.68553 -1.208495,-27.37107 -0.402832,-41.0566 z"
id="path4212" />
<path