Commit 6715e48e authored by Stefan Scherfke's avatar Stefan Scherfke

Initial import

parents
Pipeline #57186386 skipped
*.egg-info/
*.pyc
*.ropeproject
*~
.DS_Store
.cache/
.coverage
.coverage.*
.idea/
.pytest_cache/
.tox/
MANIFEST
__pycache__/
build/
dist/
docs/_build/
htmlcov/
image: forge.services.own/centos7-ownconda-develop
before_script:
- conda update --quiet --yes --all own-conda-tools
stages:
- build
- test
- upload
- docs
build:
stage: build
script:
- ownconda build conda
artifacts:
expire_in: 7 days
paths:
- artifacts
test:
stage: test
script:
- ownconda test
coverage: '/^TOTAL.*\s+(\d+\%)\s*$/'
lint:
stage: test
script:
- ownconda lint src/* tests
security:
stage: test
script:
- ownconda sec-check -d B310 -d B321 -d B402 -d B604 src/* tests
upload:
stage: upload
only:
- master
except:
- schedules
script:
- ownconda upload
make_docs:
stage: docs
only:
- master
except:
- schedules
script:
- ownconda make-docs
# Blue Oak Model License
Version 1.0.0
## Purpose
This license gives everyone as much permission to work with
this software as possible, while protecting contributors
from liability.
## Acceptance
In order to receive this license, you must agree to its
rules. The rules of this license are both obligations
under that agreement and conditions to your license.
You must not do anything with this software that triggers
a rule that you cannot or will not follow.
## Copyright
Each contributor licenses you to do everything with this
software that would otherwise infringe that contributor's
copyright in it.
## Notices
You must ensure that everyone who gets a copy of
any part of this software from you, with or without
changes, also gets the text of this license or a link to
<https://blueoakcouncil.org/license/1.0.0>.
## Excuse
If anyone notifies you in writing that you have not
complied with [Notices](#notices), you can keep your
license by taking all practical steps to comply within 30
days after the notice. If you do not do so, your license
ends immediately.
## Patent
Each contributor licenses you to do everything with this
software that would otherwise infringe any patent claims
they can license or become able to license.
## Reliability
No contributor can revoke this license.
## No Liability
***As far as the law allows, this software comes as is,
without any warranty or condition, and no contributor
will be liable to anyone for any damages related to this
software or this license, under any kind of legal claim.***
include *.cfg
include *.rst
recursive-include src/own_conda_tools/resources *
recursive-exclude * *.py[co]
recursive-exclude * __pycache__
recursive-include docs *
recursive-exclude docs *.pyc
recursive-exclude docs/_build *
==============
ownconda-tools
==============
.. image:: https://gitlab.com/ownconda/ownconda-tools/badges/master/build.svg
:alt: build status
:target: https://gitlab.com/ownconda/ownconda-tools/commits/master
.. image:: https://gitlab.com/ownconda/ownconda-tools/badges/master/coverage.svg
:alt: coverage report
:target: https://gitlab.com/ownconda/ownconda-tools/commits/master
Tools for creating and maintaining our Conda packages.
Documentation
=============
You can find the documentation at
https://forge.services.own/docs/ownconda/ownconda-tools
$PYTHON -m pip install --no-deps .
#
# NOTE: Package meta data has to be sync'ed manually with setup.py!
#
package:
name: own-conda-tools
version: {{ environ.get('GIT_DESCRIBE_TAG', 0) }}.{{ environ.get('GIT_DESCRIBE_NUMBER', 0) }}
source:
path: ..
build:
number: {{ environ.get('CI_PIPELINE_ID', 0) }}
entry_points:
- ownconda = own_conda_tools.__main__:cli
requirements:
build:
- python
- pip
- setuptools
run:
- python
- attrs
- click >=6.6
- trio
# Recipe and dependency management
- beautifulsoup4
- jinja2
- networkx
- packaging
- pip
- requests
- ruamel.yaml >=0.15
- setuptools
# Testing
- pytest >=3.0
- pytest-cov
- pytest-sugar
# Linting
- astroid
- pylint
# Sec-Check
- bandit
# Docs
- sphinx
- sphinx-rtd-theme
# GitLab
- python-gitlab
# Prune
- pendulum
test:
requires:
- bandit
- pylint
- pytest
- pytest-cov
commands:
- ownconda --help
about:
home: https://gitlab.com/ownconda/ownconda-tools
license: Proprietary
summary: Utilities for creating and maintaining our Conda packages
#!/bin/bash
#
# Bootstrap the ownconda stack inside the current working directory.
#
# This scripts builds all Conda packages for the ownconda tools, conda-build
# and their dependencies and uploads them into the "stable" channel.
#
# With these packages, you can create an ownconda distribution and the Docker
# images.
uid=$(id -u)
gid=$(id -g)
docker run --rm -i -v ~/Projects/ownconda:/ownconda centos:centos7 /bin/bash -s << EOF
# Install build dependencies
yum install -y deltarpm epel-release wget bzip2 sudo
yum upgrade -y
yum groupinstall -y 'Development Tools'
yum install -y centos-release-scl
yum-config-manager --enable rhel-server-rhscl-7-rpms
yum install -y devtoolset-7
yum install -y rh-perl526
source /opt/rh/devtoolset-7/enable
# Install miniconda, conda-build and (via pip) ownconda-tools
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O /tmp/miniconda.sh
bash /tmp/miniconda.sh -p /tmp/mc -b
source /tmp/mc/bin/activate
conda install --yes 'conda-build<3.14'
cd /ownconda
pip install ./ownconda-tools
# Configure miniconda
cat <<IEOF >/tmp/mc/.condarc
channels:
- file:///tmp/mc/conda-bld
default_channels:
- file:///tmp/mc/conda-bld
add_pip_as_python_dependency: False
anaconda_upload: False
conda-build:
filename_hashing: false
IEOF
# Build a list with all packages we need to build
ownconda dep-graph --out=txt --print-path --implicit --requirements conda-build external-recipes > build.txt
ownconda dep-graph --out=txt --print-path --implicit --requirements own-conda-tools external-recipes ownconda-tools >> build.txt
sort -u build.txt > build.txt.sorted
mv -f build.txt.sorted build.txt
# Build packages and make resulting artifacts be owned by the current user
cat build.txt | xargs ownconda build --quiet
rm build.txt
chown -R $uid:$gid artifacts
# Upload packages
ownconda upload
EOF
import time
from gitlab.exceptions import GitlabCreateError
from own_conda_tools.click_util import Action
def task(project):
branches = {b.name for b in project.branches.list()}
assert 'master' in branches
with Action('Running pipeline for "master"') as action:
pipeline = project.pipelines.create({'ref': 'master'})
while True:
time.sleep(15)
action.progress()
pipeline.refresh()
if pipeline.finished_at is not None:
if pipeline.status != 'success':
raise RuntimeError('Pipeline failed')
break
#!/bin/env python
#
# Clone all listed projects and run their pipeline.
#
# By default, the script when a pipeline fails. Pass "-i" to ignore errors
# and run all pipelines.
import functools
import json
import os
import pathlib
import shutil
import subprocess
import sys
import click
ROOT_DIR = pathlib.Path.home() /'Projects' / 'gitlab-testing'
CONDA_ROOT = ROOT_DIR / '.conda'
INSTALLER_URL = 'https://forge.services.own/conda/stable/ownconda-dev.sh'
INSTALLER_LOC = ROOT_DIR / 'ownconda-dev.sh'
PROJECTS_PREFIX = ROOT_DIR
CONDA_CHANNELS = 'staging'
# CONDA_CHANNELS = 'testing,staging'
# CONDA_CHANNELS = 'experimental,staging'
# You can get a list of projects via "ownconda gitlab print-project"
PROJECTS = """
ownconda/ownconda-tools
""".strip().splitlines()
PROJECTS = [p for p in (row.partition('#')[0].strip() for row in PROJECTS) if p]
@click.command()
@click.option(
'--ignore-errors',
'-i',
is_flag=True,
default=False,
help='Do not exit when an error occurs. Print the number of errors at the end.',
)
@click.option(
'--skip-setup',
'-s',
is_flag=True,
default=False,
help='Skip clean setup and use existing Conda env',
)
@click.option(
'--branch',
'-b',
default='develop',
show_default=True,
help='Git branch to check out',
)
@click.option(
'--ci-opts',
default='--quiet --with-tests',
show_default=True,
)
@click.option(
'--channels',
'-c',
default=CONDA_CHANNELS,
show_default=True,
help='Additional Conda channels to use',
)
def cli(ignore_errors, skip_setup, branch, ci_opts, channels):
"""Clone all listed projects and run their pipeline."""
# Delete CONDA_ variables and update PATH
env = getenv(channels)
if not skip_setup:
setup(env)
run = functools.partial(subprocess.run, env=env, shell=True)
conda_rev = json.loads(
run('conda list --revisions --json', capture_output=True, text=True).stdout
)[-1]['rev']
failed_pipelines = []
for project in PROJECTS:
# Run pipeline
rc = run(
f'ownconda ci {ci_opts} '
f'--git-branch={branch} --git-update {project} {PROJECTS_PREFIX}'
).returncode
# Clean-up
# "ownconda ci" automatically deletes the artifacts
run(f'rm -rf {CONDA_ROOT / "conda-bld"}')
run(f'conda install --quiet --yes --rev {conda_rev}', capture_output=True)
if rc != 0:
if ignore_errors:
failed_pipelines.append(project)
else:
sys.exit(rc)
if failed_pipelines:
error = functools.partial(click.secho, fg='red', bold=True, err=True)
error(f'{len(failed_pipelines)}/{len(PROJECTS)} projects failed:')
for p in failed_pipelines:
error(f'- {p}')
def getenv(channels):
env = {k: v for k, v in os.environ.copy().items() if not k.startswith('CONDA_')}
env['PATH'] = ':'.join([
str(CONDA_ROOT / 'bin'),
str(CONDA_ROOT / 'condabin'),
'/usr/lib64/ccache',
'/usr/local/bin',
'/usr/bin',
'/usr/local/sbin',
'/usr/sbin',
])
env['CONDA_CHANNELS'] = channels
return env
def setup(env):
if ROOT_DIR.is_dir():
shutil.rmtree(ROOT_DIR)
ROOT_DIR.mkdir()
run = functools.partial(subprocess.run, env=env, shell=True, check=True)
run(f'wget {INSTALLER_URL} -O {INSTALLER_LOC}')
run(f'bash {INSTALLER_LOC} -p {CONDA_ROOT} -b')
run(f'conda update --quiet --yes --all own-conda-tools')
if __name__ == '__main__':
cli() # pylint: disable=no-value-for-parameter
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS = -n
SPHINXBUILD = sphinx-build
SPHINXPROJ = ownconda-tools
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
Put static files (images, ...) in here.
This diff is collapsed.
==============================
Commands for local development
==============================
``develop`` – Install a package in editable/development mode
============================================================
.. automodule:: own_conda_tools.commands.develop
``pylintrc`` – Print the built-in *pylintrc* to *stdout*
========================================================
.. automodule:: own_conda_tools.commands.pylintrc
.. _cmd-gitlab:
``gitlab`` – Update or modifiy multiple GitLab projectes at once
================================================================
.. automodule:: own_conda_tools.commands.gitlab
==========================================
Commands for recipe and package management
==========================================
``pypi-recipe`` – Create or update recipes for PyPI packages
============================================================
.. automodule:: own_conda_tools.commands.pypi_recipe
``check-for-updates`` – Check if updates are available for external packages
============================================================================
.. automodule:: own_conda_tools.commands.check_for_updates
``update-recipes`` – Update one or more recipes
===============================================
.. automodule:: own_conda_tools.commands.update_recipes
``dep-graph`` – Create a dependency graph for a number of recipes
=================================================================
.. automodule:: own_conda_tools.commands.dep_graph
``validate-recipes`` – Check if recipes in folder are valid
===========================================================
.. automodule:: own_conda_tools.commands.validate_recipes
``show-updated-recipes`` – List recipes that are new than packages in our index
===============================================================================
.. automodule:: own_conda_tools.commands.show_updated_recipes
``prune-index`` – Delete old packages from the local Conda index at PATH
========================================================================
.. automodule:: own_conda_tools.commands.prune_index
===============================
Commands for the CI/CD pipeline
===============================
``build`` – Build a number of recipes in the right order.
=========================================================
.. automodule:: own_conda_tools.commands.build
``test`` – Wrapper for our test runner
======================================
.. automodule:: own_conda_tools.commands.test
``lint`` – Wrapper for our linter
=================================
.. automodule:: own_conda_tools.commands.lint
``sec-check`` – Run some security checks
========================================
.. automodule:: own_conda_tools.commands.sec_check
``upload`` – Upload Conda packages to the Conda repository
==========================================================
.. automodule:: own_conda_tools.commands.upload
``make-docs`` – Build and upload Sphinx documentation
=====================================================
.. automodule:: own_conda_tools.commands.make_docs
import os
import datetime
# -- General configuration ------------------------------------------------
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.doctest',
'sphinx.ext.intersphinx',
'sphinx.ext.viewcode',
]
templates_path = ['_templates']
source_suffix = '.rst'
master_doc = 'index'
language = 'en'
slug = 'ownconda-tools'
project = 'ownconda-tools'
version = os.environ.get('SPHINX_VERSION', '')
release = 'v{} ({}) – status: internal'.format(
os.environ.get('SPHINX_RELEASE', ''),
os.getenv("GIT_DESCRIBE_HASH", ''),
)
author = 'Stefan Scherfke'
date = datetime.date.today()
today_fmt = '%Y-%m-%d'
copyright = f'2016 {author} – {release} – {date:{today_fmt}}'
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
pygments_style = 'sphinx'
todo_include_todos = False
# -- Options for HTML output ----------------------------------------------
import sphinx_rtd_theme
html_static_path = ['_static']
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
html_theme = 'sphinx_rtd_theme'
html_theme_options = {
'display_version': False,
}
# -- Options for LaTeX output ---------------------------------------------
latex_engine = 'lualatex'
latex_elements = {
'papersize': 'a4paper',
'pointsize': '11pt',
'preamble': '',
'figure_align': 'htbp',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(
master_doc,
f'{slug}.tex',
project,
author.replace('&', '\\&'),
'manual',
),
]
# -- Plug-ins -------------------------------------------------------------
# Intersphinx
intersphinx_mapping = {
'python': ('https://docs.python.org/3/', None),
'click': ('http://click.pocoo.org/6/', None),
}
# Autodoc
autodoc_member_order = 'bysource'
============================================
Welcome to the ownconda-tools documentation!
============================================
Support tools for local development, CI/CD and Conda packaging.
Contents:
.. toctree::
:maxdepth: 2
install
commands
Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
======================
Installation and Setup
======================
Installation
============
The *ownconda-tools* should be installed into the root Conda env:
.. code-block:: console
$ conda install -n root own-conda-tools
Bash completion
===============
To enable Bash completion, copy the Script `ownconda-complete.sh`__ from the
repository to your machine (e.g., into your :file:`$HOME`) and source it from
your :file:`.bashrc`, e.g.:
.. code-block:: bash
source $HOME/ownconda-complete.sh
__ https://gitlab.com/ownconda/ownconda-tools/blob/master/ownconda-complete.sh
.. _configure-gitlab-access:
Configure GitLab Access
=======================
The :ref:`cmd-gitlab` commands acces GitLab via the `python-gitlab API`__ which
you need to configure first.
__ https://python-gitlab.readthedocs.io/
#. Create an access token for GitLab:
- Go to `User Settings / Access Tokens`__.
- Enter a name for your token (e.g., *python-gitlab*).
- An expiry date is not necessary.
- Check the scope :guilabel:`api`.
- Click :guilabel:`Create personal access token` and copy your token.
.. warning::
You can only view this token now. If you forget to copy it and close the
page, your token is gone forever and you need to create a new one.
__ https://gitlab.com/profile/personal_access_tokens