Commit e980ab3b authored by Deimos's avatar Deimos

Initial open-source release

parents
.vagrant
Customfile
__pycache__
*.py[co]
*.egg-info/
.cache/
.mypy_cache/
.webassets-cache/
.webassets-manifest
*.gz
*.log
# don't track the built versions of CSS and JS
tildes/static/css/*
tildes/static/js/third_party.js
tildes/static/js/tildes.js
# don't track the site-icons spritesheet(s)
tildes/static/images/site-icons*
# Contributing to Tildes development
Outside of actual code changes, there are several other ways to contribute to Tildes development:
* Known issues and plans for upcoming changes are [tracked on GitLab](https://gitlab.com/tildes/tildes/issues). If you've found a bug/problem on the site or have a suggestion, please feel free to submit an issue for it (if one doesn't already exist). If you have a Tildes account, you can also post in [the ~tildes group](https://tildes.net/~tildes) with issues, suggestions, or questions.
* The [Tildes Docs site](https://docs.tildes.net) is also open-source and accepts contributions. It has a separate repository, also on GitLab: https://gitlab.com/tildes/tildes-static-sites
* [Donating to Tildes](https://docs.tildes.net/donate) supports its development directly by enabling Deimos to continue working on it full-time. Tildes is a non-profit with no investors and no advertising. Its operation and development relies on donations.
## License
Please take note that Tildes uses [the AGPLv3 license](https://www.gnu.org/licenses/why-affero-gpl.html). This means that if you use the Tildes code to run your own instance of the site, you *must* also open-source the code for your site, including any changes that you've made. If you are not able to open-source the code for your instance, you can not use the Tildes code to run it.
## Setting up a development version
Please see this page on the Tildes Docs for instructions to set up a development version: https://docs.tildes.net/development-setup
## General development information
This page on the Tildes docs contains information about many aspects of Tildes development: https://docs.tildes.net/development
## Contributing code
**Please do not work on significant changes or features without first ensuring that the changes are desired.** If there isn't already an issue related to the change you're intending to make, please [create one first](https://gitlab.com/tildes/tildes/issues/new) to discuss your plans. Similarly, even when there *is* already an issue, if the specifics of how it should work aren't clear, please try to sort that out first (by posting comments on the issue) before doing the work.
In general, anything beyond straightforward fixes and adjustments should have an associated issue. This is for everyone's benefit—it ensures that you don't waste effort working on changes that won't be accepted, and makes it easier for other contributors to see what's already being worked on.
### Choosing what to work on
If you look at [the list of issues](https://gitlab.com/tildes/tildes/issues), there are a few indicators you can use to find a good contribution to work on:
* Make sure the issue isn't already being worked on by someone else. Check the comments on it, and whether it's set as "assigned to" someone. If someone else was working on it but there hasn't been any activity in a while, please post a comment first to check if they're still making progress.
* Issues with the "high priority" label are ones that are needed soon, and contributions for these will probably be most appreciated. However, they may also be more complex than many other issues, and might not be the best place for a newer contributor to start.
* Check the "Weight" field on the issue. If set, this is an estimate of how complex it will be to address, ranging from 1 (extremely simple, very small changes) to 9 (major changes, possibly requiring days or even weeks of work).
Once you've selected an issue to work on, please leave a comment saying so. This will allow other people to see that they shouldn't also start on that issue.
### Before submitting a merge request
After you've finished making your changes, there are a few things to check before proposing your changes as a merge request.
First, ensure that all the checks (tests, mypy and code-style) pass successfully. Merge requests that fail any of the checks will not be accepted. For more information, see this section of the development docs: https://docs.tildes.net/development#running-checks-on-your-code
Squash your changes into logical commits. For many changes there should only be a single commit, but some can be broken into multiple logical sections. The commit messages should follow [the formatting described in this article](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html), with the summary line written in the imperative form. The summary line should make sense if you were to use it in a sentence like "If applied, this commit will \_\_\_\_\_\_\_\_". For example, "Add a new X", "Fix a bug with Y", and so on.
### Merge request and code review
Once your code is ready, you can [submit a new merge request on GitLab](https://gitlab.com/tildes/tildes/merge_requests/new).
After creating the merge request, if you need to make any further changes to your code (whether in response to code review or not), please add the changes as new commits. Do not modify the existing commits, this makes it far simpler for other people to follow what changes you're making. Once the merge request is approved, a final pass can be done to squash the commits down, make updates to commit messages, etc. before it's actually merged.
This diff is collapsed.
# Tildes
This is the code behind [Tildes](https://tildes.net), a non-profit community site. The official repository is located on GitLab at https://gitlab.com/tildes/tildes
For general information about Tildes and its goals, please see [the announcement blog post](https://blog.tildes.net/announcing-tildes) and [the Tildes Docs site](https://docs.tildes.net).
## Issue tracker / plans
Known issues and plans for upcoming changes are tracked on GitLab: https://gitlab.com/tildes/tildes/issues
The "board" view is useful as an overview: https://gitlab.com/tildes/tildes/boards
## Contributing to Tildes development
Please see [the Contributing doc](CONTRIBUTING.md) for more detailed information about setting up a development version of Tildes and how to contribute to development.
# -*- mode: ruby -*-
# vi: set ft=ruby :
VAGRANT_CONFIG_VERSION = "2"
Vagrant.configure(VAGRANT_CONFIG_VERSION) do |config|
config.vm.box = "ubuntu/xenial64"
# Main application folder
config.vm.synced_folder "tildes/", "/opt/tildes/"
# Mount the salt file root and pillar root
config.vm.synced_folder "salt/salt/", "/srv/salt/"
config.vm.synced_folder "salt/pillar/", "/srv/pillar/"
config.vm.network "forwarded_port", guest: 443, host: 4443
config.vm.network "forwarded_port", guest: 9090, host: 9090
# Masterless salt provisioning
config.vm.provision :salt do |salt|
salt.masterless = true
salt.minion_config = "salt/minion"
salt.run_highstate = true
salt.verbose = true
salt.log_level = "info"
end
end
#!/bin/sh
#
# Pre-commit hook script that ensures mypy checks and tests pass
vagrant ssh -c ". activate \
&& echo 'Checking mypy type annotations...' && mypy . \
&& echo -n 'Running tests: ' && pytest -q"
#!/bin/sh
#
# Pre-push hook script that ensures mypy checks, style checks, and tests pass
vagrant ssh -c ". activate \
&& echo 'Checking mypy type annotations...' && mypy . \
&& echo -n 'Running tests: ' && pytest -q \
&& echo 'Checking code style (takes a while)...' && pylama"
# look for files on the minion, not the master
file_client: local
# set a specific minion ID for use in top file / pillars
# options:
# - "dev" for the vagrant setup
# - "prod" for (single-server) production
# - "monitoring" for the monitoring server
id: dev
# base path for SSL certificates
ca.cert_base_path: '/etc/pki'
state_verbose: False
# enable new module.run state syntax
# https://docs.saltstack.com/en/develop/topics/releases/2017.7.0.html#state-module-changes
use_superseded:
- module.run
ini_file: development.ini
ssl_cert_path: /etc/pki/tls/certs/localhost.crt
ssl_private_key_path: /etc/pki/tls/certs/localhost.key
nginx_worker_processes: 1
postgresql_version: 10
sentry_secret: 'sentry_secret_token'
sentry_email: '[email protected]'
sentry_password: 'passwordforsentrysuperuser'
sentry_server_name: 'sentry.example.com'
developer_ips: ['127.0.0.1']
prometheus_server_name: 'prom.example.com'
grafana_server_name: 'grafana.example.com'
grafana_admin_password: 'passwordforgrafanaadmin'
ssl_cert_path: /etc/pki/tls/certs/localhost.crt
ssl_private_key_path: /etc/pki/tls/certs/localhost.key
hsts_max_age: 60
nginx_worker_processes: auto
postgresql_version: 9.6
ini_file: production.ini
ssl_cert_path: /etc/letsencrypt/live/tildes.net/fullchain.pem
ssl_private_key_path: /etc/letsencrypt/live/tildes.net/privkey.pem
hsts_max_age: 63072000
nginx_worker_processes: auto
postgresql_version: 10
base:
'dev':
- dev
'prod':
- prod
'monitoring':
- monitoring
- monitoring-secrets
{% from 'common.jinja2' import app_dir, bin_dir -%}
[Unit]
Description=Boussole - auto-compile SCSS files on change
[Service]
WorkingDirectory={{ app_dir }}
Environment="LC_ALL=C.UTF-8" "LANG=C.UTF-8"
ExecStart={{ bin_dir }}/boussole watch --backend=yaml --config=boussole.yaml --poll
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
{% from 'common.jinja2' import app_dir, bin_dir %}
/etc/systemd/system/boussole.service:
file.managed:
- source: salt://boussole.service.jinja2
- template: jinja
- user: root
- group: root
- mode: 644
- require_in:
- service: boussole.service
boussole.service:
service.running:
- enable: True
- require:
- pip: pip-installs
create-css-directory:
file.directory:
- name: {{ app_dir }}/static/css
initial-boussole-run:
cmd.run:
- name: {{ bin_dir }}/boussole compile --backend=yaml --config=boussole.yaml
- cwd: {{ app_dir }}
- env:
- LC_ALL: C.UTF-8
- LANG: C.UTF-8
- require:
- file: create-css-directory
- unless: ls {{ app_dir }}/static/css/*.css
unpack-cmark-gfm:
archive.extracted:
- name: /tmp/cmark-gfm
- source:
- salt://cmark-gfm.tar.gz
- https://github.com/github/cmark/archive/0.28.0.gfm.11.tar.gz
- source_hash: sha256=a95ee221c3f6d718bbb38bede95f05f05e07827f8f3c29ed6cb09ddb7d05c2cd
- if_missing: /usr/local/lib/libcmark-gfm.so
- options: --strip-components=1
- enforce_toplevel: False
install-cmark-build-deps:
pkg.installed:
- name: cmake
install-cmark-gfm:
cmd.run:
- cwd: /tmp/cmark-gfm/
- names:
- make
- make install
- onchanges:
- archive: unpack-cmark-gfm
- require:
- pkg: install-cmark-build-deps
{% set python_version = '3.6.5' %}
{% set app_dir = '/opt/tildes' %}
{% set static_sites_dir = '/opt/tildes-static-sites' %}
{% set venv_dir = '/opt/venvs/tildes' %}
{% set bin_dir = venv_dir + '/bin' %}
{% if grains['id'] == 'dev' %}
{% set app_username = 'vagrant' %}
{% else %}
{% set app_username = 'tildes' %}
{% endif %}
/etc/systemd/system/consumer-topic_metadata_generator.service:
file.managed:
- source: salt://consumers/topic_metadata_generator.service.jinja2
- template: jinja
- user: root
- group: root
- mode: 644
consumer-topic_metadata_generator.service:
service.running:
- enable: True
{% from 'common.jinja2' import app_dir, bin_dir -%}
[Unit]
Description=Topic Metadata Generator (Queue Consumer)
Requires=rabbitmq-server.service
After=rabbitmq-server.service
PartOf=rabbitmq-server.service
[Service]
WorkingDirectory={{ app_dir }}/consumers
Environment="INI_FILE={{ app_dir }}/{{ pillar['ini_file'] }}"
ExecStart={{ bin_dir }}/python topic_metadata_generator.py
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
{% from 'common.jinja2' import app_dir, bin_dir %}
data-cleanup-cronjob:
cron.present:
- name: {{ bin_dir }}/python -c "from scripts.clean_private_data import clean_all_data; clean_all_data('{{ app_dir }}/{{ pillar['ini_file'] }}')"
- hour: 4
- minute: 10
{% from 'common.jinja2' import app_username, bin_dir %}
{% set profile_dir = '/home/' + app_username + '/.ipython/profile_default' %}
ipython-profile:
cmd.run:
- name: {{ bin_dir }}/ipython profile create
- runas: {{ app_username }}
- creates: {{ profile_dir }}
file.managed:
- name: {{ profile_dir }}/ipython_config.py
- contents:
- c.InteractiveShellApp.extensions = ['autoreload']
- c.InteractiveShellApp.exec_lines = ['%autoreload 2']
require:
- pip: pip-installs
automatic-activate:
file.append:
- name: '/home/{{ app_username }}/.bashrc'
- text: 'source activate'
{% from 'common.jinja2' import app_dir, bin_dir %}
initialize-db:
cmd.run:
- name: {{ bin_dir }}/python -c "from scripts.initialize_db import initialize_db; initialize_db('{{ app_dir }}/{{ pillar['ini_file'] }}')"
- cwd: {{ app_dir }}
- onchanges:
- postgres_database: tildes
{% if grains['id'] == 'dev' %}
insert-dev-data:
cmd.run:
- name: {{ bin_dir }}/python -c "from scripts.initialize_db import insert_dev_data; insert_dev_data('{{ app_dir }}/{{ pillar['ini_file'] }}')"
- cwd: {{ app_dir }}
- onchanges:
- cmd: initialize-db
{% endif %}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
{% for ip in pillar['developer_ips'] %}
allow {{ ip }};
{% endfor %}
deny all;
add_header Strict-Transport-Security "max-age={{ pillar['hsts_max_age'] }}; includeSubDomains; preload" always;
server_name {{ pillar['grafana_server_name'] }};
location / {
proxy_set_header Host $host;
proxy_redirect off;
proxy_pass http://localhost:3000;
}
}
[server]
domain = {{ pillar['grafana_server_name'] }}
root_url = https://{{ pillar['grafana_server_name'] }}
[analytics]
reporting_enabled = false
[security]
admin_password = """{{ pillar['grafana_admin_password'] }}"""
disable_gravatar = true
[snapshots]
external_enabled = false
[users]
allow_sign_up = false
allow_org_create = false
grafana:
pkgrepo.managed:
- name: deb https://packagecloud.io/grafana/stable/debian/ jessie main
- dist: jessie
- file: /etc/apt/sources.list.d/grafana.list
- key_url: https://packagecloud.io/gpg.key
- require_in:
- pkg: grafana
pkg.installed:
- name: grafana
- refresh: True
# note: this file must be set up before the server is started for the first
# time, otherwise the admin password will not be set correctly
file.managed:
- name: /etc/grafana/grafana.ini
- source: salt://grafana/grafana.ini.jinja2
- template: jinja
- user: root
- group: grafana
- mode: 640
service.running:
- name: grafana-server
- enable: True
- require:
- pkg: grafana
- file: /etc/grafana/grafana.ini
- watch:
- file: /etc/grafana/grafana.ini
http.query:
- name: http://localhost:3000/api/datasources
- method: POST
- username: admin
- password: {{ pillar['grafana_admin_password'] }}
- data: |
{"name": "Prometheus",
"type": "prometheus",
"url": "http://localhost:9090",
"access": "proxy",
"isDefault": true}
- header_list:
- 'Content-Type: application/json'
- status: 200
- unless:
- curl -f http://admin:{{ pillar['grafana_admin_password'] }}@localhost:3000/api/datasources/name/Prometheus
/etc/nginx/sites-available/grafana.conf:
file.managed:
- source: salt://grafana/grafana.conf.jinja2
- template: jinja
- user: root
- group: root
- mode: 644
- makedirs: True
/etc/nginx/sites-enabled/grafana.conf:
file.symlink:
- target: /etc/nginx/sites-available/grafana.conf
- makedirs: True
- user: root
- group: root
- mode: 644
d /run/gunicorn 0755 gunicorn gunicorn -
{% from 'common.jinja2' import app_dir, bin_dir -%}
[Unit]
Description=gunicorn daemon
Requires=gunicorn.socket
After=network.target