Commit 85909838 authored by Jacob Vosmaer's avatar Jacob Vosmaer

Merge branch 'pg-replica' into 'master'

Simple pg replication setup tasks on Makefile

I've documented on the Makefile some steps I used to make a working postgres replication master/slave on my local machine.

This is something that will help anyone working with GitLab Geo in the future: https://gitlab.com/gitlab-org/gitlab-ee/issues/76

This still not perfect, so I would like some help from anyone who can automate it better then I do.

Current Caveats:
* I got it working correctly only with a clean postgres install

How to use it:
1. `rm -rf postgresql`
2. `make postgresql`
3. `make postgresql-replication/cluster`
4. `foreman start postgresql`
5. `make postgresql-replication/role`
6. `make postgresql-replication/backup`
7. ~~add replication user data to pg_hba on `postgresql/data/pg_hba.conf`~~ (automated)
8. stop foreman and start it normally with `foreman start` and then bootstrap gitlab database as always

pg_hba data:
```
host    replication     gitlab_replication        127.0.0.1/32            trust
host    replication     gitlab_replication        ::1/128                 trust
```

cc @marin @jacobvosmaer

See merge request !96
parents da35f762 dfc0253e
......@@ -8,6 +8,8 @@ gitlab-runner/
Procfile
postgresql/data
postgresql/.s.*
postgresql-replica/data
postgresql-replica/.s.*
redis/*.conf
.bundle/
gitlab-openldap/bin/
......
......@@ -4,6 +4,9 @@ gitlab_runner_repo = https://gitlab.com/gitlab-org/gitlab-ci-runner.git
gitlab_workhorse_repo = https://gitlab.com/gitlab-org/gitlab-workhorse.git
gitlab_development_root = $(shell pwd)
postgres_bin_dir = $(shell pg_config --bindir)
postgres_replication_user = gitlab_replication
postgres_dir = $(realpath ./postgresql)
postgres_replica_dir = $(realpath ./postgresql-replica)
all: gitlab-setup gitlab-shell-setup gitlab-runner-setup gitlab-workhorse-setup support-setup
......@@ -120,6 +123,19 @@ postgresql: postgresql/data/PG_VERSION
postgresql/data/PG_VERSION:
${postgres_bin_dir}/initdb --locale=C -E utf-8 postgresql/data
postgresql-replication/cluster:
${postgres_bin_dir}/initdb --locale=C -E utf-8 postgresql-replica/data
cat support/pg_hba.conf.add >> postgresql/data/pg_hba.conf
postgresql-replication/role:
${postgres_bin_dir}/psql -h ${postgres_dir} -d postgres -c "CREATE ROLE ${postgres_replication_user} WITH REPLICATION LOGIN;"
postgresql-replication/backup:
psql -h ${postgres_dir} -d postgres -c "select pg_start_backup('base backup for streaming rep')"
rsync -cva --inplace --exclude="*pg_xlog*" postgresql/data postgresql-replica
psql -h ${postgres_dir} -d postgres -c "select pg_stop_backup(), current_timestamp"
./support/recovery.conf ${postgres_dir} > postgresql-replica/data/recovery.conf
.bundle:
bundle install --jobs 4
......
redis: redis-server /home/git/redis/redis.conf
postgresql: postgres -D /home/git/postgresql/data -k /home/git/postgresql -h ''
#postgresql-replica: postgres -D /home/git/postgresql-replica/data -k /home/git/postgresql-replica -h ''
#openldap: cd gitlab-openldap && libexec/slapd -F slapd.d -d2 -h "ldap://127.0.0.1:3890"
gitlab-workhorse: /home/git/gitlab-workhorse/gitlab-workhorse -authSocket /home/git/gitlab.socket -listenAddr localhost:3000 -documentRoot /home/git/gitlab/public -developmentMode
......@@ -241,6 +241,9 @@ The `Makefile` will clone the repositories, install the Gem bundles and set up
basic configuration files. Pick one:
```
# Install dependencies
bundle install
# Clone the official repositories of gitlab and gitlab-shell
make
```
......@@ -248,18 +251,21 @@ make
Alternatively, you can clone straight from your forked repositories or GitLab EE.
```
# Install dependencies
bundle install
# Clone your own forked repositories
make gitlab_repo=git@gitlab.com:example/gitlab-ce.git gitlab_shell_repo=git@gitlab.com:example/gitlab-shell.git \
gitlab_ci_repo=git@gitlab.com:example/gitlab-ci.git gitlab_runner_repo=git@gitlab.com:example/gitlab-ci-runner.git
```
If you are going to work on Gitlab Geo, you will need [PostgreSQL replication](#postgresql-replication) setup before the "Post-installation" instructions.
## Post-installation
First install the requirements for development kit, then start Redis,
PostgreSQL and GitLab-Workhorse by running the command below in the
root of the gitlab-development-kit project:
Start required services: Redis, PostgreSQL and GitLab-Workhorse by running
the command below in the root of the gitlab-development-kit project:
bundle install
bundle exec foreman start
Next, keep the above command running and install the required gems, seed the main GitLab database, and setup GitLab from a new terminal session:
......@@ -362,6 +368,35 @@ remove an individual file (e.g. `rm Procfile`) and rebuild it by
running `make`. If you want to rebuild _all_ configuration files
created by the Makefile, run `make clean-config all`.
## PostgreSQL replication
For Gitlab Geo, you will need a master/slave database replication defined.
There are a few extra steps to follow:
You must start with a clean postgres setup, (jump to next if you are installing
everything from scratch):
```
rm -rf postgresql
make postgresql
```
Initialize a slave database and setup replication:
```
# terminal window 1:
make postgresql-replication/cluster
foreman start postgresql
# terminal window 2:
make postgresql-replication/role
make postgresql-replication/backup
# go back to terminal window 1 and stop foreman by hitting "CTRL-C"
```
Follow [Post-installation](#post-installation) instructions.
## OpenLDAP
To run the OpenLDAP installation included in the GitLab development kit do the following:
......
local replication gitlab_replication trust
#!/bin/bash
# this script will output a recovery.conf file using first parameter as host/socket path
socket_path=$1
cat <<EOF
standby_mode = 'on'
primary_conninfo = 'host=${socket_path} port=5432 user=gitlab_replication'
EOF
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment