Commit 230ad67f authored by Hans-Christoph Steiner's avatar Hans-Christoph Steiner

Merge branch 'deploy-doc-and-update' into 'master'

Deploy doc and update See merge request !72
parents 9bea6df8 2b937665
Pipeline #8474038 passed with stage
in 6 minutes 23 seconds
......@@ -6,26 +6,37 @@ redirect_from:
---
Information used by update.py to compile the public index comes from two
sources:
1. the APK files in the repo directory, and
2. the metadata files in the metadata directory.
The original metadata files are simple, easy to edit text files, always
named as the application’s package ID with ’.txt’ appended.
Additionally, you can use JSON, XML, or YAML for app metadata, using the
same fields as the original ’.txt’ format.
Note that although the metadata files are designed to be easily read and
writable by humans, they are also processed and written by various
scripts. The original ’.txt’ format can be automatically cleaned up when
necessary. The structure and comments will be preserved correctly,
although the order of fields will be standardised. (In the event that
the original file was in a different order, comments are considered as
being attached to the field following them). In fact, you can
standardise all the ’.txt’ metadata in a single command, without
changing the functional content, by running:
Information used by `fdroid update` to compile the public index comes
from several sources:
* APK, media, etc files in the _repo_ sub-directory
* per-package "metadata" files in the _metadata_ sub-directory
* [localizable texts and graphics](All_About_Descriptions,_Graphics,_and_Screenshots#in-the-apps-build-metadata-in-an-fdroiddata-collection) in the _metadata_ subdirectory
* localizable texts and graphics [embedded in an app's source code](All_About_Descriptions,_Graphics,_and_Screenshots#in-the-apps-source-repository)
These metadata files are simple, easy to edit text files, always named
as the "package name" with file type appended. There are a wide range
of available fields for adding information to describe packages and/or
apps. For all of the fields like `AuthorName` that apply to all
releases of a package/app, the fields use CamelCase starting with an
upper case letter. All other fields use camelCase starting with a
lower case letter, including per-build fields, localized fields, etc.
There are three supported file types for metadata files:
* _.yml_ files in [YAML](http://www.yaml.org/start.html) format, used by f-droid.org
* _.txt_ files in the old, custom, F-Droid text-based format
* _.json_ files in [JSON](http://json.org/) format
Note that although the metadata files are designed to be easily read
and writable by humans, they are also processed and written by various
scripts. They can be automatically cleaned up when necessary. The
structure and comments will be preserved correctly, although the order
of fields will be standardised. (In the event that the original file
was in a different order, comments are considered as being attached to
the field following them). In fact, you can standardise all packages
in a repository using a single command, without changing the
functional content, by running:
```
fdroid rewritemeta
......
---
layout: page
title: Deploying the Website
---
The F-Droid website is built using [Jekyll](https://jekyllrb.com/) and
[gitlab-ci](https://about.gitlab.com/features/gitlab-ci-cd/). The
whole website now works using a standard git
["fork" workflow](https://docs.gitlab.com/ce/workflow/forking_workflow.html)
that is well supported by gitlab, and well known from services like
GitHub. For all of the pages and information about apps and packages
distributed by _f-droid.org_, those pages are generated using our
[jekyll-fdroid](https://gitlab.com/fdroid/jekyll-fdroid) plugin, which
takes the content from the
[_f-droid.org_ index file](https://f-droid.org/repo/index-v1.json).
### Staging on development forks
All development forks of
[fdroid-website](https://gitlab.com/fdroid/fdroid-website)
automatically have a staging server setup and maintained by the
_gitlab-ci_ configuration. This automatically deploys the content of
the fork's _master_ branch to
[GitLab Pages](https://pages.gitlab.io/). For example, _nicoalt_'s
git fork is at <https://gitlab.com/nicoalt/fdroid-website>, and the
_master_ branch from that is automatically deployed to
<https://nicoalt.gitlab.io/fdroid-website>.
##### Deploying merge request branches
The current _gitlab-ci_ setup is optionally capable of pushing the
resulting HTML to a live webserver and linking to it automatically
from the GitLab merge request, deploying to
[surge.sh](https://surge.sh).
Doing so will allow for easier review of your merge requests, but is
not required. If you don't follow the steps below, your merge
requests will still run through the _gitlab-ci_ continuous integration
checks, but the resulting HTML will not be accessible online.
To configure this:
* Install the [surge.sh](https://surge.sh) CLI: `npm install surge`
* Signup for [surge.sh](https://surge.sh): `surge login`
* Get a "token" to allow CI to deploy on your behalf: `surge token`
* [Add two "Secret Variables"](https://docs.gitlab.com/ce/ci/variables/README.html#secret-variables) to your fork of this project:
* Navigate to your projects Settings -> CI/CD Pipelines -> Secret Variables.
* Add the following two variables:
* `SURGE_LOGIN`: The email you used to signup with `surge login`.
* `SURGE_TOKEN`: The value given when you ran `surge token`.
Now your CI jobs should be authorized to deploy to surge.sh, and the
GitLab merge requests screen will automatically link to this
deployment.
### Staging of the official website
Like with forks, the _master_ branch of the main git repo for the
website, <https://gitlab.com/fdroid/fdroid-website>, is automatically
deployed to <https://fdroid.gitlab.io/fdroid-website>. That is the
place to review the current state of the website before tagging a
release.
### Deploying to https://f-droid.org
When an update to the website is tested and ready to go, a release
manager creates a PGP-signed release tag in the main git repo. The
deploy server monitors the main git repo for new tgs. When it sees a
new tag, it first checks the PGP signature on the git tag using a
manually configured GnuPG keyring that contains only the public keys
of the PGP keys that are allowed to tag website releases.
After the git tag is verified, the `f-droid.org` target in
[.gitlab-ci.yml](https://gitlab.com/fdroid/fdroid-website/blob/master/.gitlab-ci.yml)
is run to generate the actual files for the site. Those files are
then copied into place on the _f-droid.org_ servers.
* TODO document release tag naming scheme
### Setting up and running the deploy procedure
The deploy procedure was tested on a machine running Debian/stretch.
It should be triggered whenever the repo index is published, so it can
rebuild with the latest app information. This whole procedure can be
run as root, or just `gitlab-runner`. This final procedure is not
part of a script committed into the website git repo so that the
commands that run outside of docker cannot be modified via git.
1. [setup docker](https://docs.docker.com/engine/installation/linux/debian/)
2. [setup gitlab-ci-multi-runner](https://docs.gitlab.com/runner/install/linux-repository.html)
3. prepare _deploy-whitelist-keyring.gpg_ using GnuPG:
```console
$ gpg --recv-keys 00aa5556
$ gpg --fingerprint 00aa5556 # verify it!
$ gpg --export 00aa5556 > /path/to/deploy-whitelist-keyring.gpg
```
4. get the website source code:
```console
$ git clone https://gitlab.com/fdroid/fdroid-website
```
5. run the generation:
```console
$ cd fdroid-website
$ git fetch --tags
$ sudo gitlab-runner exec docker f-droid.org \
--pre-build-script ./prepare-for-deploy.py \
--docker-volumes "/path/to/deploy-whitelist-keyring.gpg:/root/.gnupg/pubring.gpg:ro" \
--docker-volumes `pwd`/_site:/builds/output --env DEPLOY_DIR=/builds/output
```
6. deploy the site's files to the webserver, for example:
```console
$ rsync -ax --delete --exclude repo/ --exclude archive/ _site/build/ f-droid.org:/var/www/
```
......@@ -80,6 +80,21 @@ using recent versions of the Android SDK helps. One specific case is
starting with Gradle Android Plugin v2.2.2, timestamps in the APK
file's ZIP header are automatically zeroed out.
The Android SDK tools
[were changed](https://issuetracker.google.com/issues/37132313) in
2014 to
[stick two](https://android.googlesource.com/platform/frameworks/base/+/ad2d07d%5E!/)
[data elements](https://android.googlesource.com/platform/frameworks/base/+/5283fab%5E!/)
in _AndroidManifest.xml_ as part of the build process:
`platformBuildVersionName` and `platformBuildVersionCode`.
`platformBuildVersionName` includes the "revision" of the _platforms_
package built against (e.g. _android-23_), however different
"revisions" of the same _platforms_ package cannot be installed in
parallel. Plus the SDK tools do not support specifying the required
revision as part of the build process. This often results in an
otherwise reproducible build where the only difference is the
`platformBuildVersionName` attribute.
### Build Server IDs
......
......@@ -25,7 +25,7 @@ If you see an application missing from the repository (after reading the [inclus
The client application is available in many languages, but if yours is not included, or if it needs updating or improving, please create an account and use the [translation system](https://hosted.weblate.org/projects/f-droid/) to make your changes.
There’s also a [dedicated forum section](https://forum.f-droid.org/c/translation) for translation discussions.
Start with the overview of [Translation and Localization](docs/Translation_and_Localization). There’s also a [dedicated forum section](https://forum.f-droid.org/c/translation) for translation discussions.
### Help with Development
......
......@@ -58,5 +58,6 @@ the main repository.
* [Verification Server](Verification_Server) Setting up a server to verify app builds are correct.
* [Whitelabel Builds](Whitelabel_Builds) custom builds of F-Droid
* [Privileged Extension](https://gitlab.com/fdroid/privileged-extension#f-droid-privileged-extension) How to use F-Droid Privileged Extension as a user and rom developer
* [Deploying the Website](Deploying_the_Website) How [fdroid-website.git](https://gitlab.com/fdroid/fdroid-website) becomes [f-droid.org](https://f-droid.org)
* [Work in Progress](https://f-droid.org/wiki/page/Work_in_Progress) Feel free to add your own topics and link to others.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment