Commit f0c4d60f authored by sajolida's avatar sajolida

Merge remote-tracking branch 'origin/master'

parents 88c89d6f a40e9ae6
......@@ -4,107 +4,15 @@
# The plan
## Big picture
We decided to implement a two-way strategy for this feature:
* We have dispatcher code, in JavaScript, that DAVE uses to dynamically modify the
hostname, in the download link it gets from the ISO description file
(IDF), so that each user is pointed to random mirror.
- [public read-only mirror of the Git repository](https://git-tails.immerda.ch/mirror-pool-dispatcher/)
- Vanilla JS (no frameworks)
- The code is deployed live: <https://tails.boum.org/lib/js/mirror-dispatcher.js>
- The library code has three consumers it must work with:
* a Firefox extension (DAVE)
* our website, for downloads we offer outside of the Installation
Assistant, that are not supported by DAVE, such as images for
release candidates; and in any case, for browsers that DAVE does
not support
* Tails Upgrader, that runs the library code with Node.js
- Configuration for the JS is loaded from a JSON file hosted on our
website.
See [[the configuration section|HTTP_mirror_pool#configuration]]
for details.
* Keep using DNS to point to 3-5 fast and reliable mirrors. This will be
the fallback for some use cases. So we still need a DNS dynamic update
system; we can simply re-purpose the one we already have
(`dl.amnesia.boum.org`). This fallback DNS pool will be used:
- by people who do not use JS, when downloading from our website
- for the [[wget download option|install/expert/usb]]
<a id="deployment"></a>
## Deployment plan
This only include steps for which ordering is critical. There are a few
other auxiliary tasks, tracked as sub-tasks of [[!tails_ticket 7161]],
that can be dealt with independently.
1. Make (almost) all downloads use the new mirror pool:
* website: [[!tails_ticket 8642]]
* Dave: [[!tails_ticket 11109]]
* ~~Upgrader: [[!tails_ticket 11123]]~~
2. Shrink our DNS pool into a small, fallback one: [[!tails_ticket 11284]]
3. Have the second audit of our mirror pool dispatcher library completed: [[!tails_ticket 8640]]
<a id="configuration"></a>
# Mirror pool configuration
The client dispatcher code fetches the mirror pool configuration file
from our website.
The configuration file is written in JSON (and not YAML), to avoid the
need to use a third-party parser.
The configuration file is stored in a dedicated Git repository
([[!tails_gitweb_repo mirror-pool desc="public read-only mirror"]]),
that is added to our website with the ikiwiki underlay mechanism.
Using a separate Git repository gives us finer-grained access control
possibilities in the long run, e.g. we may want to let people who
don't have commit access to Git maintain the mirrors pool. Note that
we'll need to serve the configuration file from outside our website if
we ever want to do so: an _underlay_ can affect our website in ways
that are probably (almost?) as bad as what one can do with Git access
to the website itself.
The configuration file is essentially a list of mirrors, and for each
of them we need a few values:
* url_prefix: whatever needs to be pre-pended to the path to an ISO
(e.g. `/stable/tails-i386-2.0/tails-i386-2.0.iso`) that is shared
between all mirrors, to form a complete URL to that ISO; for
example:
- in the old mirror pool design, this would be
`http://dl.amnesia.boum.org/tails`
- in the new mirror pool design, this could be e.g.
`http://42.dl.amnesia.boum.org/tails` (for mirrors who want to
use the unique VirtualHost we provide them), or something they
deal with themselves, e.g. `https://mirrors.kernel.org/tails`
* weight: the probability this mirror has to be picked by the
dispatcher code, expressed as a non-negative integer; weight
0 means that the mirror is currently disabled, and will never be
redirected to
* email (optional): the email address of the mirror's operator
* notes (optional): various additional notes that can be useful to
the managers of the mirror pool
For a more formal, and probably more up-to-date definition of the data
model, better see
[its JSON schema](https://git-tails.immerda.ch/mirror-pool/tree/schema.json).
Here is
[an example configuration file](https://git-tails.immerda.ch/mirror-pool/tree/example-mirrors.json).
See the corresponding [[design document|contribute/design/mirrors]].
<a id="speed"></a>
# Speed
This is mainly for [[!tails_ticket 10295]].
This is mainly for [[!tails_ticket 10295]]. We keep this information
around because it provides potentially useful data wrt. the reasons
why we picked some mirrors, but not others, for the fallback DNS pool.
## Fast & reliable enough mirrors
......@@ -131,7 +39,8 @@ practical purposes here.
- from France: avg. 14.5 MB/s, stdev 3.4 MB/s
- from Netherlands: 54.0 MB/s, 52.7 MB/s, 51.7 MB/s
* 195.154.14.189 aka https://16.dl.amnesia.boum.org/tails/ (France):
- being moved to another IP (<https://28.dl.amnesia.boum.org/tails/>)
- being moved to another IP (163.172.39.92 aka https://28.dl.amnesia.boum.org/tails/)
so we're not adding it yet
- from lizard: 5.08 MB/s, 5.25 MB/s, 6.26 MB/s, 6.33 MB/s, 6.17 MB/s
- from D.C.: 4.65 MB/s, 7.21 MB/s, 7.01 MB/s
- from Germany: 22.4 MB/s, 21.6 MB/s, 22.6 MB/s
......@@ -149,8 +58,8 @@ practical purposes here.
- from Germany: 15.5 MB/s, 17.1 MB/s, 16.1 MB/s
- from France: avg. 10.4 MB/s, stdev. 2.7 MB/s
- from Netherlands: 25.2 MB/s, 66.3 MB/s, 43.9 MB/s
- XXX: add 198.145.20.143 and 149.20.37.36, aka mirrors.kernel.org?
- XXX: add 208.80.154.15 aka mirrors.wikimedia.org?
- 198.145.20.143 and 149.20.37.36, aka mirrors.kernel.org
- 208.80.154.15 aka mirrors.wikimedia.org
## Too slow mirrors
......
......@@ -26,9 +26,12 @@ be copy'n'pasted as-is from the proposal sent to the sponsor.
[Last month's activity on Redmine](https://labs.riseup.net/code/projects/tails/issues?query_id=208)
can be helpful.
This report covers the activity of Tails in October 2016.
XXX: Add something about this report being the final report of the contract.
This report covers the activity of Tails in October 2016. It is the
final report for our current contract with OTF. We want to thank OTF
for their support, that enabled us to deliver a large set of
improvements to various people, including: Tails users, Tails
contributors, users of Debian and its numerous derivatives,
Thunderbird users, and more!
Everything in this report is public.
......@@ -36,11 +39,15 @@ Everything in this report is public.
- A.1.1. Secure the Icedove autoconfig wizard
This work has been merged, so this deliverable is now completed.
As [[reported last month|contribute/reports/SponsorS/2015/2016_09]],
this work has been merged, and this deliverable is now completed.
- A.1.2. Make our improvements maintainable for future versions of Icedove
This work has been merged, so this deliverable is now completed.
As [[reported last month|contribute/reports/SponsorS/2015/2016_09]],
this deliverable was completed: our patches were submitted to
Thunderbird upstream, and we keep working with them to make sure that
all users of Thunderbird benefit from a safer configuration wizard.
# B. Improve our quality assurance process
......@@ -48,48 +55,59 @@ This work has been merged, so this deliverable is now completed.
- B.3.11. Fix newly identified issues to make our test suite more robust and faster
After having worked on this continuously for over a year, the test
suite runs faster despite testing more than before, and it reports
much fewer false positives. We now are in a state where we can rely on
its results as intended:
After having worked on this continuously for over a year, our test
suite runs faster despite testing more cases than before; also, it reports
much fewer false positives. We now are in a state where we can rely on
its results as intended:
- When reviewing someone else's work, they allow us to quickly rule
out many classes of regressions.
- When reviewing someone else's work, automated testing results
allows us to quickly rule out many classes of regressions.
- When introducing new features and bug fixes in general, they make us
a lot more comfortable.
That said, we clearly underestimated the amount of work needed for the
robustness part of this deliverable, and there are some outstanding
issues still remaining before we could have no false positives. However,
it should be stressed that the issues are rare enough that the
automated test suite still is as useful as described in the previous
paragraph, so we consider this a success nevertheless.
That said, we clearly underestimated the amount of work needed for the
robustness part of this deliverable, and there are some outstanding
issues still remaining before we could have no false positives _at all_.
However, it should be stressed that the issues are rare enough that the
automated test suite still is as useful as described in the previous
paragraph, so we consider this a success nevertheless.
This deliverable is now completed.
This deliverable is now completed.
- B.3.12. Reliably wait for post-Greeter hooks ([[!tails_ticket 5666]])
This work has been merged, so this deliverable is now completed.
This work has been merged, so this deliverable is now completed.
- B.3.14. Write tests for incremental upgrades ([[!tails_ticket 6309]])
This work has been merged, so this deliverable is now completed.
This work has been merged, so this deliverable is now completed.
# C. Scale our infrastructure
## C.1. Change in depth the infrastructure of our pool of mirrors:
## C.1. Change in depth the infrastructure of our pool of mirrors
We completed the remaining bits of this deliverable, and deployed
everything to production:
- C.1.2. Write & audit the code that makes the redirection decision from our website ([[!tails_ticket 8639]], [[!tails_ticket 8640]], [[!tails_ticket 11109]])
- C.1.2. Write & audit the code that makes the redirection decision from our website
* `mirror-dispatcher.js`: our security auditor did a final
security review, and approved this code.
XXX
* Download And Verification Extension (DAVE) for Firefox: our code
to use our new mirror pool when downloading Tails ISO images was
completed, merged, released and made it into Mozilla's
stable channel.
- C.1.6. Adjust download documentation to point to the mirror pool dispatcher's URL
This deliverable is now completed.
XXX
- C.1.6. Adjust download documentation to point to the mirror pool dispatcher's URL ([[!tails_ticket 8642]], [[!tails_ticket 11329]], [[!tails_ticket 10295]])
- C.1.8. Clean up the remainers of the old mirror pool setup
This was completed back in May.
XXX
- C.1.8. Clean up the remainers of the old mirror pool setup ([[!tails_ticket 8643]], [[!tails_ticket 11284]])
This deliverable was completed as well.
## C.4. Maintain our already existing services
......@@ -157,10 +175,10 @@ of security updates.
[[!tails_ticket 11836 desc="stop stringifying Puppet facts"]] but
had no time to deal with the fallout, so we reverted this change
and documented what the next steps are.
* We had to fix the way our test suite handles the pluging and unpluging of
DVDROM devices, as well as the ejection of the ISO image to adapt to
compatibility issues between QEMU 2.6 and jessie-backport Libvirt version
on this matter. [[!tails_ticket 11874]]
* We [[!tails_ticket 11874 desc="fixed the way"]] our test suite
handles the pluging and unplugging of DVD devices, as well as the
ejection of the ISO image, to adapt to compatibility issues between
QEMU 2.7 and Jessie's libvirt on this matter.
#### Bug fixes
......@@ -180,4 +198,10 @@ of security updates.
[[!tails_ticket 11880 desc="handle our servers' logs"]] and
making plans.
XXX: look at the Git history of our Puppet code
# E. Release management
## E.1.12. Tails 2.5
As reported about a couple months ago, [[Tails 2.5|news/version_2.5]]
was released on August 2. Meanwhile,
[[Tails 2.6|news/version_2.6]] was released on September 20.
[[!meta title="HTTP mirror pool"]]
# Big picture
The Tails downloads are served using two different mirror pools,
depending on the use case:
* We have dispatcher code, in JavaScript, that DAVE uses to dynamically modify the
hostname, in the download link it gets from the ISO description file
(IDF), so that each user is pointed to random mirror.
- [public read-only mirror of the Git repository](https://git-tails.immerda.ch/mirror-pool-dispatcher/)
- Vanilla JS (no frameworks)
- The code is deployed live: <https://tails.boum.org/lib/js/mirror-dispatcher.js>
- The library code has three consumers:
* a Firefox extension (DAVE)
* our website, for downloads we offer outside of the Installation
Assistant, that are not supported by DAVE, such as images for
release candidates; and in any case, for browsers that DAVE does
not support
* Tails Upgrader, that runs the library code with Node.js
- Configuration for the JS is loaded from a JSON file hosted on our
website.
See [[the configuration section|mirrors#configuration]]
for details.
* A DNS Round Robin pool that points to a few (3-7) fast and
reliable mirrors. It is the fallback for some use cases. So we still
have a DNS dynamic update system, re-purposed from the one we already had
(`dl.amnesia.boum.org`). This fallback DNS pool is used:
- by people who do not use JS, when downloading from our website;
- for the [[wget download option|install/expert/usb]].
<a id="configuration"></a>
# Mirror pool configuration
The client dispatcher code fetches the mirror pool configuration file
from our website.
The configuration file is written in JSON, to avoid the
need to use a third-party parser.
The configuration file is stored in a dedicated Git repository
([[!tails_gitweb_repo mirror-pool desc="public read-only mirror"]]),
that is added to our website with the ikiwiki underlay mechanism.
Using a separate Git repository gives us finer-grained access control
possibilities in the long run, e.g. we may want to let people who
don't have commit access to Git maintain the mirrors pool. Note that
we'll need to serve the configuration file from outside our website if
we ever want to do so: an _underlay_ can affect our website in ways
that are probably (almost?) as bad as what one can do with Git access
to the website itself.
The configuration file is essentially a list of mirrors, and for each
of them we need a few values:
* url_prefix: whatever needs to be pre-pended to the path to an ISO
(e.g. `/stable/tails-i386-2.0/tails-i386-2.0.iso`) that is shared
between all mirrors, to form a complete URL to that ISO; for
example:
- in the old mirror pool design, this would be
`http://dl.amnesia.boum.org/tails`
- in the new mirror pool design, this could be e.g.
`http://42.dl.amnesia.boum.org/tails` (for mirrors who want to
use the unique VirtualHost we provide them), or something they
deal with themselves, e.g. `https://mirrors.kernel.org/tails`
* weight: the probability this mirror has to be picked by the
dispatcher code, expressed as a non-negative integer; weight
0 means that the mirror is currently disabled, and will never be
redirected to
* email (optional): the email address of the mirror's operator
* notes (optional): various additional notes that can be useful to
the managers of the mirror pool
For a more formal, and probably more up-to-date definition of the data
model, better see
[its JSON schema](https://git-tails.immerda.ch/mirror-pool/tree/schema.json).
Here is
[an example configuration file](https://git-tails.immerda.ch/mirror-pool/tree/example-mirrors.json).
......@@ -84,7 +84,7 @@ The big picture
------------------
All downloads are currently served from a diverse pool of mirrors
(see the [[blueprint|blueprint/HTTP_mirror_pool]] for details).
(see the [[design document|contribute/design/mirrors]] for details).
Every HTTP mirror makes our files available under a fixed URL
(e.g. `http://42.dl.amnesia.boum.org/tails/` or `https://yourdomain.org/pub/tails/`)
......
......@@ -24,7 +24,7 @@ This deliverable is now completed.
## A.1.2. Make our improvements maintainable for future versions of Icedove
We've resubmitted all of our modifications upstream. ([[!tails_ticket 6165]]).
We've resubmitted all of our modifications upstream. ([[!tails_ticket 6156]]).
We're waiting for those patches to be reviewed by upstream and to be included in Thunderbird.
We're confident that it'll happen soon.
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment