Commit 2dba904c authored by Brian Kocoloski's avatar Brian Kocoloski
Browse files

general updates to documentation

parent f9cfc217
---
date: 2020-05-01
date: 2021-05-01
title: "Merge 1.0"
linkTitle: "Announcing Merge Version 1.0"
description: "The Merge 1.0 testbed is a major step forward in testbed capabilities, stability and workflows."
......@@ -29,13 +29,13 @@ We'll start with a quick overview of what things are new or changed, and then di
- **Scalable Object Storage**: We've been talking about this for a long time, now it's finally here. In v0.9 we stored _everything_ in [Etcd](https://etcd.io/). Etcd really [does not like](https://etcd.io/docs/v3.3.12/dev-guide/limit/#:~:text=etcd%20is%20designed%20to%20handle,any%20request%20is%201.5%20MiB.) storing any values much over 1.5 MiB. When experiments get large and complex, compiled models can easily reach into the 100s of MiB. This has resulted in total lockups of Etcd in production. In v1 we use [MinIO](https://min.io/) for all object types that have the possibility to become large.
![](https://img.shields.io/static/v1?label=Portal&message=Scalability&color=blue) ![](https://img.shields.io/static/v1?label=Facility&message=Scalability&color=blue)
- **Reconciler Architecture**: In v0.9 you could consider the portal design a centralized orchestration system, where the core services were directly controlled by the API and had no real autonomy of their own. In v1 we've adopted the [reconciler architecture](https://queue.acm.org/detail.cfm?id=2898444) where the API simply communicates the desired state on behalf of user requests and the core services autonomously observe desired state, current state and continuously drive desired state to curent state. To quote the above link _"Because all action is based on observation rather than a state diagram, reconciliation loops are robust to failures and perturbations: when a controller fails or restarts it simply picks up where it left off"_. The Cogs already work somewhat in this way in v0.9, in v1 much more so.
- **Reconciler Architecture**: In v0.9 you could consider the portal design a centralized orchestration system, where the core services were directly controlled by the API and had no real autonomy of their own. In v1 we've adopted the [reconciler architecture](https://queue.acm.org/detail.cfm?id=2898444) where the API simply communicates the desired state on behalf of user requests and the core services autonomously observe desired state, current state and continuously drive desired state to current state. To quote the above link _"Because all action is based on observation rather than a state diagram, reconciliation loops are robust to failures and perturbations: when a controller fails or restarts it simply picks up where it left off"_. The Cogs already work somewhat in this way in v0.9, in v1 much more so.
![](https://img.shields.io/static/v1?label=Portal&message=Scalability&color=blue) ![](https://img.shields.io/static/v1?label=Portal&message=Reliability&color=purple)
- **Protocol Buffers base XIR**: In v0.9 XIR was embedded in JSON. This is problematic for 2 primary reasons 1) schema is implicit in code, 2) serialization is space inefficient. The impact of 1 is having no real structure around what defines XIR objects and consistency between subsystems is left up to developers sorting out what data structures should look like by looking at the code that generates them. The impact of 2 is massively inefficient representations for both interchange and storage. Movement to protocol buffers solves both of these problems.
- **Protocol Buffers based XIR**: In v0.9 XIR was embedded in JSON. This is problematic for 2 primary reasons 1) schema is implicit in code, 2) serialization is space inefficient. The impact of 1 is having no real structure around what defines XIR objects and consistency between subsystems is left up to developers sorting out what data structures should look like by looking at the code that generates them. The impact of 2 is massively inefficient representations for both interchange and storage. Movement to protocol buffers solves both of these problems.
![](https://img.shields.io/static/v1?label=Portal&message=Complexity Reduction&color=orange) ![](https://img.shields.io/static/v1?label=Facility&message=Complexity Reduction&color=orange) ![](https://img.shields.io/static/v1?label=Portal&message=Scalability&color=blue) ![](https://img.shields.io/static/v1?label=Site&message=Scalability&color=blue)
- **Virtualization Support**: We now support virtualized nodes, in expression through the Python MX library, interchange through XIR and implementation through Cogs QEMU/KVM integration.
- **Virtualization Support**: We now support virtualized nodes, in expression through the Python MX library, interchange through XIR and implementation through the QEMU/KVM based Mariner hypervisor.
![](https://img.shields.io/static/v1?label=Experimentation&message=New Capability&color=brightgreen) ![](https://img.shields.io/static/v1?label=Experimentation&message=Scalability&color=blue)
- **Experiment Mass Storage**: Experimenters can now define storage volumes either statically, or ephemerally in experiments and attach them to devices. Mass storage is available either as files systems or block devices. This system created by @lincoln is underpinned by [Ceph](https://ceph.io/ceph-storage/) and goes by the development name [Rally](https://gitlab.com/mergetb/tech/rally).
......
......@@ -3,7 +3,7 @@ title: "Experimentation"
linkTitle: "Experimentation"
weight: 2
description: >
Overview of the MergeTB system
Building experiments on a MergeTB testbed
---
......@@ -24,17 +24,16 @@ networks (LAN) interconnected by a router.
```python
import mergexp as mx
from mergexp.net import capacity
# create a topology and call it dumbbell
net = mx.Topology('dumbbell')
# create a network topology and call it dumbbell
net = mx.Network('dumbbell')
# create devices for two lans 'a' and 'b'
a = [net.device(name) for name in ['a0', 'a1', 'a2']]
b = [net.device(name) for name in ['b0', 'b1']]
# create network devices (nodes) for two lans 'a' and 'b'
a = [net.node(name) for name in ['a0', 'a1', 'a2']]
b = [net.node(name) for name in ['b0', 'b1']]
# create a router
r = net.device('r')
r = net.node('r')
# create the 'a' and 'b' LANs with the router as a common node
net.connect(a + [r])
......@@ -43,12 +42,11 @@ net.connect(b + [r])
mx.experiment(net)
```
This code is relatively straightforward. However, it's not immediately apparent
that the topology we have coded up represents the connectivity model we have in
mind. To this end, the Merge portal provides [visualization
capabilities](/docs/web/#materialization-walk-through). The
portal web interface can experiment topologies with the click of a button. The
code above results in the following visualization.
This code is relatively straightforward. However, it's not immediately apparent that the topology we
have coded up represents the connectivity model we have in mind. To this end, the Merge portal
provides <!--[visualization capabilities](/docs/web/#materialization-walk-through) --> visualization
capabilities. The portal web interface can render experiment topologies with the click of a button.
The code above results in the following visualization.
![](/img/concepts/dumbbell.png)
......@@ -149,6 +147,10 @@ topology. When we realize this experiment, this constraint will cause Merge to
launch something called a reticulator to calculate a complete set of
routes for our experiment and generate an augmented model with routes included.
<!--
Storage not yet implemented
## Experiment storage
We can also define filesystems where we can store experiment data and artifacts.
......@@ -189,7 +191,7 @@ In this example we've created a 3 node experiment, with each node mounting
the storage at /mnt/staticAsset. Each node will have access to the network
storage, creating an ideal situtation for sharing large amounts of data between
nodes.
-->
## Realization
......
......@@ -3,7 +3,7 @@ title: "Overview"
linkTitle: "Overview"
weight: 1
description: >
Overview of the MergeTB system
Overview of the MergeTB platform
---
......
......@@ -3,10 +3,9 @@ title: "Resources"
linkTitle: "Resources"
weight: 3
description: >
Overview of the MergeTB system
Modeling a collection of resources as a MergeTB facility
---
_This page covers the basic concepts involved in creating a Merge site, full
documentation is available in the [testbed operations docs](/tb/makeup/)._
......@@ -18,8 +17,8 @@ disjoint experiment and infrastructure networks.
![](/img/concepts/tb.png)
The Golang implementation Merge XIR SDK contains a layered set of libraries for
describing testbed sites. From the bottom up, these layers are
The Golang implementation of the Merge XIR SDK contains a layered set of libraries for describing
testbed sites. From the bottom up, these layers are
| Library | Description |
|---|---|
......
......@@ -3,6 +3,6 @@ title: "Experimentation"
linkTitle: "Experimentation"
weight: 2
description: >
Learn how to create experiments on Merge.
Learn how to create experiments on Merge
---
......@@ -3,7 +3,7 @@ title: "Command Line Interface Reference"
linkTitle: "CLI Reference"
weight: 3
description: >
Using the Merge command line interface.
Using the Merge command line interface
---
{{% alert title="Tips" color="primary" %}}
......
......@@ -3,38 +3,66 @@ title: "Getting Started"
linkTitle: "Getting Started"
weight: 1
description: >
Getting setup to use Merge.
Getting setup to use Merge
---
This guide covers the basic things you need to do to get started with Merge.
## Web Interface
{{% pageinfo %}}
This is a placeholder page for web interface docs
{{% /pageinfo %}}
{{% alert title="Attention" color="warning" %}}
This guide assumes you are using the reference portal at `redstartb.net`. If you are using a
different portal, substitute addresses in this document that reference `redstartb.net` accordingly.
Consult your project leader if you are not sure of the portal address.
{{% /alert %}}
## Account Setup
{{% pageinfo %}}
This is a placeholder page for web-interface account setup
{{% /pageinfo %}}
The first step to experimenting on a Merge testbed is to create an account on the Merge portal.
Accounts can be created either using the graphical web interface online or through the Merge command
line interface (CLI).
### Account Creation through the GUI
Navigate to the Merge GUI at <a
href="https://launch.redstartb.net">https://launch.redstartb.net</a>, click "Register new account",
then fill in the form and click "Sign up". You should initially see a page like the one below:
![](/img/experimentation/register-launch.png)
## Command Line Interface
### Account Creation through the CLI
In addition to the Web interface, Merge has a fully functional command line
interface that has feature parity with the web interface.
First download the Merge CLI `mrg` for the operating system and CPU architecture of your machine:
### Account Setup
| Operating System | Architecture | Download Link |
| ---------------- | ------------ | ------------- |
| Linux | x86_64 | <a href="https://gitlab.com/mergetb/portal/cli/-/jobs/artifacts/v1.0.2/raw/build/x86_64/linux/mrg?job=build">mrg</a> |
| Linux | arm64 | <a href="https://gitlab.com/mergetb/portal/cli/-/jobs/artifacts/v1.0.2/raw/build/arm64/linux/mrg?job=build">mrg</a> |
| Mac OS | x86_64 | <a href="https://gitlab.com/mergetb/portal/cli/-/jobs/artifacts/v1.0.2/raw/build/x86_64/darwin/mrg?job=build">mrg</a> |
| Mac OS | arm64 | <a href="https://gitlab.com/mergetb/portal/cli/-/jobs/artifacts/v1.0.2/raw/build/arm64/darwin/mrg?job=build">mrg</a> |
To begin using Merge you'll need an account. This in this example the username
being registered is `murph` with an associated email `murphy@mergetb.org` and a
password `muffins1701`.
### Configuring the API endpoint
The `mrg` CLI needs to be configured to point to the GRPC API endpoint of the Merge portal. The
address can be constructed by prepending "grpc." to the portal address; e.g.,:
```shell
mrg register murph murphy@mergetb.org muffins1701
mrg config set server grpc.redstartb.net
```
Once you have done this a notification will be sent to the Merge portal
administrators to approve your account. Once your account has been approved, you
will be able to start using Merge.
As noted above, consult your project leader for the appropriate portal address.
## Register account
Then, register your account using the `mrg register` command. For example, to create a user named
"murphy" with email "murphy@mergetb.org" and password "muffins1701":
```shell
mrg register murphy murphy@mergetb.org muffins1701
```
## Account Approval
Once you have created an account using either the web interface or the CLI, you should send a
notification to the Merge portal administrators to have your account approved. Speak with your
project leader for up-to-date information on how best to contact them.
Once your account has been approved, you are ready to start using Merge!
......@@ -3,16 +3,27 @@ title: "Hello World"
linkTitle: "Hello World"
weight: 2
description: >
A hello world experiment in Merge.
A hello world experiment in Merge
---
Welcome to your first guided tour of networked system experimentation in Merge.
Start by [installing the experimentation tools](/docs/experimentation/tools) on your
local system. Now let's dive in.
{{% alert title="Attention" color="warning" %}}
Welcome to your guided tour of networked system experimentation in Merge. Start by following the
["Getting Started"](/docs/experimentation/getting-started) instructions to create your account and
have it approved, download the `mrg` CLI to your local system, and configure the CLI based on the
address of the portal.
This guide assumes you are using the reference portal at `redstartb.net`. If you are using a
different portal, substitute addresses in this document that reference `redstartb.net` accordingly.
Consult your project leader if you are not sure of the portal address.
Now let's dive in.
{{% /alert %}}
{{% alert title="Tip" color="info" %}}
This guide assumes the username `murphy`. Adjust for your username accordingly.
{{% /alert %}}
This guide assumes the username `murphy`, adjust for your username accordingly.
We also assume that you are using the reference portal at `mergetb.net`. If you
are using a different portal, substitute portal address references accordingly.
## Create an Experiment
......@@ -28,8 +39,11 @@ that is automatically created when they first join a Merge portal. The personal
project has the same name as the user. Here we are using `murphy`'s personal
project for the home of our `hello` experiment.
Dot delineated hierarchical naming is pervasive in the `mrg` command line tool,
all objects are referenced using this form.
{{% alert title="Tip" color="info" %}}
Dot delineated hierarchical naming is pervasive in the `mrg` command line tool. All objects are
referenced using this form.
{{% /alert %}}
## Assessing Experiment Status
......@@ -45,13 +59,6 @@ Name.Project Description Mode
hello.murphy My first experiment Public
```
You can use prefixes for any `mrg` command to avoid excessive typing. For
example, the last command can be executed as
```shell
mrg li e
```
More detailed information about the experiment is available through the `show`
command.
......@@ -59,7 +66,7 @@ command.
mrg show experiment hello.murphy
```
```
Repo: https://git.mergetb.net/murphy/hello
Repo: https://git.redstartb.net/murphy/hello
Mode: Public
Description: My first experiment
Realizations:
......@@ -71,18 +78,27 @@ In addition to the information we've already seen, this display shows us two new
things.
1. An address for a Merge-hosted Git repository for our experiment
`https://git.mergetb.net/murphy/hello`
`https://git.redstartb.net/murphy/hello`
2. The [realizations](#realizing-an-experiment) associated with this experiment.
{{% alert title="Tip" color="info" %}}
You can use prefixes for any `mrg` command to avoid excessive typing. For
example, the following two commands are equivalent:
```shell
mrg list experiments
mrg li e
```
{{% /alert %}}
## Pushing Experiment Source
Without any source, an experiment is just an empty shell. We add source by
pushing to the Git repository associated with our experiment. We identified this
repository in the previous step as `https://git.mergetb.net/murphy/hello`.
repository in the previous step as `https://git.redstartb.net/murphy/hello`.
There are two ways to access an experiment's git repository: using standard git
tools or the `mrg` command line tool (which supports a `push` command that simplifies
the process of adding a model revision to an experiment).
tools or the `mrg` CLI
### Adding model revisions to an experiment via git
......@@ -118,8 +134,8 @@ link[b].socket.addrs = ip4('10.0.0.2/24')
experiment(net)
```
More details on this experiment can be found in the _Topology Modeling_ section
[here](/docs/hello-world).
{{% alert title="Tip" color="info" %}} More details on topology modeling can be found in the <a
href="/docs/experimentation/model-ref/">"Experiment Model Reference"</a> section. {{% /alert %}}
**The important thing to note about source repositories is that there is a
requirement about where the experiment topology file lives:**
......@@ -130,7 +146,7 @@ requirement about where the experiment topology file lives:**
Now that we have the source in a local Git repository, let's push it to Merge.
Start by adding a new remote to the repository.
```
git remote add mergetb https://git.mergetb.net/murphy/hello
git remote add mergetb https://git.redstartb.net/murphy/hello
```
Before continuing we need to make sure we are ready to enter authentication
......@@ -211,14 +227,14 @@ Date: Mon Feb 14 17:39:17 2022 +0000
{{% alert title="Note" color="warning" %}}
The `mrg push` command only currently supports tagging on the master
branch. And does not support pushing to a non-master branch.
branch. It does not support pushing to a non-master branch.
{{% /alert %}}
## Confirming Realization Model Compilation
For a realization to work, the model you pushed must compile correctly. The compilation
status is displayed via the optional arugment `--with-status` given to the
status is displayed via the optional argument `--with-status` given to the
`mrg show experiment` command.
This command shows the compilation status of all model revisions pushed to the
......@@ -231,7 +247,7 @@ mrg show experiment hello.murphy --with-status
```
```
Repo: https://git.mergetb.example.net/murphy/hello
Repo: https://git.redstartb.net/murphy/hello
Mode: Public
Description:
Revisions:
......
......@@ -99,7 +99,7 @@ the constraint name, an operator (<=, ==, >=, etc), and a value.
| metal | bool | | node should be not be virtualized (True) (default: False) |
| proc.cores | int | | has the specified number of processor cores |
| memory.capacity | int | bytes | require the memory size |
| disk.capacity | int | bytes | require the specified disk size |
<!--| disk.capacity | int | bytes | require the specified disk size |-->
{{% alert title="Example" color="info" %}}
```python
......@@ -148,7 +148,7 @@ The following constraints are defined for `Link` objects:
| layer| int | number | 1-3 | network layer |
| capacity | int | bits/sec | >0 | set the max bandwidth |
| latency | int | ns | >=0 | set the one-way packet delay |
| loss | float | percentange | 0.0-1.0 | set the packet loss rate |
| loss | float | percentage | 0.0-1.0 | set the packet loss rate |
{{% alert title="Tip" color="warning" %}}
Avoid setting constraints on links which don't require emulation, because it adds overhead to running the experiment.
......
---
title: "Experimentation Tools"
linkTitle: "Experimentation Tools"
weight: 99
description: >
Get the Merge experimentation tools.
---
{{% pageinfo %}}
This is a placeholder page for Merge experimentation tool installation and basic
usage.
{{% /pageinfo %}}
---
title: "Experiment Development Containers (XDCs)"
linkTitle: "Experiment Development Containers"
weight: 6
description: >
Using XDCs to access nodes and automate experiments
---
## Overview
Users interact with materialized experiments through experiment development containers (XDC).
Users can create XDCs and attach them to materialized experiments through either the `mrg` CLI or
the web GUI, and can connect to XDCs from their workstations using either a Jupyter web interface or
through a standard SSH command line client. Users commonly use XDCs as a single, centralized point
of command and control for experiment automation and orchestration.
This walkthrough will show the process of creating an XDC, attaching it to a materialization, and
connecting to it through a standard SSH configuration. We will then show some useful commands for
automating experiments using standard tools running in your XDC
## XDC basics
At a high level an XDC provides a typical Linux environment through which the user can connect to
and thus control their materialization nodes over a secure network. At a lower level, an XDC is a
Linux container which runs on the Merge portal, and which connects to nodes in one or more Merge
facilities through a wireguard VPN tunnel.
The following summarizes the basic characteristics of the Merge XDC:
- Any user can create an XDC either via the web GUI or `mrg` CLI.
- Each XDC is associated with a single _project_. That project may be a user's _personal
project_, or it may be a project with multiple additional members.
- Each XDC runs in an environment that is accessible to each member of the project. Each project member
has access to the XDC, even if they did not create the XDC themselves.
- Each XDC runs an Ubuntu 20.04 container with standard Ubuntu tools and binaries.
- XDCs are accessible both online via a Jupyter web interface and via SSH.
### Creating an XDC via the CLI
First, create an XDC with `mrg`
```shell
mrg new xdc x0.murphy
```
This creates a new XDC `x0` associated with the personal project of the user `murphy`.
### Attaching an XDC to a materialization
Assume the user has a materialization named `world.hello.murphy`. We now attach the XDC
to this materialization
```shell
mrg xdc attach x0.murphy world.hello.murphy
```
{{% alert title="Tip" color="info" %}}
Remember that XDCs are created in the context of a _specific project_ (which could be a user's
personal project). An XDC created in the namespace of one project **cannot be attached to
a materialization in a different project**, even if the user is a member of both projects.
{{% /alert %}}
### Reaching an XDC via SSH
{{% pageinfo %}}
This is a placeholder until `mrg` has automation in place to do this
{{% /pageinfo %}}
### Detaching an XDC from a materialization
Assume now the user wants to attach `x0` to a new materialization names `planet.hello.murphy`. We must
detach from the first materialization before attaching to the second:
```shell
mrg xdc detach x0.murphy
mrg xdc attach x0.murphy planet.hello.murphy
```
{{% alert title="Tip" color="info" %}}
An XDC can only be attached to one materialization at a time
{{% /alert %}}
## Experiment Automation
......@@ -8,9 +8,9 @@ menu:
weight: 20
---
Welcome to the Merge documentation. To get a basic understanding of what Merge
is, head to [concepts](/docs/concepts). To learn more about using Merge for
creating networked-system experiments, head to
[experimentation](/docs/experimentation). For information of operating a Merge
portal, head to [portal operations](/docs/portalops). To understand more about
testbed facility operations, head to [facility operations](/docs/facilityops).
Welcome to the Merge documentation. To get a basic understanding of what Merge is, head to
[concepts](/docs/concepts). To learn more about using Merge for creating networked-system
experiments, head to [experimentation](/docs/experimentation). For information of operating a Merge
portal, head to [portal operations](/docs/portalops). To understand more about testbed facility
operations, head to [facility operations](/docs/facilityops). To get acclimated with development in
Merge, head to [development](/docs/development).
static/img/concepts/dumbbell.png

12.7 KB | W: | H:

static/img/concepts/dumbbell.png

9.98 KB | W: | H:

static/img/concepts/dumbbell.png
static/img/concepts/dumbbell.png
static/img/concepts/dumbbell.png
static/img/concepts/dumbbell.png
  • 2-up
  • Swipe
  • Onion skin
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment