Commit 826217d1 authored by Stefan Lasiewski's avatar Stefan Lasiewski

Merge branch 'master' of gitlab.com:NERSC/nersc.gitlab.io into stefanl/spindocs

parents 6758eb4d e21e2f4d
......@@ -14,3 +14,5 @@ src
\#*\#
*.DS_Store
*__pycache__*
node_modules
package-lock.json
\ No newline at end of file
image: python:3.7
stages:
- lint
- filesize
- build
- test
- deploy
- post-deploy
build:
markdown_lint:
image: node
stage: lint
script:
- npm install -g markdownlint-cli
- bash util/lint-changes.sh
allow_failure: true
filesize:
image: ubuntu:latest
stage: filesize
before_script:
- apt-get update && apt-get install -y git
script:
- bash util/filesize.sh
allow_failure: true
mkdocs_build:
stage: build
script:
- pwd
- find .
- mkdir -p .build_cache
- pip install --cache-dir=.build_cache -r requirements.txt
- python --version
- mkdocs --version
- pip install --upgrade --cache-dir=.build_cache -r requirements.txt
- mkdocs --verbose build
artifacts:
paths:
- public
expire_in: 1 week
cache:
key: ${CI_JOB_NAME}
paths:
- .build_cache
key: "build-$CI_COMMIT_REF_SLUG"
check_links:
stage: test
......@@ -33,27 +48,20 @@ check_links:
- python util/scrape_urls.py public
allow_failure: true
cache:
key: ${CI_JOB_NAME}
paths:
- .links_cache
key: "links-$CI_COMMIT_REF_SLUG"
markdown_lint:
image: node
stage: test
script:
- npm install -g markdownlint-cli
- markdownlint docs
allow_failure: true
- good-urls-cache.txt
pages:
stage: deploy
before_script:
- pip install htmlmin
script:
- ls -l public
- gzip -k -6 -r public/assets/stylesheets
- gzip -k -6 -r public/assets/javascripts
- find public -type f -name "*.html"
- find public -type f -name "*.html" -exec htmlmin {} {} \;
- find public -type f -name "*.html" -exec gzip --keep --verbose {} \;
- gzip -k -6 -r public/assets/stylesheets
- gzip -k -6 -r public/assets/javascripts
- find public -type f -name "*.html" -exec htmlmin {} {} \;
- find public -type f -name "*.html" -exec gzip --keep --verbose {} \;
artifacts:
paths:
- public
......
{
"default": false,
"MD003": { "style": "atx" },
"MD009": true
"first-heading-h1": true,
"heading-style": { "style": "atx" },
"no-trailing-spaces": true,
"line-length": { "line_length": 130, "code_blocks": false, "tables": false},
"no-missing-space-atx": true,
"no-multiple-space-atx": true
}
......@@ -5,22 +5,34 @@ which should be observed when adding to this repositiory.
## About
This repository contains NERSC technical documentation written in
Markdown which is converted to html/css/js with the
[mkdocs](http://www.mkdocs.org) static site generator. The
[theme](https://gitlab.com/NERSC/mkdocs-material) is a fork of
[mkdocs-material](https://github.com/squidfunk/mkdocs-material) with
This repository contains NERSC technical documentation written in
Markdown which is converted to html/css/js with the
[mkdocs](http://www.mkdocs.org) static site generator. The
[theme](https://gitlab.com/NERSC/mkdocs-material) is a fork of
[mkdocs-material](https://github.com/squidfunk/mkdocs-material) with
NERSC customizations such as the colors.
## Rules
1. Follow this [Markdown styleguide](https://github.com/google/styleguide/blob/3591b2e540cbcb07423e02d20eee482165776603/docguide/style.md).
1. Do not commit large files
1. Do not commit large files
(e.g. very high-res images, binary data, executables)
* [Image optimization](https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/image-optimization)
1. Do not commit directly to master branch
1. No commits directly to the master branch
## Setup
### Merge Requests
NERSC staff who have submited a succesful Merge Request are added to
the list of the approvers.
Self-approval of Merge Requests is allowed if the change is *minor*.
That is typos, spelling corrections, etc.
When there is a change in the _meaning_ of the content then one should
wait for a peer review. Consider "assigning" the Merge Request to a
relevant approver.
## Setup
### Prerequisites
......@@ -42,20 +54,20 @@ pip install -r requirements.txt
### Edit with live preview
Open a terminal session with the appropriate conda environment
activated, navigate to the root directory of the repository (where
Open a terminal session with the appropriate conda environment
activated, navigate to the root directory of the repository (where
`mkdocs.yml` is located) and run the command `mkdocs serve`. This will
start a live-reloading local web server. You can then open
[http://127.0.0.1:8000](http://127.0.0.1:8000) in a web browser to
start a live-reloading local web server. You can then open
[http://127.0.0.1:8000](http://127.0.0.1:8000) in a web browser to
see your local copy of the site.
In another terminal (or with a GUI editor) edit existing files, add
new pages to `mkdocs.yml`, etc. As you save your changes the local
In another terminal (or with a GUI editor) edit existing files, add
new pages to `mkdocs.yml`, etc. As you save your changes the local
web serve will automatically rerender and reload the site.
### Output a static site
To build a self-contained directory containing the full html/css/js
To build a self-contained directory containing the full html/css/js
of the site:
```
......@@ -98,16 +110,21 @@ Work with a branch of the main repo.
#### Option 2
Make a fork of the repository and do all of your work on the fork.
Make a fork of the repository and do all of your work on the fork.
Submit a merge request through gitlab when you have made your changes.
#### Option 3
For some changes you do not need the full environment. It is possible
to edit Markdown files directly on gitlab. This work should be in a
to edit Markdown files directly on gitlab. This work should be in a
private fork or branch and submitted as a merge request. A new branch
can be created by clicking the "+" button next to the repository name.
### Add a new page
For a newly added page to appear in the navigation edit the top-level
`mkdocs.yml` file.
### Review a Merge Request from a private fork
1. Modify `.git/config` so merge requests are visible
......@@ -120,14 +137,14 @@ can be created by clicking the "+" button next to the repository name.
fetch = +refs/merge-requests/*/head:refs/remotes/origin/pr/*
...
```
2. Check for any Merge Requests
```shell
git fetch
```
3. Checkout a specific Merge Request for review (merge request `N`
3. Checkout a specific Merge Request for review (merge request `N`
in this example)
```shell
......@@ -138,7 +155,7 @@ can be created by clicking the "+" button next to the repository name.
## Command prompts
1. when showing a command and sample result, include a prompt
1. when showing a command and sample result, include a prompt
indicating where the command is run, eg for a command valid on any
NERSC system, use `nersc$`:
......@@ -154,7 +171,7 @@ can be created by clicking the "+" button next to the repository name.
Submitted batch job 864933
```
2. Where possible, replace the username with `elvis`
2. Where possible, replace the username with `elvis`
(i.e. a clearly-arbitrary-but-fake user name)
3. If pasting a snippet of a long output, indicate cuts with `[snip]`:
......@@ -168,9 +185,14 @@ can be created by clicking the "+" button next to the repository name.
-rw-rw---- 1 elvis elvis 407 Jan 9 15:35 index.md
[snip]
```
## Writing Style
When adding a page think about your audience.
* Are they new or advanced expert users?
* What is the goal of this content?
* [Grammatical Person](https://en.wikiversity.org/wiki/Technical_writing_style#Grammatical_person)
* [Active Voice](https://en.wikiversity.org/wiki/Technical_writing_style#Use_active_voice)
......@@ -185,5 +207,17 @@ can be created by clicking the "+" button next to the repository name.
## Slurm options
* Show both long and short option when introducing an option in text
* Use the short version (where possible) in scripts
* Use the long version (where possible) in scripts
## Markdown lint
Install the markdown linter (requires node/npm) locally
```shell
npm install markdownlint-cli
```
Run the linter from the base directory of the repository
```shell
./node_modules/markdownlint-cli/markdownlint.js docs
```
# Collaboration Accounts
## Overview
Most NERSC login accounts are associated with specific individuals and
must not be shared. Sometimes it is advantageous to have a login
account which is not tied to a person but instead to the group for the
purposes of shared access to batch jobs or data. Collaboration
Accounts are designed to facilitate collaborative computing by
allowing multiple users to use the same account. All actions
performed by the collaboration account are traceable back to the
individual who used the collaboration account to perform those actions
via gsisshd accounting logs. PIs and PI Proxies can request a
collaboration account by logging into nim and selecting "Request a
Collaboration Account" under the blue "Actions" tab.
## Logging Into Collaboration Accounts
To access your collaboration account on Cori or Edison, login to any
interactive node:
```
collabsu <collaboration account name>
<enter nersc password at the prompt>
```
On the DTN nodes, you must still use the old method using your NERSC
grid certificate. You can also do this on from any PDSF interactive
node, edisongrid, or corigrid (to access corigrid, you must first log
into cori.nersc.gov) as yourself and then do the following commands:
```
module load globus
myproxy-logon -s nerscca.nersc.gov
gsissh localhost -l <project account name>
```
If you're setting the port to 22 in your .ssh/config file, you may also
need to add a "-p 2222" flag to the gsissh command.
Alternatively you can get a proxy using grid-proxy-init instead of
myproxy-logon. In this case you use your GRID pass phrase instead of
your NIM password.
## Controlling Collaboration Account Access
PIs and PI Proxies can give users in their repo access to the
collaboration account by adding them in NIM to the corresponding
group. Each collaboration account has a linux file group associated
with it with the name c_<collaboration account\>. You can add users to
the corresponding group following the instructions
[here](https://www.nersc.gov/users/accounts/nim/nim-guide-for-pis/#toc-anchor-8).
To use the old gsissh method, you must associate their grid
certificate with the account. Log into nim.nersc.gov and click on the
repo with the collaboration account you wish to change. Then click on
the "Project Access" tab in the yellow bar partway down the page. This
will bring you to a page with a pull down menu where you can select
project account and the user you wish to give access to it. If you
encounter any errors with this, please open a ticket with NERSC
consulting.
## Use Cases
### Collaborative Data Management
Large scale experimental and simulation data are typically read or
written by multiple collaborators and are kept on disk for long
periods. A problem that often arises is that the files are owned by
the collaborator who did the work and if that collaborator changes
roles the default unix file permissions usually are such that the
files cannot be managed (deleted) by other members of the
collaboration and system administrators must be contacted. While the
problem can be addressed with the appropriate use of unix groups and
file permissions in practice this tends to be problematic and a more
seamless solution would be of great utility.
### Collaborative Software Management
The issue with managing software is similar to that of managing data –
different collaborators often need to work with the same files in a
particular software installation and unix groups and file permissions
tend to be problematic for them. The main difference between
collaborative data and software management is that software is
typically managed on a short-tem basis (hours/days) and smaller in
size (~GBs) whereas production data is managed on a long-term basis
(months/years) and much larger (~TBs to ~PBs).
### Collaborative Job Management
Production level jobs are often run by a small team of collaborators.
Project accounts would enable members of the team to manipulate jobs
submitted by other team members as necessary.
# Accounts
## Obtaining an account
In order to use the NERSC facilities you need:
......
......@@ -76,7 +76,7 @@ Department of Energy guidelines regarding passwords. The following
requirements conform to the Department of Energy guidelines regarding
passwords, namely DOE Order 205.3 and to Lawrence Berkeley National
Laboratory's
[RPM §9.02 Operational Procedures for Computing and Communications](http://www.lbl.gov/Workplace/RPM/R9.02.html).
[RPM §9.02 Operational Procedures for Computing and Communications](https://www.lbl.gov/Workplace/RPM/R9.02.html).
When users are selecting their own passwords for use at NERSC, the
following requirements must be used.
......
......@@ -20,7 +20,7 @@ username, then "sharing" has occurred and all usernames associated
with the owner of the shared username will be disabled.
If a user is disabled due to account sharing, the PI or an authorized
project manager must send a memo to consult@nersc.gov explaining why
project manager must send a memo to accounts@nersc.gov explaining why
the sharing occurred and assuring that it will not occur again. NERSC
will then determine if the user should be re-enabled.
......
# PyTorch
[PyTorch](https://pytorch.org/) is a high-productivity Deep Learning framework based on dynamic
computation graphs and automatic differentiation. It is designed to be as close
to native Python as possible for maximum flexibility and expressivity.
[PyTorch](https://pytorch.org/) is a high-productivity Deep Learning framework
based on dynamic computation graphs and automatic differentiation.
It is designed to be as close to native Python as possible for maximum
flexibility and expressivity.
## Availability on Cori
......@@ -16,6 +17,28 @@ PyTorch makes it fairly easy to get up and running with multi-node training
via its included _distributed_ package. Refer to the distributed tutorial for
details: https://pytorch.org/tutorials/intermediate/dist_tuto.html
Note the above tutorial doesn't actually document our currently recommended
approach of using DistributedDataParallelCPU. See examples below.
## Examples
Coming soon.
We're putting together a coherent set of example problems, datasets, models,
and training code in this repository:
https://github.com/NERSC/pytorch-examples
This repository can serve as a template for your research projects with a
flexibly organized design for layout and code structure. The `template` branch
contains the core layout without all of the examples so you can build your
code on top of that minimal, fully functional setup. The code provided should
minimize your own boiler plate and let you get up and running in a distributed
fashion on Cori as quickly and seamlessly as possible.
The examples include:
* A simple hello-world example
* HEP-CNN classifier
* ResNet50 CIFAR10 image classification
* HEP-GAN for generation of RPV SUSY images.
The repository will also be used to benchmark our system for single and
multi-node training.
# BerkeleyGW
The BerkeleyGW Package is a set of computer codes that calculates the
quasiparticle properties and the optical responses of a large variety
of materials from bulk periodic crystals to nanostructures such as
slabs, wires and molecules. The package takes as input the mean-field
results from various electronic structure codes such as the Kohn-Sham
DFT eigenvalues and eigenvectors computed with Quantum ESPRESSO,
PARATEC, PARSEC, Octopus, Abinit, Siesta etc.
NERSC provides modules for [BerkeleyGW](https://www.berkeleygw.org).
Use the `module avail` command to see what versions are available:
```bash
nersc$ module avail berkeleygw
```
## Example
See the [example jobs page](/jobs/examples/) for additional
examples and information about jobs.
### Edison
```
#!/bin/bash
#SBATCH --qos=regular
#SBATCH --time=01:00:00
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=24
#SBATCH --cpus-per-task=2
module load berkeleygw
srun -n 24 epsilon.cplx.x
```
## Support
* [Forum](https://groups.google.com/a/berkeleygw.org/forum/#!forum/help)
* [Documentation](https://berkeleygw.org/documentation/)
!!! tip
If after consulting with the above you believe there is an issue
with the NERSC module, please file a
[support ticket](https://help.nersc.gov).
......@@ -12,11 +12,30 @@ Use the `module avail` command to see what versions are available:
nersc$ module avail cp2k
```
## Example - Cori KNL
```bash
#!/bin/bash
#SBATCH --qos=regular
#SBATCH --time=300
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=64
#SBATCH --cpus-per-task=4
#SBATCH --core-spec=2
module unload craype-haswell
module load craype-mic-knl
module load cp2k/6.1
srun --cpu-bind=cores cp2k.popt -i example.inp
```
## Support
* [Reference Manual](https://manual.cp2k.org/)
* [Forum](http://groups.google.com/group/cp2k)
* [Forum](https://groups.google.com/group/cp2k)
* [FAQ](https://www.cp2k.org/faq)
* [Makefile and build script](https://gitlab.com/NERSC/nersc-user-software/tree/master/cp2k/)
!!! tip
If *after* consulting with the above you believe there is an
......
# Applications
NERSC provides [modules](docs/environment/#nersc-modules-environment)
for a variety of common science codes such as:
* [VASP](vasp/index.md)
* [Quantum ESPRESSO](quantum-espresso/index.md)
* [NAMD](namd/index.md)
# Mathematica
## ![Mathematica logo](./images/mathematica-spikey.png)<br/>Description and Overview
Mathematica is a fully integrated environment for technical
computing. It performs symbolic manipulation of equations, integrals,
differential equations, and most other mathematical
expressions. Numeric results can be evaluated as well.
## How to Use Mathematica
### Running in the Notebook Interface
To use the graphical interface to Mathematica, you will need to
connect to a NERSC machine with X11.
```shell
mylaptop$ ssh -X edison.nersc.gov
```
We highly recommend the use
of [NX](https://www.nersc.gov/users/connecting-to-nersc/using-nx/) to
invoke an X11 session and run the Mathematica notebook interface.
Next, you will need to load the Mathematica module. To use the default
version of Mathematica, use
```shell
nersc$ module load mathematica
```
To start up Mathematica once the module is loaded, you can simply type
```shell
nersc$ mathematica
```
You should see the Mathematica logo ("Spikey") appear, followed by the
application interface.
### Mathematica Licensing
NERSC's Mathematica licenses are for a limited number of seats. Once
you are finished using Mathematica, please be sure you disconnect your
X11 session so that your seat becomes available to the next user. If
you use NX, remember to exit Mathematica within your NX session before
closing, as NX will otherwise keep the session alive, preventing other
users from launching Mathematica.
When starting up a new Mathematica session, check to be sure that you
don't already have an instance of Mathematica running. The most common
issue with Mathematica licensing at NERSC is that another user is
inadvertently using multiple seats.
### Running Mathematica Scripts
To run Mathematica scripts, you can do so in interactive mode or in
batch mode. Both approaches require the use of the job scheduler. On
both Cori and Edison, this
is
[SLURM](https://www.nersc.gov/users/computational-systems/cori/running-jobs/slurm-at-nersc-overview/).
To run in interactive mode, use salloc to obtain access to a compute
node. To avoid using multiple license seats at once (always a good
thing), specify a single node and a single task per node. If you want
to take advantage of parallelism in your script, you can specify more
than one cpu per task. An allocation to run on four cores in the
regular queue would be obtained with a command like the following:
```shell
nersc$ salloc -N 1 -n 1 -c 4 -p regular -t 00:10:00
```
Running the script is then as simple as
```shell
nersc$ srun ./mymathscript.m
```
To run in batch mode, you will need to write a Slurm batch script and
then use sbatch to put your job into the queue. An example batch
script that makes 4 cores available to your script follows.
```shell
#! /bin/bash -l
#SBATCH -p regular
#SBATCH -N 1
#SBATCH -t 00:02:00
#SBATCH -n 1
#SBATCH -c 4
module load mathematica
srun ./mersenne-cori.m
```
If you want to take advantage of parallelism in Mathematica, you can
use the application's built-in parallel commands. (See the Wolfram web
site for more
about
[parallel commands in Mathematica](https://reference.wolfram.com/language/ParallelTools/tutorial/GettingStarted.html).)
Following is an example script that works on Cori. With Cori, you can
use up to 16 cores; with Edison, you can use up to 12 cores. Be sure
the first line of your script points to the correct directory for the
machine on which you're running.
```
#!/global/common/cori/software/mathematica/10.3.0/bin/MathematicaScript -script
# Limit Mathematica to requested resources
Unprotect[$ProcessorCount];$ProcessorCount = 4;
# calculate some Mersenne primes on one core, then on many