Commit 2dd31f56 authored by Brandon's avatar Brandon

Remove Edison references

parent a9a0f739
......@@ -19,15 +19,16 @@ nersc$ module avail abinit
See the [example jobs page](../../jobs/examples/index.md) for additional
examples and infortmation about jobs.
### Edison
### Cori
```
```slurm
#!/bin/bash
#SBATCH --qos=regular
#SBATCH --time=01:00:00
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=24
#SBATCH --ntasks-per-node=32
#SBATCH --cpus-per-task=2
#SBATCH --constraint=haswell
module load abinit
srun abinit < example.in
......
......@@ -21,18 +21,19 @@ nersc$ module avail berkeleygw
See the [example jobs page](../../jobs/examples/index.md) for additional
examples and information about jobs.
### Edison
### Cori
```
```slurm
#!/bin/bash
#SBATCH --qos=regular
#SBATCH --time=01:00:00
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=24
#SBATCH --ntasks-per-node=32
#SBATCH --cpus-per-task=2
#SBATCH --constraint=haswell
module load berkeleygw
srun -n 24 epsilon.cplx.x
srun epsilon.cplx.x
```
## Support
......
......@@ -15,12 +15,13 @@ To use the graphical interface to Mathematica, you will need to
connect to a NERSC machine with X11.
```shell
mylaptop$ ssh -X edison.nersc.gov
mylaptop$ ssh -X cori.nersc.gov
```
We highly recommend the use
of [NX](https://www.nersc.gov/users/connecting-to-nersc/using-nx/) to
invoke an X11 session and run the Mathematica notebook interface.
!!! tip
We highly recommend the use
of [NX](https://www.nersc.gov/users/connecting-to-nersc/using-nx/) to
invoke an X11 session and run the Mathematica notebook interface.
Next, you will need to load the Mathematica module. To use the default
version of Mathematica, use
......@@ -55,10 +56,8 @@ inadvertently using multiple seats.
### Running Mathematica Scripts
To run Mathematica scripts, you can do so in interactive mode or in
batch mode. Both approaches require the use of the job scheduler. On
both Cori and Edison, this
is
[SLURM](https://www.nersc.gov/users/computational-systems/cori/running-jobs/slurm-at-nersc-overview/).
batch mode. Both approaches require the use of
the [job scheduler](../../jobs/index.md).
To run in interactive mode, use salloc to obtain access to a compute
node. To avoid using multiple license seats at once (always a good
......@@ -99,9 +98,8 @@ site for more
about
[parallel commands in Mathematica](https://reference.wolfram.com/language/ParallelTools/tutorial/GettingStarted.html).)
Following is an example script that works on Cori. With Cori, you can
use up to 16 cores; with Edison, you can use up to 12 cores. Be sure
the first line of your script points to the correct directory for the
machine on which you're running.
use up to 16 cores. Be sure the first line of your script points to
the correct directory for the machine on which you're running.
```
#!/global/common/cori/software/mathematica/10.3.0/bin/MathematicaScript -script
......@@ -119,12 +117,6 @@ fast = AbsoluteTiming[Parallelize[Select[Range[5000], MersennePrimeQ]]]
WriteString[$Output, fast]
```
For Edison, the first line of this script would be
```
#!/global/common/edison/das/mathematica/10.1.0/bin/MathematicaScript -script
```
## Documentation
Extensive on-line documentation is available at the Wolfram web site.
......
......@@ -29,7 +29,7 @@ for matrix computation.
## How to Use MATLAB on Cori
MATLAB is available at NERSC on Cori (and Edison until it retires). The number of MATLAB
MATLAB is available at NERSC on Cori. The number of MATLAB
licenses at NERSC is not very large (currently 16), so users should
not be running a MATLAB session when it is not being actively used. If
you use NX, it's particularly easy to think you've released your
......@@ -37,9 +37,9 @@ license when you haven't, since license checkouts persist between NX
sessions.
MATLAB can be run interactively or in batch mode. MATLAB can run on a
compute node with exclusive access to the full usable memory (about 62
GB on Edison nodes and 120 GB on Cori Phase 1 nodes) by submitting a
batch job to the regular queue.
compute node with exclusive access to the full usable memory (about
120 GB on Cori Haswell nodes) by submitting a batch job to the regular
queue.
### Running Interactively
......@@ -52,7 +52,13 @@ module load matlab
matlab
```
It is also possible to run MATLAB on a Cori login node directly. Production computing should not be undertaken on login nodes, however. For long-running or compute intensive jobs, use a batch script.
It is also possible to run MATLAB on a Cori login node
directly. Production computing should not be undertaken on login
nodes, however.
!!! warn
For long-running or compute intensive jobs, use a batch
script.
```
module load matlab
......@@ -64,7 +70,7 @@ matlab
To run one instance of MATLAB non-intearctively through a batch job,
you can use the following job script on Cori:
```
```slurm
#!/bin/bash -l
#SBATCH -q regular
#SBATCH -N 1
......@@ -108,7 +114,7 @@ end
For loop-level parallelism, MATLAB provides the parfor construct.
In order to make MATLAB work in parallel on Edison, you need to make
In order to make MATLAB work in parallel on Cori, you need to make
sure it will access the MATLAB libraries before the standard cray
libraries. You can do this by running this command for bash:
......@@ -123,20 +129,20 @@ setenv LD_PRELOAD "/global/common/sw/cray/cnl6/ivybridge/matlab/R2016b/bin/glnxa
```
However, if you have `LD_PRELOAD` set on a login node, this will
interfere with the regular Edison and Cori environment. It's important
that you only set the `LD_PRELOAD` variable in your parallel MATLAB job
script (or interactive job).
interfere with the regular environment. It's important that you only
set the `LD_PRELOAD` variable in your parallel MATLAB job script (or
interactive job).
#### Parallelism with the MATLAB Compiler
Another way to run MATLAB in parallel is to run multiple instances of
a compiled MATLAB program. By compiling, you create a stand-alone
application that doesn't need to obtain a separate license from the
NERSC license server to run. See [MATLAB Compiler](matlab_compiler) for details.
NERSC license server to run. See [MATLAB Compiler](matlab_compiler)
for details.
## Documentation
Extensive [on-line documentation](http://www.mathworks.com/) is
available. You may subscribe to the MATLAB Digest, a monthly e-mail
newsletter by sending e-mail to subscribe@mathworks.com.
newsletter by sending e-mail to <subscribe@mathworks.com>.
#!/bin/bash
#SBATCH --qos=regular
#SBATCH --time=02:00:00
#SBATCH --nodes=10
#SBATCH --ntasks-per-node=24
#SBATCH --cpus-per-task=2
# make sure correct hugepages module is loaded
module unload $(module -l list 2>&1 | grep craype-hugepages | awk '{print $1}')
module load craype-hugepages8M
module load rca
module load namd
export HUGETLB_DEFAULT_PAGE_SIZE=8M
export HUGETLB_MORECORE=no
srun namd2 ${INPUT_FILE}
......@@ -18,11 +18,6 @@ such that *only* the number of nodes needs to be changed.
--8<-- "docs/applications/namd/cori-hsw.sh"
```
??? example "Edison"
```shell
--8<-- "docs/applications/namd/edison.sh"
```
## Support
* [User's Guide](https://www.ks.uiuc.edu/Research/namd/current/ug/)
......
......@@ -20,18 +20,19 @@ nersc$ module avail paratec
See the [example jobs page](../../jobs/examples/index.md) for additional
examples and information about jobs.
### Edison
### Cori
```
```slurm
#!/bin/bash
#SBATCH --qos=regular
#SBATCH --time=01:00:00
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=24
#SBATCH --ntasks-per-node=32
#SBATCH --cpus-per-task=2
#SBATCH --constraint=haswell
module load paratec
srun -n 24 paratec.mpi
srun paratec.mpi
```
## Support
......
This diff is collapsed.
......@@ -14,7 +14,7 @@ methods.
## Example run scripts
!!! example "Cori KNL"
```bash
```slurm
#!/bin/bash
#SBATCH --qos=regular
#SBATCH --time=01:30:00
......@@ -27,7 +27,7 @@ methods.
```
??? example "Cori Haswell"
```shell
```slurm
#!/bin/bash
#SBATCH --qos=regular
#SBATCH --time=01:30:00
......@@ -39,18 +39,6 @@ methods.
srun siesta < test.fdf > test.out
```
??? example "Edison"
```shell
#!/bin/bash
#SBATCH --qos=regular
#SBATCH --time=01:30:00
#SBATCH --nodes=2
#SBATCH --tasks-per-node=24
module load siesta
srun siesta < test.fdf > test.out
```
## Support
* [Manuals](https://departments.icmab.es/leem/siesta/Documentation/Manuals/manuals.html)
......
......@@ -13,7 +13,7 @@ schemes and an efficient Pulay mixing.
VASP is available only to NERSC users who already have an existing
VASP license. If you have a VASP license, send your license
information to the VASP support at Vienna at
vasp.materialphysik@univie.ac.at (and copy vasp_licensing@nersc.gov)
vasp.materialphysik@univie.ac.at (and copy <vasp_licensing@nersc.gov>)
requesting that they confirm your VASP license to NERSC staff at
vasp_licensing@nersc.gov in order to gain access to the VASP binaries
at NERSC.
......@@ -66,25 +66,15 @@ However, some older versions of the vasp module provide:
### Basic
#### Edison
```bash
--8<-- "docs/applications/vasp/examples/edison-simple.sh"
```
#### Cori Haswell
!!!note
The additional `--constraint=haswell` option is needed because
Cori contains both Haswell and KNL nodes.
```bash
```slurm
--8<-- "docs/applications/vasp/examples/cori-haswell-simple.sh"
```
#### Cori KNL
!!! note
!!! warn
A different module is needed besides the default for the KNL
optimized code.
......@@ -93,6 +83,6 @@ However, some older versions of the vasp module provide:
will be dedicated to the operating system (`--core-spec=2`) to
improve efficiency.
```bash
```slurm
--8<-- "docs/applications/vasp/examples/cori-knl-simple.sh"
```
This diff is collapsed.
# Login Nodes
Opening an [SSH connection](ssh.md) to NERSC systems results in a
connection to a login node. Systems such as Cori and Edison have
multiple login nodes which sit behind a load balancer. New connections
will be assigned a random node. If an account has recently connected
the load balancer will attempt to connect to the same login node as
the previous connection.
connection to a login node. Typically systems will have multiple login
nodes which sit behind a load balancer. New connections will be
assigned a random node. If an account has recently connected the load
balancer will attempt to connect to the same login node as the
previous connection.
## Usage
......
......@@ -205,7 +205,7 @@ You can transfer a file to or from a NERSC machine with scp, in the
same manner as you use ssh:
```shell
$ scp myfile edison.nersc.gov:~/
$ scp myfile cori.nersc.gov:~/
*****************************************************************
* *
* NOTICE TO USERS *
......@@ -236,14 +236,14 @@ Password + OTP:
$ nersc_host # You're on Cori
cori
$ ssh edison # no authentication required
$ ssh cori # no authentication required
*****************************************************************
* *
* NOTICE TO USERS *
* --------------- *
...
$ nersc_host # You're now on Edison
edison
$ nersc_host # You're now on Cori
cori
```
## SSH Control Master
......@@ -272,7 +272,7 @@ platform, you can set the following in `~/.ssh/config`:
```shell
$ cat ~/.ssh/config
Host cori.nersc.gov edison.nersc.gov
Host cori.nersc.gov cori.nersc.gov
ControlMaster auto
ControlPath ~/.ssh/%r@%h:%p
<...Other configurations you may have...>
......@@ -366,13 +366,12 @@ rolled out.
### MFA for ssh keys (sshproxy)
The __sshproxy__ is NERSC-developed solution to provide single
sign-on (SSO) ssh capability, and enable automated workflows. With
sshproxy, you can get an ssh key (using MFA) that you can use for
a limited amount of time (default of 24 hours). You can use the key
to access NERSC systems (e.g., Cori, Edison, Genepool, ...) via
ssh, without further authentication. Options will be available to
request keys with longer lifetimes if you have a specific need.
The __sshproxy__ is NERSC-developed solution to provide single sign-on (SSO)
ssh capability, and enable automated workflows. With sshproxy, you can get an
ssh key (using MFA) that you can use for a limited amount of time (default of
24 hours). You can use the key to access NERSC systems (e.g., Cori) via ssh,
without further authentication. Options will be available to request keys with
longer lifetimes if you have a specific need.
### MFA for MyProxy
......@@ -416,10 +415,9 @@ for creating tokens.
## Status of MFA on User Systems
Currently, MFA is supported on most of the systems that users access
via ssh, such as Cori, Edison, etc. Web and other services will
start supporting MFA in September 2018. The table below shows the
status of MFA on NERSC systems and services.
Currently, MFA is supported on most of the systems that users access via ssh,
such as Cori. Web and other services will start supporting MFA in September
2018. The table below shows the status of MFA on NERSC systems and services.
#### MFA Available Now
......@@ -427,7 +425,6 @@ status of MFA on NERSC systems and services.
[comment]: <> (| Authentication | Host |)
[comment]: <> (|:---:|:---:|)
[comment]: <> (| SSH | Cori |)
[comment]: <> (| | Edison |)
[comment]: <> (| | Denovo |)
[comment]: <> (| | PDSF |)
[comment]: <> (| | Genepool |)
......@@ -446,9 +443,6 @@ status of MFA on NERSC systems and services.
<td style="text-align: center;">Cori</td>
</tr>
<tr>
<td style="text-align: center;">Edison</td>
</tr>
<tr>
<td style="text-align: center;">Denovo</td>
</tr>
<tr>
......
......@@ -15,19 +15,12 @@ encrypted, including passwords. Most versions of SSH provide login
secure ftp client (sftp). Additionally, SSH allows secure X Window
connections.
!!! example
```bash
$ ssh elvis@edison.nersc.gov
elvis@edison.nersc.gov's password: [enter NIM password for user elvis]
edison$
```
## Password-less logins and transfers
Consult the documentation on using the
[SSH Proxy](../mfa/#mfa-for-ssh-keys-sshproxy) service in the MFA
documentation section for ways to connect to NERSC systems without reentering
your password and one-time password.
your password and one-time password.
## SSH certificate authority
......@@ -50,11 +43,6 @@ to confirm the current fingerprints.
4096 SHA256:35yiNfemgwzHCHFrPGWrJBCCqERqLtOVSrR36s1DaPc cori.nersc.gov (RSA)
```
* Edison
```
4096 SHA256:riR+3TGNnPs0uqJxJBbvPU+JR3e/Z0xUzBRsip3ZOJ8 edison.nersc.gov (RSA)
```
* PDSF
```
2048 SHA256:4JGbnhiMkJ5kv1S5+UI5ggTY+Z6DzdEeBQvoFRfN9lw pdsf.nersc.gov (RSA)
......@@ -84,12 +72,6 @@ These are the entries in `~/.ssh/known_hosts`.
cori.nersc.gov ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCsxGw19ZL8EN+NZ9HhD+O/nATuvgZxcEuy/yXnGqz5wMzJj6rK7TsrdU8rdJNrhZDe3yjpCiKvqkbSKp22jK2/iMAeWDQvYpMgC6KyiNd0hztowtMFJEwb8gVmtkVioqIaf9ufJnOO0LX5A5J/4fQhICfbyPiX8SsjX0p655/kIm3T6hr7t89b4IkRu19/uWufbNaV/mZSFWl7asLKXJNTMhzEn6bsTcAqlm55Tp4NvCe1hvv6OY/vU5luDz09UDmnDfr/uukmVm5aIjtlZBGqbOe7huNJGIWhoGCN/SoArRu9T9c9fjOlRMOHcf0QYMQmxFQnR0TkJZQoJ5N+EYNUIB9dvnJs2mlN0ZEuUU0RwAUOge7RwujiZ2AWp/dV/PNvLGmDVUxiyXC0Uuw57Ga2e49hYisYU/J/NPp9AbHqO8M6kZqYdqWKYueIsM3FDti3vUbjV4J6sL6mOBbxuJpUhUEX5UXxGbR39hDVx9Lsj4dszu+mcBFnDNcpRCDjw3z+hDqdNNpzhIRlbHQErLBWL3vnn2MLnb/3z163gyRtu1iTuR5myBIs9jLDAsX94VbBzKWdCFe22x4Eo6HwB6u+UHlXov0fnBXtAmgwRegc1gQwxi2FXB/ty0q1EO+PYo3fjUVRRb4uqBBIvpFarwtL0T6iYAYgHY11vH9Z2BFAHQ==
```
### Edison
```
edison.nersc.gov ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQPHTSKic9gj6kyhfSkBWZdMgnESIQVI0vg7eNIkshb+pXGGGxWQXrIX3IL1W9bvwrjGNc2JAd9c5Y1CfV0B6sKHDm70pPTnZ3bkvWwq81bloepzJHWE9NpNd7xIlVaM7sh7fMURD5/e7E46qldrpBBtudJG9ZgmjxdmDSlWBTp6scDqehBZ+yaRg0N7zoEA86synBi/0DEDzqarHlvXwXS5mWBGnlC3ZI5Uz/QHD4K26y86SeKYf2EKUI0am+gcPRbUHjDLyThK+qFGveRe9G41eEw40hxmO/yooYgCjCEKVbfU4Po2uR6qb/p/cpeMLOyZ3itrZce6WMgQxw+8g9MPKWuCKH7SJnJ/70YQyLuSlc98mr3AE6fjcZZy8Uf4ckH49qXTH4ILYPEzqLzI86eM+tJltayUWV9aQVAG6lBn14DyCvAyAfts+RCE8JkGJzcUSu7UILiIrqEyOpMOrZ/z2wM7mYJlAVBbrjT0LO6hXh/ET/npo7mMhotjtptXk9qg7DLUfL647OZvWjxQxZlE6jtpHilOaCcXpY3pXUZTtza7kv3pRbnPmzWU0iKKLmqsjtAT773SIvJ/78MwwqIF4pEBiPx7Ixmf+rHwpQV/P6ADBadpTfx28297ZjzvQZ+gTscBWULxeUFNfZtm+jmpsMGPNJTXAAlyVW13zO9w==
```
## Troubleshooting
### "Access Denied" or "Permission Denied"
......@@ -109,7 +91,7 @@ This message may appear when a connection to a new machine is first
established:
```
The authenticity of host 'edison.nersc.gov' can't be established.
The authenticity of host 'cori.nersc.gov' can't be established.
RSA key fingerprint is <omitted>
Are you sure you want to continue connecting (yes/no)?
```
......@@ -132,12 +114,12 @@ It is also possible that a host key has just been changed.
```
Ensure that your `~/.ssh/known_hosts` file contains the correct entries for
Edison and Cori and confirm the fingerprints using the posted fingerprints
above. Add the [certificate authority](#ssh-certificate-authority) line to
your known_hosts file if you version of ssh supports SSH certificates.
Cori and confirm the fingerprints using the posted fingerprints above. Add the
[certificate authority](#ssh-certificate-authority) line to your known_hosts
file if you version of ssh supports SSH certificates.
1. Open `~/.ssh/known_hosts`
1. Remove any lines referring Cori or Edison and save the file
1. Remove any lines referring Cori and save the file
1. Paste the host key entries from above or retry connecting to the host and
accept the new host key after verify that you have the correct "fingerprint"
from the above list.
......@@ -45,10 +45,10 @@ data analysis.
### Scratch File Systems
Our two primary computational systems, Edison and Cori, have dedicated
parallel file systems based on Lustre that are optimized for
short-term storage of application output and checkpoints.
Our primary computational system, Cori, has a dedicated parallel file system
based on Lustre that is optimized for short-term storage of application output
and checkpoints.
!!! warning "Purging on Scratch File Systems"
Running commands with the intent of circumventing purge policies
on scratch filesystems is **not allowed**.
......@@ -341,12 +341,11 @@ and PI Proxies can request a special directory by contacting NERSC consulting.
#### Intent:
Cori and Edison each have large, local, parallel scratch file systems
dedicated to the users of those systems. The scratch file systems are
intended for temporary uses such as storage of checkpoints or
application result output. If you need to retain files longer than
the purge period (see below), the files should be copied to the
project or home file systems, or to HPPS.
Cori has a large, local, parallel scratch file system dedicated to the users of
those systems. The scratch file system is intended for temporary uses such as
storage of checkpoints or application result output. If you need to retain
files longer than the purge period (see below), the files should be copied to
the project or home file systems, or to HPSS.
!!! warning "Purging on Scratch File Systems"
Running commands with the intent of circumventing purge policies
......@@ -373,13 +372,12 @@ account is deactivated.
#### Default Allocation:
Each user is allocated a directory in the local scratch file systems
and in the global scratch file system. The default local scratch
quota on Cori and Edison is 20 TB and 10TB. Users may request an
increase in quota using a web form. Quota increase requests are
evaluated by the NERSC data group team, which considers the merits and
duration of the request, and also must consider the amount of disk
space available.
Each user is allocated a directory in the local scratch file systems and in the
global scratch file system. The default local scratch quota on Cori is 20 TB.
Users may request an increase in quota using a web form. Quota increase
requests are evaluated by the NERSC data group team, which considers the merits
and duration of the request, and also must consider the amount of disk space
available.
### Burst Buffer
......
......@@ -181,7 +181,7 @@ Users can check their HPSS usage and quotas with the hpssquota command
on Cori. You view usages on a user level:
```
cori03> hpssquota -u usgtest
cori$ hpssquota -u usgtest
HPSS Usage for User usgtest
REPO STORED [GB] REPO QUOTA [GB] PERCENT USED [%]
-----------------------------------------------------------------------------------
......@@ -200,7 +200,7 @@ maximum amount your PI has allocated for you to store data, and the
You can also view usage on a repo level:
```
edison03> hpssquota -r ntrain
nersc$ hpssquota -r ntrain
HPSS Usage for Repo ntrain
USER STORED [GB] USER QUOTA [GB] PERCENT USED [%]
......
......@@ -25,8 +25,7 @@ shared across all systems, you may want to install your software
stacks into separate subdirectories depending on the system or the
processing architecture. For some general programs you can use the
same installs across all systems, but for best performance, we
recommend separate installs for each system and architecture (e.g. for
edison vs. for Cori KNL).
recommend separate installs for each system and architecture.
!!! warning
Global common is mounted read-only on the computes to
......
......@@ -5,7 +5,7 @@
Due to backfill scheduling short and variable length jobs generally
start quickly resulting in much better job throughput.
```bash
```slurm
#SBATCH --time-min=<lower_bound>
#SBATCH --time=<upper_bound>
```
......@@ -20,17 +20,19 @@ checkpoint/restart chained together.
## I/O performance
Edison and Cori each have dedicated large, local, parallel scratch
file systems. The scratch file systems are intended for temporary
uses such as storage of checkpoints or application input and
output. Data and I/O intensive applications should use the local
scratch (or Burst Buffer) filesystems.
Cori has dedicated large, local, parallel scratch file systems. The
scratch file systems are intended for temporary uses such as storage
of checkpoints or application input and output. Data and I/O intensive
applications should use the local scratch (or Burst Buffer)
filesystems.
These systems should be referenced with the environment variable
`$SCRATCH`.
!!! tip
On Cori the [Burst Buffer](examples/index.md#burst-buffer-test) offers the best I/O performance.
!!! tip
On Cori
the [Burst Buffer](examples/index.md#burst-buffer-test) offers the
best I/O performance.
!!! warn
Scratch filesystems are not backed up and old files are
......@@ -42,7 +44,7 @@ A batch job will not start if the specified file system is unavailable
due to maintenance or an outage or if a performance issue with
filesystem is detected.
```bash
```slurm
#SBATCH --license=SCRATCH,project
```
......@@ -56,23 +58,10 @@ filesystem is detected.
* `seqfs`
* `cvmfs`
### Edison
* `scratch1` (or `SCRATCH`)
* `scratch2` (or `SCRATCH`)
* `scratch3`
* `cscratch1`
* `project`
* `projecta`
* `projectb`
* `dna`
* `seqfs`
* `cvmfs`
## Running Large Jobs (over 1500 MPI tasks)
Large jobs may take a longer to start up, especially on KNL nodes. The
srun option --bcast=<destination_path> is recommended for large jobs
srun option `--bcast=<destination_path>` is recommended for large jobs
requesting over 1500 MPI tasks. By default Slurm loads the executable
to the allocated compute nodes from the current working directory,
this may take long time when the file system (where the executable
......@@ -92,14 +81,14 @@ For jobs which are sensitive to interconnect (MPI) performance and
utilize less than ~300 nodes it is possible to request that all nodes
are in a single Aries dragonfly group.
Slurm has a concept of "switches" which on Cori and Edison are
configured to map to Aries electrical groups. Since this places an
additional constraint on the scheduler a maximum time to wait for the
requested topology can be specified.
Slurm has a concept of "switches" which on Cori are configured to map
to Aries electrical groups. Since this places an additional constraint
on the scheduler a maximum time to wait for the requested topology can
be specified.
!!! example
Wait up to 60 minutes
```bash
```slurm
sbatch --switches=1@60 job.sh
```
......@@ -166,7 +155,7 @@ There are other modes available with the `MPICH_RANK_REORDER_METHOD`
environment variable, including one which lets the user provide a file
called `MPICH_RANK_ORDER` which contains a list of each task's
placement on each node. These options are described in detail in the
`intro_mpi` man page on Cori and Edison.
`intro_mpi` man page.
#### `grid_order`
......@@ -192,7 +181,7 @@ then set `MPICH_RANK_REORDER_METHOD=3` before running the job, which
tells Cray MPI to read the `MPICH_RANK_ORDER` file to set the MPI task
placement. For more information, please see the man page `man
grid_order` (available when the `perftools-base` module is loaded) on
Cori and Edison.
Cori.
## Serial jobs
......
#!/bin/bash
#SBATCH --qos=debug
#SBATCH --time=5
#SBATCH --nodes=2
#SBATCH --tasks-per-node=24
srun check-mpi.intel.edison
Hello from rank 0, on nid00117. (core affinity = 0,24)
Hello from rank 24, on nid00118. (core affinity = 0,24)
Hello from rank 5, on nid00117. (core affinity = 14,38)
Hello from rank 6, on nid00117. (core affinity = 3,27)
Hello from rank 7, on nid00117. (core affinity = 15,39)
Hello from rank 9, on nid00117. (core affinity = 16,40)
Hello from rank 10, on nid00117. (core affinity = 5,29)
Hello from rank 11, on nid00117. (core affinity = 17,41)
Hello from rank 13, on nid00117. (core affinity = 18,42)
Hello from rank 14, on nid00117. (core affinity = 7,31)
Hello from rank 15, on nid00117. (core affinity = 19,43)
Hello from rank 17, on nid00117. (core affinity = 20,44)
Hello from rank 19, on nid00117. (core affinity = 21,45)
Hello from rank 20, on nid00117. (core affinity = 10,34)
Hello from rank 21, on nid00117. (core affinity = 22,46)
Hello from rank 22, on nid00117. (core affinity = 11,35)
Hello from rank 1, on nid00117. (core affinity = 12,36)
Hello from rank 2, on nid00117. (core affinity = 1,25)
Hello from rank 3, on nid00117. (core affinity = 13,37)
Hello from rank 4, on nid00117. (core affinity = 2,26)
Hello from rank 8, on nid00117. (core affinity = 4,28)
Hello from rank 12, on nid00117. (core affinity = 6,30)
Hello from rank 16, on nid00117. (core affinity = 8,32)
Hello from rank 18, on nid00117. (core affinity = 9,33)
Hello from rank 23, on nid00117. (core affinity = 23,47)
Hello from rank 26, on nid00118. (core affinity = 1,25)
Hello from rank 32, on nid00118. (core affinity = 4,28)
Hello from rank 34, on nid00118. (core affinity = 5,29)
Hello from rank 35, on nid00118. (core affinity = 17,41)
Hello from rank 38, on nid00118. (core affinity = 7,31)
Hello from rank 41, on nid00118. (core affinity = 20,44)
Hello from rank 42, on nid00118. (core affinity = 9,33)
Hello from rank 44, on nid00118. (core affinity = 10,34)
Hello from rank 46, on nid00118. (core affinity = 11,35)
Hello from rank 25, on nid00118. (core affinity = 12,36)
Hello from rank 27, on nid00118. (core affinity = 13,37)
Hello from rank 28, on nid00118. (core affinity = 2,26)
Hello from rank 29, on nid00118. (core affinity = 14,38)
Hello from rank 30, on nid00118. (core affinity = 3,27)
Hello from rank 31, on nid00118. (core affinity = 15,39)
Hello from rank 33, on nid00118. (core affinity = 16,40)
Hello from rank 36, on nid00118. (core affinity = 6,30)
Hello from rank 37, on nid00118. (core affinity = 18,42)
Hello from rank 39, on nid00118. (core affinity = 19,43)
Hello from rank 40, on nid00118. (core affinity = 8,32)
Hello from rank 43, on nid00118. (core affinity = 21,45)
Hello from rank 45, on nid00118. (core affinity = 22,46)
Hello from rank 47, on nid00118. (core affinity = 23,47)
#!/bin/bash
#SBATCH --qos=debug
#SBATCH --time=5
#SBATCH --nodes=2
#SBATCH --tasks-per-node=2
#SBATCH --cpus-per-task=24