Commit 3d83a859 authored by Brandon's avatar Brandon

Merge branch 'lgerhard/replace_proj_with_community' into 'master'

replace project with community

See merge request !579
parents 705b7bc8 ee3fa82f
......@@ -274,16 +274,15 @@ line on a Unix-like computer.
##### Installing the Client
You can download the bash client `sshproxy.sh` from a project
directory:
You can download the bash client `sshproxy.sh` via scp:
```
$ scp myusername@dtn01.nersc.gov:/project/projectdirs/mfa/NERSC-MFA/sshproxy.sh .
$ scp myusername@dtn01.nersc.gov:/global/cfs/cdirs/mfa/NERSC-MFA/sshproxy.sh .
```
where `myusername` is your NERSC login ID. The command uses a data
transfer node (dtn01), but you can use any machine which you can
access that can access the `/project` file system.
access that can access the Community file system.
##### Using sshproxy
......@@ -498,7 +497,7 @@ Use your favorite file transfer tool and download the following
Windows executable:
```
/project/projectdirs/mfa/NERSC-MFA/sshproxy.exe
/global/cfs/cdirs/mfa/NERSC-MFA/sshproxy.exe
```
##### Using sshproxy
......
......@@ -25,9 +25,9 @@ the Results of Federally Funded Scientific Research
and the DOE has issued a [Statement on Digital Data
Management](https://science.osti.gov/Funding-Opportunities/Digital-Data-Management/).
NERSC resources are intended for users with active allocations, and as
NERSC resources are intended for users with active allocations and, as
described below, NERSC cannot guarantee long-term data access without
a prior, written, service-level agreement. Please carefully consider
a prior, written, service-level agreement. Please carefully consider
these policies, including their limitations and restrictions, as you
develop your data management plan.
......@@ -56,7 +56,7 @@ managing their data:
## Storage Resources
### NERSC Global Filesystem (NGF)
### NERSC Global Filesystems (NGF)
NGF is a collection of centerwide file systems, based on IBM’s
Spectrum Scale, available on all the systems at NERSC. There are three
......@@ -65,7 +65,7 @@ main file systems that comprise NGF: one providing home directories
common login user environment for all our systems ([global
common](../filesystems/global-common.md)), and one for sharing data
among collaborators on a science project or team
([project](../filesystems/project.md)). The main focus of NGF is data
([Community](../filesystems/community.md)). The main focus of NGF is data
sharing, ease of workflow management (i.e., not moving data around or
maintaining unnecessary copies of data), and data analysis.
......@@ -100,12 +100,12 @@ on the [Quota and Purging page](../filesystems/quotas.md).
## Data Retention Policy
Data in a group's project directory will be retained as long as the
Data in a group's Community directory will be retained as long as the
group has an allocation at NERSC. Allocations are renewed on a yearly
basis (see our section on accounts for information on renewal). If a
group's allocation is not renewed, any data in their project directory
will be automatically archived into HPSS. Please see the [project file
system](../filesystems/project.md#lifetime) page for details
group's allocation is not renewed, any data in their Community directory
will be automatically archived into HPSS. Please see the [Community File
System](../filesystems/community.md#lifetime) page for details
on the timing of this process.
Currently NERSC has no plans to delete data in HPSS. Should NERSC ever
......@@ -121,13 +121,13 @@ be deleted according to the individual system policies.
### [Data Transfer Nodes](../systems/dtn/index.md)
To enable high speed data transfer, we provide a set of parallel
transfer servers tuned primarily for WAN data movement (into and out
of the facility), and secondarily for high speed local transfers
(e.g., NGF to HPSS). Staff from NERSC and ESnet with extensive data
transfer experience are available to aid scientists in organizing,
packaging, and moving their data to the most appropriate storage
resource at our facility.
To enable high speed data transfer, we provide a set of [parallel
transfer servers](../systems/dtn/index.md) tuned primarily for WAN data
movement (into and out of the facility), and secondarily for high
speed local transfers (e.g., NGF to HPSS). Staff from NERSC and ESnet
with extensive data transfer experience are available to aid
scientists in organizing, packaging, and moving their data to the most
appropriate storage resource at our facility.
### Globus
......@@ -140,7 +140,7 @@ self-managed transfers.
### Direct Web Access to Data
For rapid and simple data sharing, users can enable web access to
files in their project file system. See
files in their Community file system. See
the
[Science Gateways page](../services/science-gateways.md#getting-started).
......@@ -232,8 +232,8 @@ usage, performance, and data retention needs.
The [home file system](../filesystems/global-home.md) is intended to
hold source files, configuration files, etc., and is optimized for
small to medium sized files. It is NOT meant to hold the output from
your application runs; the scratch or project file systems should be
used for computational output.
your application runs; the scratch or (infrequently) the Community file
systems should be used for computational output.
#### Stability
......@@ -259,13 +259,13 @@ year from the date your account is deactivated.
Each user is allocated a directory with a 40 GB hard quota in the home
file system. This is the default allocation granted to all users.
### Project File System
### Community File System
#### Intent
The [project file system](../filesystems/project.md) is primarily
The [Community file system](../filesystems/community.md) is primarily
intended for sharing data within a team or across computational
platforms. The project file systems are parallel file systems
platforms. The Community file systems is a parallel file system
optimized for high-bandwidth, large-block-size access to large
files. Once any active production and/or analysis is completed and you
don't need regular access (> 1 year) to the data any longer, you
......@@ -274,7 +274,7 @@ transfer it back to your home institution.
#### Stability
These file systems have redundancy in the servers and storage and
This file system has redundancy in the servers and storage and
duplicate copies of metadata, so we expect good stability and
reliability. With high demand and capacity shared usage, we do expect
some degradation on availability of the system (97% is the target
......@@ -282,8 +282,8 @@ overall availability).
#### Backup
Project directories are not backed up. Instead they use a snapshot
capability to provide users a seven-day history of their project
Community directories are not backed up. Instead they use a snapshot
capability to provide users a seven-day history of their
directories. Users can use this capability to recover accidentally
deleted files. It is recommended that users back up any essential data
in our tape archive system or at another location.
......@@ -292,28 +292,30 @@ in our tape archive system or at another location.
Data for active users is not purged from this space. A user or project
will be considered inactive if they do not have an active allocation
and have not accessed the data in at least one year. All project
and have not accessed the data in at least one year. All Community
directories will be archived to tape and maintained on disk for one
year from the date your account is deactivated. For details on the
process, see project file system.
process, see [Community file system](../filesystems/community.md).
#### Default Allocation
Each repository is allocated a directory in the project file
system. The default quota for project directories is 1 TB and the
default name of a project directory is the repository name
Each repository is allocated a directory in the Community file
system. The default quota for Community directories is 20 TB and the
default name of a Community directory is the repository name
(i.e. m767). These directories are owned by the Principal
Investigators (PIs) and are accessible to everyone in the Unix group
associated with the repository. If files need to be shared with a
group that is different from a repository group, PIs and PI Proxies
can request a special project directory by filling out the Project
can request a special Community directory by filling out the Project
Directory Request form in the "Request Forms" section at the [NERSC
help portal](https://nersc.service-now.com/).
NERSC is working to deploy a new file system to replace the project
file system. After this is deployed (expected late 2019), quotas on
this system will be determined by DOE allocations managers based on
information you provide on your ERCAP form.
Quotas above the defaults are approved by DOE allocations managers
based on information projects provide on their ERCAP
application. Short term quota increases can be arranged at NERSC's
discretion, but only until the end of the allocation year when they
must be approved by a DOE allocation manager via the ERCAP process.
### Global Common File System
......@@ -362,7 +364,7 @@ Cori has a large, local, parallel scratch file system dedicated to the
users of the system. The scratch file system is intended for temporary
uses such as storage of checkpoints or application result output. If
you need to retain files longer than the purge period (see below), the
files should be copied to the project file systems or to HPSS.
files should be copied to the Community file systems or to HPSS.
!!! warning "Purging on Scratch File Systems"
Running commands with the intent of circumventing purge policies
......@@ -484,18 +486,19 @@ users' activities or undermines the long term viability of the system.
NERSC establishes defaults for allocations, retention, and backup to
serve the needs of the majority of our users, avoid oversubscription,
and place practical limits on the costs of storage provided. We are
willing to work with individual projects to accommodate special
and place practical limits on the costs of storage provided.
Quotas above the defaults on the Community file system are approved by
DOE allocations managers based on information projects provide on their
ERCAP application. Short term quota increases can be arranged at NERSC's
discretion, but only until the end of the allocation year when they
must be approved by a DOE allocation manager via the ERCAP process.
We are willing to work with individual projects to accommodate special
needs. If you need additional space, please fill out the "Request a
Disk Quota Change" in form in the "Request Forms" section at the
[NERSC help portal](https://nersc.service-now.com/).
NERSC is working to deploy a new file system to replace the project
file system. After this is deployed (expected late 2019), quotas on
this system will be determined by DOE allocations managers based on
information you provide on your ERCAP form.
### Proper Use of Software and Data
Please consult the NERSC User Agreement for the policies regarding
......
......@@ -16,7 +16,7 @@ separare directory that is as locked down as possible.
### Sharing with Other Members of Your Project
NERSC's [Project file system](../filesystems/project.md) is set up
NERSC's [Community file system](../filesystems/community.md) is set up
with group read and write permissions and is ideal for sharing with
other members of your project. There is a directory for every active
project at NERSC and all members of that project should have access to
......@@ -62,7 +62,6 @@ For a full list of options pass the `--help` flag.
Files that remain untaken 12 weeks after being given will be
purged from the staging area.
## Sharing Data outside of NERSC
You can easily and quickly share data over the web using our [Science
......
......@@ -28,7 +28,7 @@ NERSC and other sites:
!!! tip
**"Do you need to transfer at all?" If your data is on NERSC
Global File Systems (`/global/project`, `/global/projecta`,
Global File Systems (`/global/cfs`, `/global/projecta`,
`/global/cscratch`), data transfer may not be necessary because
these file systems are mounted on almost all NERSC
systems. However, if you are doing a lot of IO with these files,
......
......@@ -280,7 +280,7 @@ You can create and install your own modules for your convenience or
for sharing software among collaborators. The module definition files
can be placed in the following locations:
* project directory
* Community directory
* your home directory
* available file system.
......@@ -292,13 +292,13 @@ want to use the software.
As an example, we have modulefile named *myzlib* located in
`/global/project/projectdirs/mpccc/usg/modulefiles/cori`
`/global/cfs/cdirs/mpccc/usg/modulefiles/cori`
To register this modulefile with our modules environment we run the
following commands:
```bash
nersc$ module use /global/project/projectdirs/mpccc/usg/modulefiles/cori
nersc$ module use /global/cfs/cdirs/mpccc/usg/modulefiles/cori
nersc$ module load myzlib/1.2.7
```
......
......@@ -7,7 +7,7 @@
### Snapshots
Global homes and project use a *snapshot* capability to provide users
Global homes and Community use a *snapshot* capability to provide users
a seven-day history of their directories. Every directory and
sub-directory in global homes contains a ".snapshots" entry.
......@@ -28,8 +28,9 @@ few days to complete.
## Purging
Files in `$SCRATCH` directories may be purged if they are older than
12 weeks (defined by last access time).
!!! note
See [Filesystem Quotas and Purging](quotas.md) for detailed information about inode,
space quotas, and file system purge policies.
!!! warning
`$SCRATCH` directories are **not** backed up
# Community File System
# Community File System (CFS)
The Community File System will be deployed January 21, 2020. It will
replace the Project File System as a globally available file system
for sharing scientific data. Please see
[the slides](https://www.nersc.gov/assets/Uploads/CFS-Deployment-Talk-for-NUG.pdf)
and
[the video](https://youtu.be/ftBJTDjL49I) from the December 2019 NUG
videoconference for details on the deployment.
The Community File System (CFS) is a global file system available on all
NERSC computational systems. It allows sharing of data between users,
systems, and the "outside world".
Quotas on Community will be determined by DOE Program Managers based on
## Usage
Every MPP repository has an associated Community directory and unix
group. Community directories are created in `/global/cfs/cdirs`. All
members of the project have access through their membership in the
unix group.
Occasionally there are cases where the above model is too
limiting. For example, large projects with multiple working groups may
wish to have separate Community directories with separate quotas for
each working group.
In these cases, a PI or PI Proxy for a repository may request an
additional Community directory with a specific name. This will result
in the creation of a new Unix group with that name, consisting solely
of the requestor, followed by the creation of the Community directory
itself. The PI or PI Proxy must then use NIM to add users to the
newly-created Unix group to give them access to the new Community
directory.
!!! info
All of a repository's Community directories share one quota. The
PI or PI Proxy can adjust the relative quota amounts by opening a
ticket with NERSC.
## Quotas
Quotas on Community are determined by DOE Program Managers based on
information PIs supply in their yearly ERCAP requests. Temporary quota
increases may be granted at NERSC's discretion but only until the end
of the current allocation year.
......@@ -17,3 +43,65 @@ of the current allocation year.
See [quotas](quotas.md) for detailed information about inode,
space quotas, and file system purge policies.
## Performance
The system has a peak aggregate bandwidth of at least 100 GB/sec
bandwidth for streaming I/O. While user applications that depend on
high-bandwidth for streaming large files *can* use the Community File
System, it is recommended to use [Cori scratch](../cori-scratch) or
the [Burst Buffer](../cori-burst-buffer) instead.
## Backup
All NERSC users should backup important files on a regular
basis. Ultimately, it is the user's responsibility to prevent data
loss. However, NERSC provides some mechanisms in protecting against
data loss.
### Snapshots
Community directories use a *snapshot* capability to provide users a
seven-day history of their contents. Every directory and sub-directory
in a Community directory contains a ".snapshots" entry.
* `.snapshots` is invisible to `ls`, `ls -a`, `find` and similar commands
* Contents are visible through `ls -F .snapshots`
* Can be browsed normally after `cd .snapshots`
* Files cannot be created, deleted or edited in snapshots
* Files can *only* be copied *out* of a snapshot
## Lifetime
Community directories will remain in existence as long as the owning
project is active. Projects typically "end" at the end of a NERSC
Allocation Year. This happens when the PI chooses not to renew the
project, or DOE chooses not to provide an allocation for a renewal
request. In either case, the following steps will occur following the
termination of the project:
1. **-365 days** - The start of the new Allocation Year and no Project
renewal
The data in the Community directory will remain available on the
Community File System until the start of the next Allocation
Year.
1. **+0 days** - The start of the following Allocation Year
PIs notified that the affected Community directory will be
archived, and then removed from the file system in 90 days.
1. **+30 days**
The Community directory will become read-only.
1. **+60 days**
The full pathname to the Community directory will be
modified. Automated scripts will likely fail.
1. **+90 days**
User access to the directory will be terminated. The directory
will then be archived in HPSS, under ownership of the PI, and
subsequently removed from the file system.
......@@ -50,4 +50,4 @@ loss.
## Lifetime
Cori scratch directories are [purged](quotas.md). Cori scratch
directories may be deleted after a user is no longer active.
\ No newline at end of file
directories may be deleted after a user is no longer active.
......@@ -32,9 +32,9 @@ performance for parallel jobs. Referenced by the environment variable
A performant platform to install software stacks and compile
code. Mounted read-only on compute nodes.
### [Project](project.md)
### [Community](community.md)
Large, permanent, medium-performance file system. Project directories
Large, permanent, medium-performance file system. Community directories
are intended for sharing data within a group of researchers.
### [Archive](archive.md) (HPSS)
......
# Project filesystem
The project file system is a global file system available on all NERSC
computational systems. It allows sharing of data between users,
systems, and the "outside world".
## Usage
Every MPP repository has an associated project directory and unix
group. Project directories are created in `/project/projectdirs`.
All members of the project have access through their membership in the
unix group.
Occasionally there are cases where the above model is too
limiting. For example:
* large projects with multiple MPP repositories
* long-term projects which outlive specific MPP repositories
In these cases, a project directory administrator may request the
creation of a "designer" project directory with a specific name. This
will result in the creation of a new Unix group with that name,
consisting solely of the project directory administrator, followed by
the creation of the project directory itself. The project directory
administrator must then use Iris to add users to the newly-created Unix
group.
!!! info
If you are a _PI_ or a _PI Proxy_, you can request a designer project
directory by submitting a ticket.
## Quotas
!!! note
See [quotas](quotas.md) for detailed information about inode,
space quotas, and file system purge policies.
## Performance
The system has a peak aggregate bandwidth of 130 GB/sec bandwidth for
streaming I/O. While user applications that depend on high-bandwidth
for streaming large files *can* use the Project file system, it is
recommended to use [Cori scratch](../cori-scratch) or the [Burst
Buffer](../cori-burst-buffer) instead.
## Backup
All NERSC users should backup important files on a regular
basis. Ultimately, it is the user's responsibility to prevent data
loss. However, NERSC provides some mechanisms in protecting against
data loss.
### Snapshots
Project directories use a *snapshot* capability to provide users a seven-day
history of their project directories. Every directory and
sub-directory in a project directory contains a ".snapshots" entry.
* `.snapshots` is invisible to `ls`, `ls -a`, `find` and similar commands
* Contents are visible through `ls -F .snapshots`
* Can be browsed normally after `cd .snapshots`
* Files cannot be created, deleted or edited in snapshots
* Files can *only* be copied *out* of a snapshot
## Lifetime
Project directories will remain in existence as long as the owning
project is active. Projects typically "end" at the end of a NERSC
Allocation Year. This happens when the PI chooses not to renew the
project, or DOE chooses not to provide an allocation for a renewal
request. In either case, the following steps will occur following the
termination of the project:
1. **-365 days** - The start of the new Allocation Year and no Project
renewal
The data in the project directory will remain available on the
project file system until the start of the next Allocation
Year. Archival process begins.
1. **+0 days** - The start of the following Allocation Year
Users notified that the affected project directory will be
archived, and then removed from the file system in 90 days.
1. **+30 days**
The project directory will become read-only.
1. **+60 days**
The full pathname to the project directory will be
modified. Automated scripts will likely fail.
1. **+90 days**
User access to the directory will be terminated. The directory
will then be archived in HPSS, under ownership of the PI, and
subsequently removed from the file system.
......@@ -8,9 +8,9 @@ time period shown below.
| file system | space | inodes | purge time | Consequence for Exceeding Quota |
|-----------------|-------|--------|------------|---------------------------------|
| Project | 1 TB | 1 M | - | No new data can be written |
| Global HOME | 40 GB | 1 M | - | No new data can be written |
| Global common | 10 GB | 1 M | - | No new data can be written |
| Community | 20 TB | 20 M | - | No new data can be written |
| Global HOME | 40 GB | 1 M | - | No new data can be written |
| Global common | 10 GB | 1 M | - | No new data can be written |
| Cori SCRATCH | 20 TB | 10 M | 12 weeks | Can't submit batch jobs |
## Policy
......@@ -37,16 +37,16 @@ To see current usage for home and available scratch filesystems:
nersc$ myquota
```
For project you can use
For Community you can use
```
nersc$ prjquota <project_name>
nersc$ cfsquota <project_name>
```
or use `myquota` with the full path to the directory
```
nersc$ myquota --path=/project/projectdirs/<project_name>
nersc$ myquota --path=/global/cfs/cdirs/<project_name>
```
For global common software you can use
......@@ -64,8 +64,12 @@ nersc$ myquota --path=/global/common/software/<project_name>
### Increases
If you or your project needs additional space you may request it via
the
[Disk Quota Increase Form](https://nersc.service-now.com/nav_to.do?uri=catalog_home.do).
the [Disk Quota Increase
Form](https://nersc.service-now.com/nav_to.do?uri=catalog_home.do).
Quota on the Community file system are determined by DOE Program
Managers based on information PIs supply in their yearly ERCAP
requests. Temporary quota increases may be granted at NERSC's
discretion but only until the end of the current allocation year.
## Purging
Some NERSC file systems are purged. This means the files not read
......
......@@ -106,13 +106,13 @@ due to maintenance or an outage or if a performance issue with
filesystem is detected.
```slurm
#SBATCH --license=SCRATCH,project
#SBATCH --license=SCRATCH,community
```
### Cori
* `cscratch1` (or `SCRATCH`)
* `project`
* `community`
* `projecta`
* `projectb`
* `dna`
......
......@@ -693,7 +693,7 @@ flag) or allow multiple applications to share a node (with the
#SBATCH --cpus-per-task=2
#SBATCH --time=01:00:00
#SBATCH --job-name=my_job
#SBATCH --licenses=project
#SBATCH --licenses=community
#SBATCH --exclusive
srun --cpu-bind=cores ./mycode.exe # pure MPI, 64 MPI tasks
......@@ -721,7 +721,7 @@ slots on the node (total of 64 CPUs, or 64 slots) by specifying the
#SBATCH --mem=10GB
#SBATCH --time=01:00:00
#SBATCH --job-name=my_job2
#SBATCH --licenses=project
#SBATCH --licenses=community
#SBATCH --shared
export OMP_NUM_THREADS=4
......@@ -743,7 +743,7 @@ slots on the node (total of 64 CPUs, or 64 slots) by specifying the
#SBATCH --mem=16GB
#SBATCH --time=01:00:00
#SBATCH --job-name=my_job3
#SBATCH --licenses=project,SCRATCH
#SBATCH --licenses=community,SCRATCH
#SBATCH --shared
export OMP_NUM_THREADS=6
......
......@@ -2,7 +2,7 @@
#SBATCH --qos=debug
#SBATCH --nodes=4
#SBATCH --time=10:00
#SBATCH --licenses=project,cscratch1
#SBATCH --licenses=community,cscratch1
#SBATCH --constraint=haswell
srun -n 128 -c 2 --cpu_bind=cores ./a.out
......
......@@ -22,7 +22,7 @@ uses this same API to provide a command line interface that can be used
interchangeably with the python API. Run "qdo -h" to see the task line options.
```
module use /project/projectdirs/cosmo/software/modules/$NERSC_HOST
module use /cfs/cdirs/cosmo/software/modules/$NERSC_HOST
module load qdo/0.7
```
......
......@@ -44,25 +44,74 @@ not be able to achieve the peak potential performance of roughly 6.5
GB/s per node.
## Known Issues
There are a number of known issues to be aware of when using the Burst Buffer on Cori. This page will be updated as problems are discovered, and as they are fixed.
### General Issues
* Do not use a decimal point when you specify the burst buffer capacity - slurm does not parse this correctly and will allocate you one grain of space instead of the full request. This is easy to work around - request 3500GB instead of 3.5TB, etc.
* Data is at risk in a Persistent Reservation if an SSD fails - there is no possibility to recover data from a dead SSD. Please back up your data!
* If you request a too-small allocation on the Burst Buffer (e.g. request 200GB and actually write out 300GB) your job will fail, and the BB node will go into an undesirable state and need to be recovered manually. Please be careful of how much space your job requires - if in doubt, over-request.
* If you use "dwstat" in your batch job, you may occasionally run into "[SSL: CERTIFICATE_VERIFY_FAILED]" errors, which may fail your job. If you see this error, it is due to a modulefile issue - please use the full path to the dwstat command: "/opt/cray/dws/default/bin/dwstat".
* If you have multiple jobs writing to the same directory in a persistent reservation, you will run into race conditions due to the DataWarp caching. The second job will likely fail with "permission denied" or "No such file or directory" messages, as the metadata in the compute node cache does not match the reality of the metadata on the BB.
* If the primary SLURM controller is down, the secondary (backup) controller will be scheduling jobs - and the secondary controller does not know about the Burst Buffer. If you happen to submit a job when the backup scheduler is running your jobs will fail with the message "sbatch: error: burst_buffer/cray: Slurm burst buffer configuration error / sbatch: error: Batch job submission failed: Burst Buffer request invalid". If you receive this error and your submission script is correct, please check [MOTD](https://www.nersc.gov/live-status/motd/) for SLURM downtime/issues, and try again later.
### Staging Issues
There are a number of known issues to be aware of when using the Burst
Buffer on Cori. This page will be updated as problems are discovered,
and as they are fixed.
* The command "squeue -l -u username" will give you useful information on how your stage_in process is going. If you see an error message (e.g. "(burst_buffer/cray: dws_data_in: Error staging in session 20868 configuration 6953 path /global/cscratch1/sd/username/stagemein.txt -> /stagemein.txt: offline namespaces: [44223] - ask a system administrator to consult the dwmd log for more information") then you may have a permissions issue with the files you are trying to stage_in, or be trying to stage_in a non-existent file.
### General Issues
* The Burst Buffer cannot access GPFS for staging data (copying data is fine). If you have data that will be staged in to the BB, you will need to have those files in $SCRATCH. Data in your home or project directories can be transferred using "cp" within your compute job.
* stage_in and stage_out using access_mode=private does not work (by design).
* Do not use a decimal point when you specify the burst buffer
capacity - slurm does not parse this correctly and will allocate you
one grain of space instead of the full request. This is easy to work
around - request 3500GB instead of 3.5TB, etc.
* Data is at risk in a Persistent Reservation if an SSD fails - there
is no possibility to recover data from a dead SSD. Please back up
your data!
* If you request a too-small allocation on the Burst Buffer
(e.g. request 200GB and actually write out 300GB) your job will
fail, and the BB node will go into an undesirable state and need to
be recovered manually. Please be careful of how much space your job
requires - if in doubt, over-request.
* If you use "dwstat" in your batch job, you may occasionally run into
"[SSL: CERTIFICATE_VERIFY_FAILED]" errors, which may fail your
job. If you see this error, it is due to a modulefile issue - please
use the full path to the dwstat command:
"/opt/cray/dws/default/bin/dwstat".
* If you have multiple jobs writing to the same directory in a
persistent reservation, you will run into race conditions due to the
DataWarp caching. The second job will likely fail with "permission
denied" or "No such file or directory" messages, as the metadata in
the compute node cache does not match the reality of the metadata on
the BB.
* If the primary SLURM controller is down, the secondary (backup)
controller will be scheduling jobs - and the secondary controller
does not know about the Burst Buffer. If you happen to submit a job
when the backup scheduler is running your jobs will fail with the
message "sbatch: error: burst_buffer/cray: Slurm burst buffer
configuration error / sbatch: error: Batch job submission failed:
Burst Buffer request invalid". If you receive this error and your
submission script is correct, please check
[MOTD](https://www.nersc.gov/live-status/motd/) for SLURM
downtime/issues, and try again later.
### Staging Issues
* If you have multiple files to stage in, you will need to tar them up and use type=file, or keep them all in one directory and use type=directory.
* type=directory fails with large directories (>~200,000 files) due to a timeout error. In this case, consider tar-ing your files and staging in the tarball.
* Symbolic links are not preserved when staging in, the link will be lost.
* The command "squeue -l -u username" will give you useful information
on how your stage_in process is going. If you see an error message
(e.g. "(burst_buffer/cray: dws_data_in: Error staging in session
20868 configuration 6953 path
/global/cscratch1/sd/username/stagemein.txt -> /stagemein.txt:
offline namespaces: [44223] - ask a system administrator to consult
the dwmd log for more information") then you may have a permissions
issue with the files you are trying to stage_in, or be trying to
stage_in a non-existent file.
* The Burst Buffer cannot access GPFS for staging data (copying data
is fine). If you have data that will be staged in to the BB, you
will need to have those files in $SCRATCH. Data in your home or
Community directories can be transferred using "cp" within your
compute job.
* stage_in and stage_out using access_mode=private does not work (by
design).
* If you have multiple files to stage in, you will need to tar them up
and use type=file, or keep them all in one directory and use
type=directory.
* type=directory fails with large directories (>~200,000 files) due to
a timeout error. In this case, consider tar-ing your files and
staging in the tarball.
* Symbolic links are not preserved when staging in, the link will be
lost.
* Staging in/out with hard links does not work.
......@@ -66,9 +66,6 @@ Here, we list additional support resources for NERSC users, as well as pointers
### User support at NERSC
+ [Consulting services](../../help) provided by the NERSC User Services Group
### Resource requests
+ Quota increases (space or inodes) for NGF project and global scratch, as well as the Lustre local scratch filesystems, may be submitted [here](https://www.nersc.gov/users/storage-and-file-systems/file-systems/data-storage-quota-increase-request/)
+ New project directory requests may be submitted as a ticket by PIs and PI proxies
### Previous and ongoing I/O research projects contributed to by NERSC and LBL researchers
+ The [ExaHDF5](https://sdm.lbl.gov/exahdf5/) group is working to develop next-generation I/O libraries and middleware to support scientific I/O (focused in particular on the HDF5 data format)
......
......@@ -9,7 +9,7 @@ Never use the version of Python found at /usr/bin/python, it is an older version
## Select the Right File System for Installing Your Own Packages
* If you mostly run serial Python jobs or use multiprocessing on a single node, you might be able to just install Python packages in your $HOME directory or on the /project file system without seeing any substantial performance issues.
* If you mostly run serial Python jobs or use multiprocessing on a single node, you might be able to just install Python packages in your $HOME directory or on the Community file system without seeing any substantial performance issues.
* The best-performing shared file system for launching parallel Python applications is /global/common.
This file system is mounted read-only on Cori compute nodes with client-side caching enabled.
This is the file system that NERSC uses to install software modules (such as Python).
......
......@@ -68,16 +68,16 @@ be almost as good as Shifter, but Shifter is always the best choice. Users can
also install software to this read-optimized file system. Read more about how
to do that [here.](../../../../filesystems/global-common)