benchmarks.md 2.56 KB
Newer Older
Steven Farrell's avatar
Steven Farrell committed
1 2 3 4 5 6 7 8 9 10 11 12 13 14
# Machine Learning benchmarking at NERSC

NERSC uses both standard framework-oriented benchmarks as well as scientific
benchmarks from research projects in order to characterize our systems
for scientific Deep Learning.

## Framework benchmarks

### TensorFlow

We run a version of the tf_cnn_benchmarks repository as well as a DCGAN
model on Cori.

**Training results**
Mustafa Mustafa's avatar
Mustafa Mustafa committed
15 16
![Training results](images/TF_benchmark.png)
![Training results](images/TF_benchmark_rltv.png)
Steven Farrell's avatar
Steven Farrell committed
17 18 19 20 21 22 23

### PyTorch

We have a repository of benchmarks with standard computer vision models,
LSTM, and 3D convolutional models here:
https://github.com/sparticlesteve/pytorch-benchmarks

Steven Farrell's avatar
Steven Farrell committed
24 25 26 27
We compare PyTorch software installations, hardware, and analyze scaling
performance using the PyTorch distributed library with MPI. See the notebooks
in the links below for numbers and plots.

28
#### Software versions
Steven Farrell's avatar
Steven Farrell committed
29 30 31 32 33 34 35

Results for a handful of software versions available on Cori are in this
notebook:

https://github.com/sparticlesteve/pytorch-benchmarks/blob/master/notebooks/SoftwareAnalysis.ipynb

Training throughput results:
Steven Farrell's avatar
Steven Farrell committed
36 37
![Training results](images/pytorch_training_benchmarks.png)

38
#### Hardware comparisons
Steven Farrell's avatar
Steven Farrell committed
39 40 41 42 43

Results comparing training throughput on Cori Haswell, KNL, and GPU are here:

https://github.com/sparticlesteve/pytorch-benchmarks/blob/master/notebooks/HardwareAnalysis.ipynb

44
#### Scaling analysis
Steven Farrell's avatar
Steven Farrell committed
45 46 47 48 49

Throughput scaling results on Cori Haswell with Intel PyTorch v1.0.0 are
available here:

https://github.com/sparticlesteve/pytorch-benchmarks/blob/master/notebooks/ScalingAnalysis.ipynb
Steven Farrell's avatar
Steven Farrell committed
50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70

## Scientific Deep Learning Benchmarks

### HEP-CNN

The HEP-CNN benchmark trains a simple Convolutional Neural Network to
classify LHC collision detector images as signal or background.

* Framework: TensorFlow
* Multi-node library: Horovod or Cray PE ML Plugin
* Papers: https://arxiv.org/abs/1711.03573, https://arxiv.org/abs/1708.05256
* Code: https://github.com/sparticlesteve/hep_cnn_benchmark/tree/benchmark-dev

### CosmoFlow

The CosmoFlow benchmark trains a 3D Convolutional Neural Network to predict
cosmological parameters from simulated universe volumes.

* Framework: TensorFlow
* Multi-node library: Cray PE ML Plugin
* Paper: https://arxiv.org/abs/1808.04728
71
* Code: https://github.com/sparticlesteve/cosmoflow-benchmark
Steven Farrell's avatar
Steven Farrell committed
72 73 74 75 76 77 78

### CosmoGAN

* Framework: TensorFlow
* Paper: https://arxiv.org/abs/1706.02390
* Code: https://github.com/MustafaMustafa/cosmoGAN

79
### Deep Learning Climate Segmentation
Steven Farrell's avatar
Steven Farrell committed
80 81 82 83

* Framework: TensorFlow
* Multi-node library: Horovod
* Paper: https://arxiv.org/abs/1810.01993
84
* Code: https://github.com/sparticlesteve/climate-seg-benchmark