Apache License

Version 2.0, January 2004

http://www.apache.org/licenses/

END OF TERMS AND CONDITIONS

APPENDIX: How to apply the Apache License to your work.

To apply the Apache License to your work, attach the following

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

[LAMB](https://arxiv.org/pdf/1904.00962.pdf) stands for Layerwise Adaptive Moments based optimizer, is a large batch optimization technique that helps accelerate training of deep neural networks using large minibatches. It allows using a global batch size of 65536 and 32768 on sequence lengths 128 and 512 respectively, compared to a batch size of 256 for Adam. The optimized implementation accumulates 1024 gradients batches in phase 1 and 4096 steps in phase 2 before updating weights once. This results in 15% training speedup. On multi-node systems, LAMB allows scaling up to 1024 GPUs resulting in training speedups of up to 72x in comparison to [Adam](https://arxiv.org/pdf/1412.6980.pdf). Adam has limitations on the learning rate that can be used since it is applied globally on all parameters whereas LAMB follows a layerwise learning rate strategy.

Mixed precision is the combined use of different numerical precisions in a computational method. [Mixed precision](https://arxiv.org/abs/1710.03740) training offers significant computational speedup by performing operations in half-precision format, while storing minimal information in single-precision to retain as much information as possible in critical parts of the network. Since the introduction of [Tensor Cores](https://developer.nvidia.com/tensor-cores) in the Volta and Turing architecture, significant training speedups are experienced by switching to mixed precision -- up to 3x overall speedup on the most arithmetically intense model architectures. Using mixed precision training requires two steps:

To download, verify, extract the datasets, and create the shards in hdf5 format, run:

6. Start pretraining.

- `<seed>` random seed for the run.

- `<allreduce_post_accumulation>` - If set to `true`, performs allreduce only after the defined number of gradient accumulation steps.

- `<allreduce_post_accumulation_fp16>` - If set to `true`, performs allreduce after gradient accumulation steps in FP16.

- `<training_batch_size_phase2>` is per-GPU batch size used for training in phase 2. Larger batch sizes run more efficiently, but require more memory.

- `<learning_rate_phase2>` is the base learning rate for training phase 2.

- `<warmup_proportion_phase2>` is the percentage of training steps used for warm-up at the start of training.

- `<training_steps_phase2>` is the total number of training steps for phase 2, to be continued in addition to phase 1.

- `<gradient_accumulation_steps_phase2>` an integer indicating the number of steps to accumulate gradients over in phase 2. Effective batch size = `training_batch_size_phase2` / `gradient_accumulation_steps_phase2`.

For example:

Trains BERT-large from scratch on a DGX-1 32G using FP16 arithmetic. 90% of the training steps are done with sequence length 128 (phase1 of training) and 10% of the training steps are done with sequence length 512 (phase2 of training).

|8 | 4 | 8| 512| 68.16| 247.04| 3.62| 7.57| 7.64

|16 | 4 | 8| 512| 135.68| 488.96| 3.60| 15.08| 15.13

Note: Multi-node performance numbers below are on DGX-2H whereas the single node performance numbers above are on DGX-2.

Our results were obtained by running the `scripts/run_pretraining_inference.sh` and `scripts/run_squad.sh` scripts in the pytorch:19.07-py3 NGC container on NVIDIA DGX-1 with (1x V100 32G) GPUs.

Our results were obtained by running the `scripts/run_pretraining_inference.sh` and `scripts/run_squad.sh` scripts in the pytorch:19.07-py3 NGC container on NVIDIA DGX-2 with (1x V100 32G) GPUs.

To achieve these same results, follow the steps in the [Quick Start Guide](#quick-start-guide).

September 2019

- Scripts to support multi-node launch

- Update pretraining loss results based on the latest data preparation scripts

bert_preprocessing_command += ' --random_seed=' + str(args.random_seed)

bert_preprocessing_command += ' --dupe_factor=' + str(args.dupe_factor)

bert_preprocessing_process = subprocess.Popen(bert_preprocessing_command, shell=True)

last_process = bert_preprocessing_process

bert_preprocessing_command += ' --random_seed=' + str(args.random_seed)

bert_preprocessing_command += ' --dupe_factor=' + str(args.dupe_factor)

bert_preprocessing_process = subprocess.Popen(bert_preprocessing_command, shell=True)

last_process = bert_preprocessing_process