segmentation fault running train-text-from-scratch as described into the documentation #4227
Description
Prerequisites
build as described on documentation, latest release => b1567
Please answer the following questions for yourself before submitting an issue.
- I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- I carefully followed the README.md.
- [ x I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
- I reviewed the Discussions, and have a new bug or useful enhancement to share.
Current Behavior
doing this:
wget https://raw.githubusercontent.com/brunoklein99/deep-learning-notes/master/shakespeare.txt
# train
./bin/train-text-from-scratch \
--vocab-model ../models/ggml-vocab-llama.gguf \
--ctx 64 --embd 256 --head 8 --layer 16 \
--checkpoint-in chk-shakespeare-256x16-LATEST.gguf \
--checkpoint-out chk-shakespeare-256x16-ITERATION.gguf \
--model-out ggml-shakespeare-256x16-f32-ITERATION.gguf \
--train-data "shakespeare.txt" \
-t 6 -b 16 --seed 1 --adam-iter 256 \
--no-checkpointing
Please provide a detailed written description of what llama.cpp did, instead.
save_checkpoint_file: saving to chk-shakespeare-256x16-250.gguf
save_checkpoint_file: saving to chk-shakespeare-256x16-LATEST.gguf
save_llama_model_file: saving to ggml-shakespeare-256x16-f32-250.gguf
save_llama_model_file: saving to ggml-shakespeare-256x16-f32-LATEST.gguf
train_opt_callback: iter= 250 sample=4001/27520 sched=0.868198 loss=10.343768 dt=515.2ms eta=00:00:03 |->
train_opt_callback: iter= 251 sample=4017/27520 sched=0.867197 loss=10.342506 dt=509.5ms eta=00:00:02 |->
train_opt_callback: iter= 252 sample=4033/27520 sched=0.866192 loss=10.343074 dt=516.1ms eta=00:00:02 |->
train_opt_callback: iter= 253 sample=4049/27520 sched=0.865185 loss=10.343571 dt=513.9ms eta=00:00:01 |->
train_opt_callback: iter= 254 sample=4065/27520 sched=0.864174 loss=10.342989 dt=515.4ms eta=00:00:01 |->
train_opt_callback: iter= 255 sample=4081/27520 sched=0.863161 loss=10.343344 dt=516.5ms eta=516.5ms |->
train_opt_callback: iter= 256 sample=4097/27520 sched=0.862144 loss=10.344069 dt=541.4ms eta=0.0ms |->
main: total training time: 00:02:20
save_checkpoint_file: saving to chk-shakespeare-256x16-256.gguf
save_checkpoint_file: saving to chk-shakespeare-256x16-LATEST.gguf
save_llama_model_file: saving to ggml-shakespeare-256x16-f32-256.gguf
save_llama_model_file: saving to ggml-shakespeare-256x16-f32-LATEST.gguf
[1] 95248 segmentation fault ./bin/train-text-from-scratch --vocab-model ../models/ggml-vocab-llama.gguf
-
Operating System, e.g. for Linux:
Darwin PC100406 23.0.0 Darwin Kernel Version 23.0.0: Fri Sep 15 14:41:43 PDT 2023; root:xnu-10002.1.13~1/RELEASE_ARM64_T6000 arm64 -
SDK version, e.g. for Linux:
$ python3 --version => Python 3.11.6
$ make --version
GNU Make 3.81
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
This program built for i386-apple-darwin11.3.0
$ g++ --version
Apple clang version 15.0.0 (clang-1500.0.40.1)
Target: arm64-apple-darwin23.0.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin