Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings
Discussion options

Is it possible the train only the punctuation model for a different language, i found out how to finetune only for the ASR models. If it's possible what trainset structure do i need and how to modify the training/finetuning to work for the punctuation model?

You must be logged in to vote

Yes, the CT-Transformer can be trained as a punctuation-only model; it does not need audio. It is a token-classification task: for every input text token, provide exactly one label describing the punctuation after that token.

There is an important version caveat, though: current FunASR main does not ship a complete punctuation training recipe. The historical punc_train.py / PunctuationTask path was removed during the 1.0 refactor, and the current registered datasets do not produce the punc and punc_lengths fields required by CTTransformer.forward. Therefore, changing the model name in an ASR fine-tuning command is not sufficient, and I would not recommend copying an old v0.x command into…

Replies: 1 comment

Comment options

Yes, the CT-Transformer can be trained as a punctuation-only model; it does not need audio. It is a token-classification task: for every input text token, provide exactly one label describing the punctuation after that token.

There is an important version caveat, though: current FunASR main does not ship a complete punctuation training recipe. The historical punc_train.py / PunctuationTask path was removed during the 1.0 refactor, and the current registered datasets do not produce the punc and punc_lengths fields required by CTTransformer.forward. Therefore, changing the model name in an ASR fine-tuning command is not sufficient, and I would not recommend copying an old v0.x command into a current environment.

Data contract

The historical trainer used two Kaldi-style, two-column files with identical utterance IDs. For a character tokenizer:

# text
utt1 hello
utt2 hi

# punc
utt1 ____,
utt2 _?

The aligned labels mean:

h  e  l  l  o
_  _  _  _  ,

_ means "no punctuation". The number of punctuation labels must equal the number of tokens after tokenization. With the historical char tokenizer, do not write _ _ _ _ ,: those spaces are also tokenized and corrupt the alignment. I verified the loader contract and a real forward/backward/optimizer step on the last pre-removal implementation; the required field names were exactly text and punc (historical source).

The historical punctuation vocabulary file was:

<unk>:1.0
_:1.0
,:1.0
.:1.0
?:1.0

The value after : is the class weight. You can replace the punctuation symbols with the classes needed by your language.

Adapting the 471,067-token checkpoint

iic/punc_ct-transformer_cn-en-common-vocab471067-large is tied to its original tokenizer and 471,067-row input embedding.

  • If your target-language tokenizer only emits tokens already present in that vocabulary, you can keep the embedding and fine-tune the model.
  • If the target script has poor vocabulary coverage and maps heavily to <unk>, create a new token list. That changes vocab_size, so the old embedding cannot be loaded unchanged; initialize the new embedding and reuse only compatible encoder weights.
  • If you change the punctuation classes, the decoder output dimension changes, so reinitialize the decoder as well.

Before training, assert this for every sample:

assert len(text_ids) == len(punc_ids)

For current FunASR, the practical route is a small custom dataset/collator that returns text, text_lengths, punc, and punc_lengths, then calls CTTransformer directly. I also verified one current-main issue: the YAML-style punc_weight list reaches .to() during training and raises AttributeError; passing it as a torch.Tensor makes an aligned mini-batch train correctly. So the model architecture is trainable, but the stock current training path is not yet an end-to-end supported recipe.

If you post the target language, tokenizer choice (character/word/subword), and desired punctuation set, we can provide an exact token-alignment example and determine how much of the Chinese/English checkpoint can be reused.

You must be logged in to vote
0 replies
Answer selected by LauraGPT
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
🙏
Q&A
Labels
None yet
2 participants
Morty Proxy This is a proxified and sanitized view of the page, visit original site.