-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Description
processor = Blip2Processor.from_pretrained("/home/flyvideo/Sata2/pb/LAVIS/models/blip2-flan-t5-xl")
在我运行blip2-flan-t5-xl模型时,会提示分词器错误,但模型文件中的tokenizer.json文件是完好的会,这是什么原因?
File "/home/flyvideo/Sata2/anaconda3/envs/blip2/lib/python3.9/site-packages/transformers/processing_utils.py", line 226, in from_pretrained
args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, **kwargs)
File "/home/flyvideo/Sata2/anaconda3/envs/blip2/lib/python3.9/site-packages/transformers/processing_utils.py", line 270, in _get_arguments_from_pretrained
args.append(attribute_class.from_pretrained(pretrained_model_name_or_path, **kwargs))
File "/home/flyvideo/Sata2/anaconda3/envs/blip2/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py", line 736, in from_pretrained
return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
File "/home/flyvideo/Sata2/anaconda3/envs/blip2/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 1854, in from_pretrained
return cls._from_pretrained(
File "/home/flyvideo/Sata2/anaconda3/envs/blip2/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 2017, in _from_pretrained
tokenizer = cls(*init_inputs, **init_kwargs)
File "/home/flyvideo/Sata2/anaconda3/envs/blip2/lib/python3.9/site-packages/transformers/models/gpt2/tokenization_gpt2_fast.py", line 140, in init
super().init(
File "/home/flyvideo/Sata2/anaconda3/envs/blip2/lib/python3.9/site-packages/transformers/tokenization_utils_fast.py", line 110, in init
fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
Exception: data did not match any variant of untagged enum PyPreTokenizerTypeWrapper at line 964 column 3