What should be the padding token for a Huggingface model?

13 views Asked by At

Should it be the eos token or the PAD token? I got this error message:

ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as pad_token (tokenizer.pad_token = tokenizer.eos_token e.g.) or add a new pad token via tokenizer.add_special_tokens({'pad_token': '[PAD]'}).

So which of those tokens should I use?

0

There are 0 answers