What padding values should be used for huggingface tokenizers?

20 views Asked by At

I am using Mbart50 to convert Nepalese to English and am not sure what values I should use for padding. I need to figure out the padding values for both English and Nepalese. My tokenizer code:

tokenizer = MBart50TokenizerFast.from_pretrained(model_name)
tokenizer.src_lang = "ne_NP"
tokenizer.tgt_lang = "en_XX" 
0

There are 0 answers