I am running into issues of evaluating huggingface's BERT model ('bert-base-uncased') on large input sequences.
model = BertModel.from_pretrained('bert-base-uncased', output_hidden_states=True)
token_ids = [101, 1014, 1016, ...] # len(token_ids) == 33286
token_tensors = torch.tensor([token_ids]) # shape == [1, 33286]
segment_tensors = torch.tensor([[1] * len(token_ids)]) # shape == [1, 33286]
model(token_tensors, segment_tensors)
Traceback
self.model(token_tensors, segment_tensors)
File "/home/.../python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/.../python3.8/site-packages/transformers/modeling_bert.py", line 824, in forward
embedding_output = self.embeddings(
File "/home/.../python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/.../python3.8/site-packages/transformers/modeling_bert.py", line 211, in forward
embeddings = inputs_embeds + position_embeddings + token_type_embeddings
RuntimeError: The size of tensor a (33286) must match the size of tensor b (512) at non-singleton dimension 1
I noticed that model.embeddings.positional_embeddings.weight.shape == (512, 768)
. I.e. when I restrict the input size to model(token_tensors[:, :10], segment_tensors[:, :10])
it works. I am misunderstanding how the the token_tensors
and segment_tensors
should be shaped. I thought they should be sized (batch_size, sequence_length)
Thanks for the help
I just discovered that pretrained BERT models from huggingface have a maximum input length of 512 ( https://github.com/huggingface/transformers/issues/225 )