torchserve model not running and giving a load of errors

Question

I ran the following commands
torch-model-archiver --model-name "bert" --version 1.0 --serialized-file ./bert_model/pytorch_model.bin --extra-files "./bert_model/config.json,./bert_model/vocab.txt" --handler "./handler.py"

I created all the files and then I created a new directory and copied the model into it.

Then I executed the following command:

torchserve --start --model-store model_store --models bert=bert.mar

It then displayed a slew of errors.

Here is my error text. It is too long and repetitive; hence, I posted it on paste bin.

error

Answer 1

I would suggest lowering down the number of workers per model (Default workers per model: 12) now you get the maximum number that your can handle

How? Go to config.properties file and add (the first line indicates the workers to 2):

 default_workers_per_model=2

Then when you will do the torchserve add this (ts-config option to point on the location of you config.properties file):

torchserve  --start \
            --model-store  ./deployment/model-store \
            --ts-config ./deployment/config.properties \
            --models bert=bert.mar

Let me know if this solves the error.

Note : you can add other parameters as well in the config.properties file such as :

inference_address=http://0.0.0.0:8080
management_address=http://0.0.0.0:8081
metrics_address=http://0.0.0.0:8082
default_workers_per_model=2
number_of_netty_threads=1
netty_client_threads=1
prefer_direct_buffer=true