Torchserve Error: number of batch response mismatched

159 views Asked by At

We deployed NER Model with n1-standard-8 machine without GPU with below config properties. when we kept batch size as 1, it is taking more time to process the simultaneous requests. when we try to increase the batch size, we are getting below error. (we tried with different batch size like 16,32,64,8 etc and max workers as 1 and 8)

Response: response_data: {'code': 503, 'type': 'InternalServerException', 'message': 'number of batch response mismatched'}

I am using below Torchserve configuration

inference_address=http://0.0.0.0:8080
management_address=http://0.0.0.0:8081
metrics_address=http://0.0.0.0:8082
install_py_dep_per_model=true
prefer_direct_buffer=true
job_queue_size=10000
async_logging=true
number_of_netty_threads=8
netty_client_threads=8
default_workers_per_model=1
models={\
  "model": {\
    "1.0": {\
        "defaultVersion": true,\
        "marName": "legal_description.mar",\
        "minWorkers": 1,\
        "maxWorkers": 8,\
        "batchSize": 16,\
        "maxBatchDelay": 65,\
        "responseTimeout": 100\
    }\
  }\
}

I want to process multiple threads simultaneously. Please suggest solution. Do I need to change handler script, if yes, how?

0

There are 0 answers