How to compose Ollama server with Nextjs?

Question

I have been attempting to create a composition involving the Ollama server with LLMs such as Mistral and Llama2, along with a Next.js server to interact with it. I am building this project to learn Docker, and I am currently facing an issue.

So I have created a docker file for Nextjs like this :

FROM node:18-alpine

WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install

COPY . .
RUN npm run build
CMD ["npm", "start"]    

Along with this I seek to run a llama server with cached dependencies. For this I tried to create a docker-compose file :

version: '3.8'
services:
  ollama:
    image: ollama/ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama:/root/.ollama
    command: ollama pull llama2

  ollama-ui:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "3000:3000"

This way I am trying to run a command called ollama pull llama2, intending for llama2 to be pulled before it starts Nextjs server. But there is an error :

ollama-1 | Error: unknown command "ollama" for "ollama".

Goal of this project is a composed container running ollama server with some LLMs running along with my Nextjs server and I want the process of pulling LLMs to be cached. Is this a proper approach? If not what should it be?

I am sure this is some silly problem but will be glad if someone schools me here.

Thanks!

Answer 1

I have faced the same problem and found that despite being in /bin, compose can't find the ollama command while running the image. One hack I've found around this is to run a script after docker compose is running to pull llama2.

#!/usr/bin/bash

curl -X POST \
-H "Content-Type: application/json" \
-d '{"name":"llama2"}' \
 http://localhost:11434/api/pull

Answer 2

In case anyone is still looking for a better solution, the issue is that the docker image's entrypoint is already the ollama command, so you can just directly do pull llama2 without the ollama bit.

EDIT: Unfortunately this causes a different issue, because docker-compose doesn't easily let you start the server and then run the pull command, so the API approach is still the best for actually achieving the desired goal.