How can I deploy a BOSH Director on BOSH Lite

Question

I'm currently trying to install a BOSH Director on BOSH Lite - it's clear to me that BOSH Lite already ships with a Director, but I would like to test a release containing a Director "on top of that". Here is my setup:

Everything works fine until I add the warden_cpi job. I would like to configure the Warden CPI to connect to Warden running on the Virtual Machine hosting BOSH Lite and still being available to the Director . So what I tried is this:

releases:
- name: bosh-warden-cpi
  url: https://bosh.io/d/github.com/cppforlife/bosh-warden-cpi-release?v=29
  sha1: 9cc293351744f3892d4a79479cccd3c3b2cf33c7
  version: latest

instance_groups:
- name: bosh-components
  jobs:
  - name: warden_cpi
    release: bosh-warden-cpi
    properties:
      warden_cpi:
        host_ip: 10.254.50.4  # host IP of BOSH Lite Vagrant Box
        warden:
          connect_network: tcp
          connect_address: 10.254.50.4:7777 # again host IP and Port of garden-linux on BOSH Lite Vagrant Box
        agent:
          mbus: nats://user:[email protected]:4222 
          blobstore:
            provider: dav
            options:
              endpoint: http://127.0.0.1:25250
              user: user
              password: password

where 10.254.50.4 is the IP address of the Vagrant Box and 7777 is the port of garden-linux.

During the deployment, I get this message from bosh vms

+----------------------------------------------------------+--------------------+-----+---------+--------------+
| VM                                                       | State              | AZ  | VM Type | IPs          |
+----------------------------------------------------------+--------------------+-----+---------+--------------+
| bosh-components/0 (37a1938e-e1df-4650-bec6-460e4bc3916e) | unresponsive agent | n/a | small   |              |
| bosh-director/0 (2bb47ce1-0bba-49aa-b9a3-86e881e91ee9)   | running            | n/a | small   | 10.244.102.2 |
| jumpbox/0 (51c895ae-3563-4561-ba3f-d0174e90c3f4)         | running            | n/a | small   | 10.244.102.4 |
+----------------------------------------------------------+--------------------+-----+---------+--------------+

As an error message from bosh deploy, I get this:

Error 450002: Timed out sending `get_state' to e1ed3839-ade4-4e12-8f33-6ee6000750d0 after 45 seconds

After the error occurs, I can see the VM with bosh vms:

+----------------------------------------------------------+---------+-----+---------+--------------+
| VM                                                       | State   | AZ  | VM Type | IPs          |
+----------------------------------------------------------+---------+-----+---------+--------------+
| bosh-components/0 (37a1938e-e1df-4650-bec6-460e4bc3916e) | running | n/a | small   | 10.244.102.3 |
| bosh-director/0 (2bb47ce1-0bba-49aa-b9a3-86e881e91ee9)   | failing | n/a | small   | 10.244.102.2 |
| jumpbox/0 (51c895ae-3563-4561-ba3f-d0174e90c3f4)         | running | n/a | small   | 10.244.102.4 |
+----------------------------------------------------------+---------+-----+---------+--------------+

But when I ssh into the bosh-components VM, there are no jobs in /var/vcap/jobs.

When I remove the warden_cpi block from the jobs list, everything runs as expected. The full jobs list for my BOSH components VM:

• nats
• postgres
• registry
• blobstore

The Director itself runs on another machine. Without the Warden CPI the two machines can communicate as expected.

Can anybody point out to me how I have to configure the Warden CPI so that it connects to the Vagrant Box as expected?

Answer 1

The question is very old, it's a BOSH v1 CLI whereas now BOSH v2 is an established standard, Garden Linux had been deprecated a long time ago in favor of Garden runC, but still, having experimented a lot with BOSH-Lite, I'd like to answer this one.

First, a semantics remark: you shouldn't say “on top of that”, but “as instructed by” instead, because a BOSH Director just instructs some underlying (API-based) infrastructure to do something, that eventually makes it run some workloads.

Second, there are two hurdles you might hit here:

1. The main problem is that the Warden CPI talks to both the Garden backend and the local Linux kernel for setting up various things around those Garden containers. As a direct consequence, you cannot run a Warden CPI inside a BOSH-Lite container.

2. The filesystem used (here by the long-gone Garden Linux, but nowadays the issue would be similar with Garden runC) might not work inside a Garden container, as managed by the pre-existing Warden CPI.

All in all, then main thing to be aware of, is this idea that the Warden CPI not only talks to the Garden backend through some its REST API. More than that, the Warden CPI needs to be co-located with the Linux kernel that runs Garden, in order to make system calls and run local commands for mounting persistent storage and other things.