How to bind config items in an array to environment variables

Question

Here below is my configuration file in toml format.

[[hosts]]
name = "host1"
username = "user1"
password = "password1"

[[hosts]]
name = "host2"
username = "user2"
password = "password2"

... and here is my code to load it:

import (
    "fmt"
    "github.com/spf13/viper"
    "strings"
)

type Config struct {
    Hosts []Host
}

type Host struct {
    Name      string `mapstructure:"name"`
    Username  string `mapstructure:"username"`
    Password  string `mapstructure:"password"`
}

func main() {
    viper.AddConfigPath(".")
    viper.AddConfigPath("./config")
    viper.SetConfigName("app")

    if err := viper.ReadInConfig(); err != nil {
        return nil, fmt.Errorf("error reading config file, %v", err)
    }

    config := new(Config)
    if err := viper.Unmarshal(config); err != nil {
        return nil, fmt.Errorf("error parsing config file, %v", err)
    }

    var username, password string

    for i, h := range config.Hosts {
        if len(h.Name) == 0 {
            return nil, fmt.Errorf("name not defined for host %d", i)
        }

        if username = os.Getenv(strings.ToUpper(h.Name) + "_" + "USERNAME"); len(username) > 0 {
            config.Hosts[i].Username = username
        } else if len(h.Username) == 0 {
            return nil, fmt.Errorf("username not defined for %s", e.Name)
        }

        if password = os.Getenv(strings.ToUpper(e.Name) + "_" + "PASSWORD"); len(password) > 0 {
            config.Hosts[i].Password = password
        } else if len(h.Password) == 0 {
            return nil, fmt.Errorf("password not defined for %s", e.Name)
        }

        fmt.Printf("Hostname: %s", h.name)
        fmt.Printf("Username: %s", h.Username)
        fmt.Printf("Password: %s", h.Password)
    }
}

For instance, I first check whether environment variables HOST1_USERNAME1, HOST1_PASSWORD1, HOST2_USERNAME2, and HOST2_PASSWORD2 exist... if they do, then I set the configuration items to their values, otherwise I try to get the values from the configuration file.

I know viper offers method AutomaticEnv to do something similar... but does it work with a collection like mine (AutomaticEnv shall be invoked after environment variable binding)?

Given my code above, is it possible to use the mechanism provided by viper and remove os.GetEnv?

Thanks.

UPDATE

Here below is my updated code. In I've defined environment variables HOST1_USERNAME and HOST1_PASSWORD and set the corresponding settings in my config file to an empty string.

Here is my new config file:

[host1]
username = ""
password = ""

[host2]
username = "user2"
password = "password2"

And here is my code:

package config

import (
    "fmt"
    "github.com/spf13/viper"
    "strings"
    "sync"
)

type Config struct {
    Hosts []Host
}

type Host struct {
    Name     string
    Username string
    Password string
}

var config *Config

func (c *Config) Load() (*Config, error) {
    if config == nil {
        viper.AddConfigPath(".")
        viper.AddConfigPath("./config")
        viper.SetConfigName("myapp")
        viper.AutomaticEnv()
        viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))

        if err := viper.ReadInConfig(); err != nil {
            return nil, fmt.Errorf("error reading config file, %v", err)
        }

        allSettings := viper.AllSettings()
        hosts := make([]Host, 0, len(allSettings))

        for key, value := range allSettings {
            val := value.(map[string]interface{})

            if val["username"] == nil {
                return nil, fmt.Errorf("username not defined for host %s", key)
            }

            if val["password"] == nil {
                return nil, fmt.Errorf("password not defined for host %s", key)
            }

            hosts = append(hosts, Host{
                Name:      key,
                Unsername: val["username"].(string),
                Password: val["password"].(string),
            })
        }

        config = &Config{hosts}
    }

    return config, nil
}

I works now (thanks to Chrono Kitsune) and I hope it helps, j3d

Answer 1

From viper.Viper:

The priority of the sources is the following:

1. overrides
2. flags
3. env. variables
4. config file
5. key/value store
6. defaults

You might encounter a problem in determining the name of the environment variable. You essentially need a way to bind hosts[0].Username to the environment variable HOST1_USERNAME. However, there's no way to do this in viper currently. In fact, viper.Get("hosts[0].username") returns nil, meaning array indices apparently cannot be used with viper.BindEnv. You also would need to use this function for as many hosts as can be defined, meaning if you have 20 hosts listed, you'd need to call viper.BindEnv 40 or 60 times, depending on whether the name of a host could be overridden by an environment variable. To work around this limitation, you'd need to dynamically work with hosts as independent tables rather than an array of tables:

[host1]
username = "user1"
password = "password1"

[host2]
username = "user2"
password = "password2"

You can then use viper.SetEnvKeyReplacer with a strings.Replacer to handle the environment variable issue:

// host1.username => HOST1_USERNAME
// host2.password => HOST2_PASSWORD
// etc.
viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))

Note that at the time of this writing some bugs exist when it comes to the resolution order. This issue affects viper.Unmarshal and viper.Get: environment variables should override config file values, but currently the config file values are still used. Strangely, viper.AllSettings works fine. If it didn't, you couldn't do something like the following to work with the hosts in the above format:

// Manually collect hosts for storing in config.
func collectHosts() []Host {
    hosts := make([]Host, 0, 10)
    for key, value := range viper.AllSettings() {
        // viper.GetStringMapString(key)
        // won't work properly with environment vars, etc. until
        //   https://github.com/spf13/viper/issues/309
        // is resolved.
        v := value.(map[string]interface{})
        hosts = append(hosts, Host{
            Name:     key,
            Username: v["username"].(string),
            Password: v["password"].(string),
        })
    }
    return hosts
}

To sum things up:

1. Values are supposed to be taken from the first provided of: overrides, flags, environment variables, config files, key/value stores, defaults. Unfortunately, this isn't always the order followed (because bugs).
2. You need to change your config format and use a string replacer to leverage the convenience of viper.AutomaticEnv.