Twelve-Factor Applications: How Do You Validate Your Configuration?

Docker has significantly affected the architecture of our applications. If at the beginning we used it to help local development, we now use it in production, too. It's also a great tool to deliver complex applications split by service, as docker images. This type of design provides a lot of flexibility, but also a lot of complexity. The twelve-factor methodology is a reliable reference for building software-as-a-service apps.

This post refers to the third point of these 12 factors: the configuration.

The Problem

The 12 factors guidelines recommend:

a strict separation of config from code. Config varies substantially across deploys, code does not. [..] The twelve-factor app stores config in environment variables.

So far, in production, our twelve-factor applications use either Docker Compose or Docker Swarm on the customer's servers. We did not yet have the opportunity to try Kubernetes. The customer provides us with a registry on which we deliver the images of the services. Together with the customer, we maintain the docker-compose.yml file (or swarm.yml). All the environment variables are managed inside a environmentName.env file, injected into the containers. It's the server's operating manager who is responsible for the content of this file.

// in swarm.yml

version: "3.4"

services:
  service1:
    image: service1
    env_file:
      - ./staging.env
  service2:
    image: service2
    env_file:
      - ./staging.env
  ...

But how do we ensure that the customer's environment variables are all present and valid for each version delivered, while keeping them strictly separate from the delivered code (access to databases, internal web services, etc...)?

Our Current Solution

The simplest solution we have found for the moment is to deliver a specific image whose only role is to validate these environment variables.

And for that, we used a javascript configuration management tool that we have been using for quite some time: convict.

convict allows defining a schema in which a configuration is described as:

VARIABLE_NAME:{
    doc: "Variable description",
    format: "the `format` property specifies either a built-in convict format (`ipaddress`,`port`,`int`, etc.), or it can be a function to check a custom format.",
    default: "Every setting_must_have a default value.",
    env: "If the variable specified by `env` has a value, it will overwrite the setting's default value."
}

The idea is to define all our environment variables in this schema, to describe them with doc, to set them to an empty default value and to systematically fill env.

For example, let's consider that the configuration of our application requires three environment variables NODE_ENV, POSTGRES_PASSWORD and POSTGRES_USER. Here's what the schema can look like:

// in src/config
const convict = require("convict");

const config = convict({
  NODE_ENV: {
    default: "",
    doc: "The application environment.",
    env: "NODE_ENV",
    format: ["production", "development", "test"],
  },
  POSTGRES_PASSWORD: {
    default: "",
    doc: "PostgreSQL's user password",
    env: "POSTGRES_PASSWORD",
    format: String,
  },
  POSTGRES_USER: {
    default: "",
    doc: "PostgreSQL's user",
    env: "POSTGRES_USER",
    format: String,
  },
});

module.exports = config;

Then, the validate convict method applied to an empty configuration file (config.load({})) will ensure that validation is done only on the current environment variables.

// in src/index.js
const signale = require("signale");
const config = require("./config");

const validateConfiguration = () => {
  config.load({});
  try {
    config.validate({ allowed: "strict" });
    signale.success();
  } catch (error) {
    signale.error(`\\n${error.message}\\n`);
  }
};

Note: signale is used to make the console output more readable.

All that remains is to create a Docker image from the two files index.js and config.js:

// in Dockerfile
FROM node:dubnium-alpine

COPY ./src ./validator
WORKDIR /validator
COPY ./package.json ./package.json
COPY ./yarn.lock ./yarn.lock
RUN yarn install --non-interactive  --frozen-lockfile

CMD ["node", "index.js", "validate"]

docker build -t myapp_conf_validation:latest

It is now possible to launch the validation of the file myenv.env:

docker run --rm -it --env-file=myenv.env myapp_conf_validation:latest

The code is available on Github

Conclusion

This tool has allowed us to improve collaboration with our customers teams. It has also helped us avoid several errors during deployments. From this perspective, it's a good tool because it solves a problem.

But, it's still very imperfect. It cannot be integrated into a complete deployment automation. If it's possible to use it in a script to secure a deployment, for example in a makefile recipe, it remains a manual procedure.

// in makefile
deploy: ## Check current conf file before deploy latest images
    docker pull myapp_conf_validation:latest
    docker run --rm -it --env-file=myenv.env myapp_conf_validation:latest
    docker stack deploy myStack

Overall, what really prevents it from being used in a real continuous deployment process is the way we manage environment variables: in a file maintained only by the ops team.

On your side, how do you validate your configuration for your twelve-factor applications?