Nginx dynamic hostname discovery for Docker

I've read several references mentioning the best way to put several LAMP (Linux-Apache-MySQL-PHP) based Docker containers on the same server was to put them behind an Nginx container. Nginx acts as a light-weight proxy. The problem I found however, was no one explained the best way to tackle the problem of each LAMP container having a list of hostnames that Nginx needs to know about in order to proxy the right traffic to the right container. At ShopIgniter, each container has multiple hostnames it listens on, and Nginx needs to route traffic accordingly.

Preface

First let me point out that my way is not by any means the only way to implement this. The easiest and quickest way would be to mount a volume from Nginx's sites-enabled directory to the host machine where you can then manually add all the proxy configs there on the host. I won't detail how to do that in this post. Since we're aiming for Continuous Deployment with Docker, this doesn't work how I envision it should.

Why not just manually maintain Nginx configs? I like the idea of the containers being as isolated as possible. The Nginx container shouldn't be rebuilt for each new service added, and I want the host server to do nothing more but start, stop, and link containers. I also do not want the host server to know our business logic and the containers' hostnames.

My solution: the containers should broadcast their own hostnames.

Link All The Things!

The way I implemented it was by linking all our LAMP containers to the Nginx container, and add some discovery endpoints on the LAMP for Nginx to read and then create its own configs.

To create the discovery endpoints on the LAMP container, I simply added a new Apache VirtualHost that listens on port 8080 (this port can be anything other than 80, really). Traffic coming into this port then is then assumed to be from Nginx, and is secure as it is not exposed to the outside world.

Add a new VirtualHost to your LAMP containers that looks like such.

# Nginx-only exposed endpoints
Listen 8080  
NameVirtualHost *:8080  
<VirtualHost *:8080>  
    DocumentRoot /var/www/vhosts/ops
    <Directory /var/www/vhosts/ops>
        AllowOverride All
        Order allow,deny
        allow from all
    </Directory>
</VirtualHost>  

In the Dockerfile used to create your LAMP container, you'll need to ADD an ops folder. The ops folder can house more than just hostname broadcasting, but for now, just add a PHP file called 'hosts.php'.

The hosts.php file should do any logic you need it to determine the hostnames of its own container's app. For us, we loaded the Symfony YAML loader and read our Symfony's parameters which had the hostname information in it. To show a working example, however, let's stick with this. I found it easiest to use Ruby on Nginx to parse the results; so we can use YAML to format our hostnames. You can also use JSON.

<?php  
# your logic to determine hostnames should go here, if needed
header('Content-Type: application/x-yaml');  
echo "---  
hostnames:  
  - example.com
  - www.iana.org";

Now for the Nginx logic. Each time Nginx starts, it will need to call out to each linked container and determine their hostnames. To know all the containers its connected to, I prefix the LAMP container names with app_. This will make it easier to find all the linked containers inside Nginx. When starting Nginx, it looks a little like this:

docker run -d -name nginx -link mylamp1:app_mylamp1 -link mylamp2:app_mylamp2 -p 80:80 nginx  

To find the linked containers each time Nginx is started, you can create a run script and ADD it, or a Dockerfile ENTRYPOINT. For the latter: ENTRYPOINT ruby /proxy_targets.rb && nginx. Before it starts up Nginx, it runs a ruby script called 'proxy_targets.rb'.

proxy_targets.rb parses all the ENV variables looking for any that start with APP_ that we prefixed above. The ENV we're looking for specifically is APP_MYLAMP1_PORT. This field has both the container's IP and Port. However, it is in a tcp://ip:port format, so it needs to broken apart as well. Afterwards, it makes a copy of our template config with token placeholders in it and replaces them with real values, placing the new config in the correct folder.

The template looks as such:

server {  
  listen 80;
  server_name {{hostnames}};

  proxy_set_header Host $http_host;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

  location / {
    proxy_pass http://{{container-target-ip}}:{{container-target-port}};
  }
}

The proxy_targets.rb now has everything it needs. It should find the environments, parse them, copy the template conf, and replace the values.

require 'fileutils'  
require 'net/http'  
require 'yaml'

# Find all our linked containers via environmentals
apps = {}  
ENV.each do |env, value|  
  env_parts = env.split('_')
  if env_parts.length == 3 && env_parts[0] == 'APP' && env_parts[2] == 'PORT'
    cont = env_parts[1].downcase
    ip_port = value.sub(/^.*?:\/\//, '').split(':')
    apps[cont] = {}
    apps[cont][:ip] = ip_port[0]
    apps[cont][:port] = ip_port[1]
  end
end

# Copy default nginx site conf template and change out tokens
apps.each do |cont, app|  
  # get details from destination container
  response = Net::HTTP.get_response(URI.parse("http://#{app[:ip]}:8080"))
  data = YAML.load(response.body)
  hostnames = data['hostnames'];

  # copy the template conf to new file
  new_file = "/etc/nginx/sites-enabled/#{cont}"
  FileUtils.cp '/nginx-template.conf', new_file
  # open new conf
  text = File.read(new_file)
  # replace tokens
  text.gsub! '{{hostnames}}', hostnames.join(' ')
  text.gsub! '{{container-target-ip}}', app[:ip]
  text.gsub! '{{container-target-port}}', app[:port]
  # write to file
  File.open(new_file, "w") { |file| file.puts text }
end  

Now you can start your LAMP containers, followed by your Nginx container. This should be automated, but is not part of this post.

docker run -d -name mylamp1 mylamp1image  
docker run -d -name mylamp2 mylamp2image  
docker run -d -name nginx -link mylamp1:app_mylamp1 -link mylamp2:app_mylamp2 -p 80:80 nginximage  

Voila! Test your hostnames. Look at the dynamically created nginx configs. Let me know if you have any issues!