Note: this tutorial has been written for Ansible Molecule 3 (Releases · ansible-community/molecule · GitHub)

To test your roles or playbooks, you can use VirtualBox or Hyper-V to build new VMs with fresh OS installs. It involves a lot of steps (creating/building a new VM, configuring SSH keys and inventory, run the Ansible playbook, destroy VM). This process can be further enhanced using Vagrant (to provision and destroy VM instances) or Docker, for shorter feedback. However, it can be very cumbersome manually dealing with Vagrant and underlying VMs.

Molecule solves this problem by automating this process and can be seen as an orchestrator: it will take charge of spinning up fresh installs & destroy them after the role/playbook has been executed. Depending on the chosen driver, Molecule will provision instances (delegated) or containers (docker and podman) to test against.

In this tutorial, I will use the docker driver as it's the default one and is often a good choice.

If you are interested in learning more about the delegated driver, here is a good write-up: https://medium.com/@fabio.marinetti81/validate-ansible-roles-through-molecule-delegated-driver-a2ea2ab395b5.

Ansible testing levels

You probably heard of the test pyramid in software development. The test pyramid defines three layers : unit tests, integrations tests and end-to-end tests. Infrastructure as Code (IaC) testing using Ansible tools involves the same concepts:

There are multiple levels of testing with Ansible (from bottom to top):

• Unit tests:
  • Testing yaml structure: yamllint
  • Testing Ansible playbook structure: ansible-playbook --syntax-check
  • Check for bad practices: ansible-lint
• Integration tests: molecule test
• End-to-end tests: testing the actual role or playbook against a production environment using Ansible's check mode (Dry Run) ansible-playbook --check. You can use this mode to check that your role or playbook is idempotent

Notes:

• As ansible-playbook --syntax-check is only a static check, more integration tests are needed to ensure that dynamic includes (include_tasks) work as expected
• Ansible's dry-run mode will not make any changes on the target system. It will only report what changes would have been made without the check mode.
• Idempotency is the ability to run a task multiple times with the same result (ie. don't run the task again if the target has the desired state).

How it works?

Testing steps

When running tests (molecule test), Molecule goes through a series of steps (the test matrix). Here is a summary:

• dependency: collect required dependencies (roles, collections) using specified dependency manager (in molecule.yml), Galaxy is the default one
• lint: lint project using an external shell command (ansible-lint is recommended)
• cleanup: (using a provided cleanup.yml, specified in molecule.yml). This playbook is used to clean up test infrastructure set up in the prepare phase. This step is executed directly before every destroy step.
• destroy: destroy the target instance against which the playbook has been run
• create: create the target instance, using the defined driver in molecule.yml
• prepare: prepare the instance: install any needed packages for the tested role/playbook
• converge: the actual test, import the role/playbook and run it
• verify: verify that the role/playbook has been correctly imported/executed, using the specified verifier in the molecule.yml file
• idempotence: run again the converge phase, and ensure that the role/playbook is idempotent. Under the hood, Molecule runs the playbook used at the converge step and checks for the changed boolean in the return values, indicating that a task had to make changes and that the idempotency is not guaranteed. The step will fail with errors.

Additionally, each step can be run independently:

molecule <step>

Installing Molecule

Molecule is easy to install:

pip3 install molecule

Note: The Molecule team highly recommends to install it in a Python virtual environment using pyenv.

If you are not very familiar with Python or don't have any valid Python install, I recommend its use through Docker:

docker run --rm -it quay.io/ansible/molecule:3.0.0 molecule --version

It's fair easy and allows you to get started without installing Python, pip, and its various dependencies.

Initializing a new role

Initializing a new role using Molecule is also easy:

molecule init role

Using Docker (Linux):

docker run --rm -it \
    -v "$(pwd)":/molecule/:ro \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -w /molecule/ \
    quay.io/ansible/molecule:3.0.0 \
    molecule init role role.name

On Windows:

docker run --rm -it ^
-v "%cd%":/molecule/:ro ^
-v /var/run/docker.sock:/var/run/docker.sock ^
-w /molecule/ ^
quay.io/ansible/molecule:3.0.0 ^
molecule init role role.name

Ansible's Galaxy users will not be lost as Molecules uses it to generate role layouts. If you are not familiar with Ansible Galaxy, you can review the directory structure here.

Molecule layout

molecule
└── default
    ├── converge.yml
    ├── Dockerfile.j2
    ├── INSTALL.rst
    ├── molecule.yml
    └── verify.yml

The molecule.yml file

The molecule.yml file is particularly important as it is used to configure Molecule.

---
dependency:
  name: galaxy
driver:
  name: docker
platforms:
  - name: instance
    image: docker.io/pycontribs/centos:7
    pre_build_image: true
provisioner:
  name: ansible
verifier:
  name: ansible
lint: |
  set -e
  yamllint .
  ansible-lint

Here are the most important sections:

verifier:

As of Molecule 3.0, Ansible is now the default verifier. It simply is an Ansible playbook where you can write specific state checking tests on the target instance. Optionally, you can use Testinfra if you are familiar with this tool. I prefer sticking with the ansible verifier as I don't want to switch language to write my tests. For the rest of this tutorial, I will assume that the verifier is set to ansible.

platforms:

In this section, you can specify the Docker image used to create the target instance. You can also mount volumes or publish ports. If you want to test against multiple distributions (CentOS, Fedora, Debian), you can use an environment variable:

platforms:
  - name: instance
    image: ${DOCKER_IMAGE_DISTRIBUTION}
    command: ${DOCKER_IMAGE_COMMAND:-""}
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:ro
    published_ports:
      - 0.0.0.0:80:8080
    privileged: true
    pre_build_image: true

lint:

This section is used to specify an external command that Molecule will use to handle project linting.

The converge.yml file

This is where you import your role. It simply is an Ansible playbook that Molecule runs right after the instance creation (setup). Typically, this file will be similar to this:

---
- name: Converge
  hosts: all
  become: true

  pre_tasks:
    - name: Ensure openssh-server is installed.
      package:
        name:
          - openssh-server
        state: present

  roles:
    - role: my-role

The pre_tasks section allows you to prepare the instance before importing your role.

This file can be run independently using this command:
molecule converge

The verify.yml file

Finally, the verify.yml file will contain Ansible instructions to verify that your role has been correctly installed on the instance. This another Ansible playbook is run immediately after the role import (converge.yml file).

For instance, here is a verify.yml file used to check that Nginx is correctly serving web requests:

- name: Verify
  hosts: all
  
  tasks:
    - name: Verify Nginx is serving web requests
      uri:
        url: http://localhost/
        status_code: 200

Similarly to the converge step, the verify step will run the verify.yml playbook. This command can be used to run it without launching the entire Molecule sequence:
molecule verify

Molecule will run this playbook against the target instance, created earlier.

Typical test workflow

As stated at the beginning of this tutorial, Molecule will go through a long series of steps (the test matrix), listed below by Molecule itself:

--> Test matrix
    
└── default
    ├── dependency
    ├── lint
    ├── cleanup
    ├── destroy
    ├── syntax
    ├── create
    ├── prepare
    ├── converge
    ├── idempotence
    ├── side_effect
    ├── verify
    ├── cleanup
    └── destroy

This test matrix is similar to the Maven lifecycle, for those who know well Maven (welcome Java developers ;).

As the entire test matrix takes some time to complete, a typical workflow when developing a role or playbook could be:

1. molecule create: create the target instance.
2. molecule converge: run the actual playbook against the created instance. Additionally, lint your files using molecule lint.
3. molecule verify: ensure that the written tests are green.
4. Bring modifications to your playbook.
5. Run again molecule converge / molecule verify to test your modifications.
6. Additionally, if you left the target instance in a broken state, you can destroy it using molecule destroy, you would then recreate it using molecule create.
7. Run molecule converge to test your changes.
8. Finally, run the entire test cycle molecule test to ensure your role is correctly working, in particular idempotency.

Note: another way to ensure that your playbook is idempotent would be to run molecule converge twice. However you would have to pay attention to the Ansible tasks return values, and check for changed, as specified in the testing steps section.

Debugging a Molecule test

When your tests are failing, it can be very useful to inspect the created instance to see what's happening inside.

The --destroy=never flag

The --destroy=never flag simply tells Molecule to not destroy the created instance after running the tests, allowing you to inspect it:

molecule test --destroy=never

Using Docker (Linux):

docker run --rm -it \
	-v "$(pwd)":/molecule/:ro \
	-v /var/run/docker.sock:/var/run/docker.sock \
	-v $HOME/.cache/:/root/.cache/ \
	-w /molecule/ \
	quay.io/ansible/molecule:3.0.0 \
	molecule test --destroy=never

On Windows:

docker run --rm -it ^
-v "%cd%":/molecule/:ro ^
-v ~/.cache/:/root/.cache/ ^
-v /var/run/docker.sock:/var/run/docker.sock ^
-w /molecule/ ^
quay.io/ansible/molecule:3.0.0 ^
molecule test --destroy=never

Login into the instance

To inspect the instance state, simply issue this command:

molecule login

This will directly logs you into the Molecule instance used to test your playbook against.

Using Docker (Linux):

docker run --rm -it \
    -v "$(pwd)":/molecule/:ro \
    -v $HOME/.cache/:/root/.cache/ \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -w /molecule/ \
    quay.io/ansible/molecule:3.0.0 \
    molecule login

On Windows:

docker run --rm -it ^ 
-v "%cd%":/molecule/:ro ^
-v ~/.cache/:/root/.cache/ ^
-v /var/run/docker.sock:/var/run/docker.sock ^
-w /molecule/ ^
quay.io/ansible/molecule:3.0.0 ^
molecule login

Note: If you use Molecule through Docker, you won't be able to log into the target instance, unless you mount the $HOME/.cache host directory into the container, for each Molecule commands involving interactions with the target instance.

ERROR: Instances not created.  Please create instances first.

Indeed, Molecule uses this directory to store its state (the Docker container name of the created instance for example).

As the test instance is created using Docker (Docker driver), you can also issue a docker ps command and execute an interactive shell on the container:

docker exec -it <container_id> sh

Testing Ansible playbooks

As the converge.yml file is just an Ansible playbook that is run to execute tests, testing a playbook is quite similar to testing a role. First, issue this command to create a new scenario in your playbook's directory:

molecule init scenario

Then, instead of including a role in the converge.yml file, simply import a playbook:

---
- name: Converge
  hosts: all
  become: true

  pre_tasks:
    - name: Ensure openssh-server is installed.
      package:
        name:
          - openssh-server
        state: present

- import_playbook: ../../playbook.yml

Run the molecule converge command to execute the playbook. You can also use a verify.yml file to ensure your playbook works as expected.

Make sure the target hosts in your playbook match the ones defined in the converge.yml file.

Recommended resources

Books

The must-read book on this topic is obviously Ansible for DevOps by the famous Jeff Geerling, which is also the author of 99 Ansible roles.

Courses

For those who prefer watching screencasts over reading books, here is a great one on Pluralsight, authored by RedHat:
https://app.pluralsight.com/library/courses/ansible-fundamentals/table-of-contents

For an in-depth tutorial, head over this course:
https://app.pluralsight.com/library/courses/getting-started-ansible/table-of-contents

If you don't own a Pluralsight account yet, use this link to get 50% off your first month or 15% off an annual subscription.

Docs

The official documentation.

Unit testing Docker images with Google Container Structure Tests

To help maintainers write unit tests for their Docker images, the Google's Container Tools team has released a framework which provides a simple way to test the structure and content of a Docker image. Mostly written in Go, it is pretty mature since Google's teams have been using their framework for more than a year.

This tutorial has been written for the 1.3.0 version

Types of unit tests for Docker images

By providing 4 types of unit tests, this framework will help you to ensure that the required content/commands are available at runtime when shipping the Docker image:

• command tests (run the specified command inside the container and verify the correct execution)
• file existence tests (check the existence of a specified file inside the container)
• file content tests (check the content of a specified file)
• metadata tests (check the container configuration: environment variables, volumes, entrypoints, ports, etc.).

Running tests

You can either run the tests using the binary (Linux) or the Docker image (Windows).

The framework runs the specified tests from a given .yaml or .json file. The available tests types are listed in the next section.

Using the Google Container Structure test binary (Linux)

Simply download the latest binary here. The usage is straightforward:

./container-structure-test-linux-amd64 test --image sample-docker-image sample_test_config.yaml

Using a Docker image

As the Google Container Structure Tests binary is only compatible with Linux, I have developed a Docker image which make easy to run Docker tests in a Windows environment:

docker run --rm -v "<path-to-tests-config-file>:/test-config/tests_config.yaml" \
  -v /var/run/docker.sock:/var/run/docker.sock flopes/container-structure-test-docker "test --image <image-to-test> --config tests_config.yaml"

Testing Dockerfile instructions

As said before, the test configurations listed below have to be placed in a .yaml or .json file.

COPY, ADD

FROM alpine:3.7
ADD entrypoint.sh /entrypoint.sh

To test the Docker ADD/COPY instruction, you can use the following test configuration:

schemaVersion: '2.0.0'
fileExistenceTests:
- name: 'entrypoint'
  path: '/entrypoint.sh'
  shouldExist: true
  permissions: '-rwxr-xr-x'

This section will ensure that the specified file exists at the specified location (path) with the correct permissions. Note the possibility to test the nonexistence of a file.

To enhance this test, you can also add a fileContentTests section:

schemaVersion: '2.0.0'
fileContentTests:
- name: 'entrypoint'
  path: '/entrypoint.sh'
  expectedContents: ['echo']

Note that you can also use the excludedContents field to ensure that the specified file does NOT contain the given content.

RUN

To test the RUN instruction, you can either use a fileExistenceTests or fileContentTests, if you download or create a file using this instruction.
However, when using the RUN instruction to install packages or binaries, the commandTests will be relevant:

FROM alpine:3.7

RUN apk add --update curl

The corresponding test in the .yaml file:

schemaVersion: '2.0.0'
commandTests:
  - name: "curl package installation"
    setup: [["/entrypoint.sh"]]
    command: "which"
    args: ["curl"]
    expectedOutput: ["/usr/bin/curl"]

Testing Image metadatas

The metadataTest section will check the following container instructions:

• ENV
• LABEL
• ENTRYPOINT
• CMD
• EXPOSE
• VOLUME
• WORKDIR

To show its usage, let's test the following Dockerfile:

FROM alpine:3.7

LABEL MAINTAINER="Florian Lopes"

ENV PROFILE DEV

ADD entrypoint.sh /entrypoint.sh

VOLUME /volume

WORKDIR /

EXPOSE 80

ENTRYPOINT ["/entrypoint.sh"]

CMD ["--help"]

docker build -t metadata .

The test configuration:

schemaVersion: '2.0.0'
metadataTest:
  env:
  - key: 'PROFILE'
    value: 'DEV'
  labels:
  - key: 'MAINTAINER'
    value: 'Florian Lopes'
  volumes: ['/volume']
  workdir: ['/']
  exposedPorts: ['80']
  entrypoint: ['/entrypoint.sh']
  cmd: ['--help']

Run the tests:

 ./container-structure-test-linux-amd64 test --image test --config test_config.yaml

=========================================
====== Test file: tests_config.yml ======
=========================================

=== RUN: Metadata Test
--- PASS

=========================================
================ RESULTS ================
=========================================
Passes:      1
Failures:    0
Total tests: 1

PASS

Advanced usage

Setup/teardown commands

Setup command

Sometimes, an image needs an ENTRYPOINT instruction in order to properly initialize the container. As the Google Container Structure Tests framework works by overriding container entrypoint (see here), the defined ENTRYPOINT in the Dockerfile will not be honored.
To overcome this limitation, you can use the setup field to run an entrypoint script.

For example, let's say we want to install the curl package at the container startup:

FROM alpine:3.7

COPY entrypoint.sh /entrypoint.sh

RUN chmod +x /entrypoint.sh

ENTRYPOINT ["/entrypoint.sh"]

#!/bin/sh
apk add --update curl

To ensure the /entrypoint.sh script is executed, we provide the setup field with the /entrypoint.sh script.

schemaVersion: '2.0.0'
commandTests:
  - name: "curl package installation"
    setup: [["/entrypoint.sh"]]
    command: "which"
    args: ["curl"]
    expectedOutput: ["/usr/bin/curl"]

Teardown command

As the setup field, the teardown one can be used to execute commands after the actual test command.

schemaVersion: '2.0.0'
commandTests:
  - name: "curl package installation"
    teardown: [["/entrypoint.sh"]]
    command: "which"
    args: ["curl"]
    expectedOutput: ["/usr/bin/curl"]

Samples

You can see a full sample for the spring-boot-docker image or the container-structure-test Docker image itself.

Automating Docker images tests

Automating Docker image tests is easy, simply provide your CI environment with the container-structure-test binary or the Docker image.

An example demonstrating Travis CI integration is available here.

Testing Spring MVC form validations using a tool for MockMvcRequestBuilder

Spring MVC Test Framework is great to test controllers without even running a Servlet container. However, it’s not always straightforward when dealing with form validation, especially when your forms have a lot of properties. This post will show you how to test your form validations easily using a tool that allows Spring's MockMvcRequestBuilder to post an entire form object to a controller.

Validating forms with @Valid annotation

A quick recap about JSR-303 support in Spring MVC.

The @Valid annotation tells Spring MVC to trigger validation on the annotated bean once a request is made:

    @PostMapping("/add")
    public String addUser(@Valid AddUserForm addUserForm, BindingResult bindingResult, RedirectAttributes redirectAttributes) {
        if (bindingResult.hasErrors()) {
            return ADD_USER_VIEW;
        } else { // Save new user 
            redirectAttributes.addFlashAttribute("flash", "User added");
            return "redirect:" + ADD_USER_URL;
        }
    }

public class AddUserForm {
    @NotNull
    @Size(min = 1)
    private List firstNames;
    @NotNull
    @Size(min = 3)
    private String name;
    @NotNull
    private LocalDate birthDate;
    @NotNull
    @Valid
    private Address address;
    @NotNull
    @Size(min = 1)
    private String[] hobbies;
    @NotNull
    private Gender gender;
}

Submitting form objects with MockMvc

Although MockMvc is a useful tool, it is not so convenient when dealing with huge forms submission. To test a form containing a lot of fields, you have to map each one to an HTTP parameter like this:

this.mockMvc.perform(MockMvcRequestBuilders.post("url")
    .param("field", "fieldValue")
    .param("field2.nestedField", "nestedFieldValue");

This method can be used if the form doesn’t contain too many fields (or nested ones!). However, it makes things more complicated as it’s error prone (field name, missing field, etc.). It’s also repetitive if you have multiples forms validations to test.

This is why I built a tool for MockMVCRequestBuilder (https://github.com/f-lopes/spring-mvc-test-utils).

Note: A better approach would be to reduce the number of fields in your form. If you can’t do it for some reasons, you could still use this tool.

Sending form objects using a custom MockMvcRequestBuilder

This tool allows sending an entire form object using MockMvcRequestBuilder.

The usage is straightforward:

final AddUserForm addUserForm = new AddUserForm(Arrays.asList("John", "Jack"), "Doe",
        LocalDate.now(), new Address(1, "Amber", "New York"));
this.mockMvc.perform(MockMvcRequestBuilderUtils.postForm("/users/add", addUserForm))
	.andExpect(MockMvcResultMatchers.model().attributeErrorCount("addUserForm", 1));

This builder finds fields using reflection and pass them to the request as HTTP parameters:

firstNames[0]=John 
firstNames[1]=Jack 
lastName=Doe 
address.streetNumber=1 
address.street=Amber 
address.city=New York

It also supports property editors to format the fields the way you want:

MockMvcRequestBuilderUtils.registerPropertyEditor(LocalDate.class, new CustomLocalDatePropertyEditor("dd/mm/yyyy");

Get the tool

Add these lines to your pom.xml:

<dependency>
    <groupId>io.florianlopes</groupId>
    <artifactId>spring-mvc-test-utils</artifactId>
    <version>2.2.1</version>
</dependency>

See the documentation for more info: https://github.com/f-lopes/spring-mvc-test-utils.
The example code for this post is available here: https://github.com/f-lopes/spring-mvc-form-validation-tests.

5 tips to reduce Docker image size

Docker images can quickly weight 1 or more GB.  Although the gigabyte price is decreasing, keeping your Docker images light will bring some benefits. This post will give you 5 tips to help reduce your Docker images size and why focusing on it is important.

Update: Docker 1.13 introduced a new –squash option to squash the image layers (experimental): https://docs.docker.com/engine/reference/commandline/build/#/squash-an-images-layers—squash-experimental-only (thanks @SISheogorath).

Why is the image size so important?

Reducing Docker final image size will eventually lead to:

• Reduced build time
• Reduced disk usage
• Reduced download time
• Better security due to smaller footprint
• Faster deployments

What is a layer?

To reduce an image size, it’s important to understand what a layer is.
Every Docker image is composed of multiple intermediate images (layers) which form the final image. This layers stack allows Docker to reuse images when a similar instruction is found.

Each Dockerfile instruction creates a layer at build time:

FROM ubuntu                  # This base image is already composed of X layers (4 at the time of writing)
MAINTAINER Florian Lopes     # One layer
RUN mkdir -p /some/dir       # One layer
RUN apt-get install -y curl  # One layer

Let’s build this image:

$ docker build -t curl .
[...]

$ docker images curl
REPOSITORY            TAG            IMAGE ID            CREATED            VIRTUAL SIZE
test                  latest         732afd2af5a9        About an hour ago  199.3 MB

To see the intermediate layers of an image, type the following command:

$ docker history curl
IMAGE               CREATED             CREATED BY                                      SIZE
732afd2af5a9        About an hour ago   /bin/sh -c apt-get install -y curl              11.32 MB
912b76f3dd8e        About an hour ago   /bin/sh -c mkdir -p /some/dir                   0 B
525804109d88        About an hour ago   /bin/sh -c #(nop) MAINTAINER Florian Lopes      0 B
c88b54fedc4f        9 days ago          /bin/sh -c #(nop) CMD ["/bin/bash"]             0 B
44802199e669        9 days ago          /bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$/   1.895 kB
74a2c71e6050        9 days ago          /bin/sh -c set -xe                                                  && echo '#!/bin/sh' > /u   194.5 kB
140d9fb3c81c        9 days ago          /bin/sh -c #(nop) ADD file:ed7184ebed5263e677   187.8 MB

You can see below that each layer have a size and a command associated to create it. The final image built from this Dockerfile contains 3 layers plus all Ubuntu image layers.

Although this can be somehow difficult to understand, this structure is very important as it allows Docker to cache layers to make the builds much faster. When building an image, the Docker daemon will check if the intermediate image (layer created by the instruction) already exists in its cache to reuse it. If the intermediate layer is not found or has changed, the Docker daemon will pull or rebuild it.

How to reduce image size

As we just saw, the layers play an important role in the final image size. To reduce the final size, we have to focus on the intermediate layers.
Although some of them cannot be reduced (especially the one you start from), we can use a few tips to help reduce the final image size.

Group commands in ONE instruction when possible

Do not perform multiple installs in multiple RUN instructions. Let's compare multiple and single instructions by installing/removing packages :

Installing packages

Separate instructions

To illustrate this statement, let’s build an image with two separate RUN instructions which install curl and mysql-client packages:

FROM ubuntu:16.04

MAINTAINER Florian Lopes

RUN apt-get update
RUN apt-get install -y curl
RUN apt-get install -y mysql-client

$ docker build  -t tip1 .
[...]
$ docker images tip1 
 REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
 tip1                latest              7e9105c27586        3 minutes ago       248.4 MB

Single instruction

Now, let’s gather the two instructions in only one:

FROM ubuntu:16.04

MAINTAINER Florian Lopes
RUN apt-get update && apt-get install -y curl mysql-client

Let’s build our image again:

$ docker build  -t tip1 .
[...]
$ docker images tip1 
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
tip1                latest              2886d17dc7f4        9 seconds ago       248 MB

Although the size difference is not so significant, you can expect better results when installing multiple packages.

Removing packages

Separate instructions

Let’s see another interesting example in which we remove a temporary package in a separate instruction:

FROM ubuntu:16.04                    
MAINTAINER Florian Lopes          
RUN apt-get update && apt-get install -y curl && curl http://[...]
RUN apt-get remove -y curl

You can see here that the curl package is immediately removed after being installed, in a separate instruction.
Let’s see the final image size:

$ docker build -t tip2 .
[...]
$ docker images tip2 
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE 
tip1                latest              632f4bf8667c        8 seconds ago       182.7 MB

Single instruction

This time, let’s combine these instructions into one line:

FROM ubuntu:16.04
MAINTAINER Florian Lopes
RUN apt-get update && apt-get install -y curl && curl http://[...] && apt-get remove -y curl

$ docker build -t tip3 .
[...]
$ docker images tip3
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
tip1                latest              bfea5f186684        11 seconds ago      182.1 MB

You can see that the size of the image has slighltly reduced. Again, the difference is not very significant here because we only remove one package.

Why is there a difference?

As we saw earlier, the Docker daemon creates an image for each instruction to execute the associated command. In the separates instructions example, the superposition of all these images creates the final one. Because of this strategy, the mysql-client package is still part of the final image (in the third layer actually) although being removed further.

Do not install packages recommendations (-–no-install-recommends) when installing packages

RUN apt-get update apt-get install -y --no-install-recommends curl

Remove  no longer needed packages or files, in the SAME instruction if possible

Packages example:

RUN apt-get update && \ 
apt-get install -y --no-install-recommends curl && \
curl <a href="http://download.app.com/install.sh">http://download.app.com/install.sh</a> && \
.install.sh && apt-get remove -y curl

In this example, the package curl is only needed to retrieve an install file. Since it is not needed anymore, it can be removed (in the SAME instruction).

Files example:

RUN wget ${APP_URL} -o /tmp/app/install.sh && \
./tmp/app/install.sh && \ rm -rf /tmp/app/ && \
rm -rf /var/lib/apt/lists/*

Start with a smaller base image

Do you need every Ubuntu (or other base images) packages? If not, you should consider starting with a smaller base image like Alpine (https://hub.docker.com/_/alpine/) which will likely become the base image for all official Docker images (Jenkins, Maven). This base image weights around 5MB whereas Ubuntu one is about 188MB. You can see a great comparison of Docker base images here: https://www.brianchristner.io/docker-image-base-os-size-comparison/.

Inspecting images from DockerHub

To easily inspect a DockerHub image, you can use the MicroBadger service:https://microbadger.com/.

TL;DR

1. Group commands in ONE instruction when possible
2. Do not install packages recommendations (–no-install-recommends)
3. Remove  no longer needed packages or files, in the SAME instruction
4. Clean apt-cache after packages installs
5. Start with a smaller base image: Alpine

If you are too busy to focus on reducing your image size, here is a tool you could consider: https://github.com/jwilder/docker-squash.

Spring Boot & Docker with debug and Spring profiles

Spring Boot and Docker are extremely popular. The Spring Boot adoption now hits 34% according to this survey: http://www.baeldung.com/java-8-spring-4-and-spring-boot-adoption. Docker adoption has more than doubled from 13% to 27% in 2016 according to a RightScale survey.

Spring Boot helps application development process while Docker contributes to streamline deployment.

The spring-boot-docker image will help you run your Spring Boot applications with Docker, either in development or production stage.

Features of spring-boot-docker image

Although there are many images for this purpose, this one provides two useful features:

Easily debug your application

Simply set the variable DEBUG to true in docker-compose.yml file and the application will start in debug mode:

environment: 
- "DEBUG=true"

Run your application with desired Spring profile

Spring profiles are a useful way to separate applications resources between different stages. To specify a Spring profile your application should run with, simply set the  SPRING_PROFILES_ACTIVE variable in docker-compose.yml:

environment:
 - "SPRING_PROFILES_ACTIVE=dev"

To run the application within the Docker container, simply place your executable jar into the assets directory, renamed as follows: spring-boot-application.jar.

Then launch the container:

docker-compose up -d

More instructions are available here: https://github.com/f-lopes/spring-boot-docker

Host multiple subdomains/applications on a single host using Docker

Docker becomes more and more suitable for personal environments, especially with private servers, which can be migrated very often.

A developer usually have more than one app living on his own private server such as a blog, some development apps like Jenkins, GitLab and so on. These apps are likely to be using the standard web port 80. As this port is already bound to your main site for example, your Docker instances will not be accessible throughout this one.

This post will show you one way to host multiples applications such as a blog, a personal website and many others, on a single host using Docker containers.

Target architecture

The ideal architecture for hosting multiples apps within a dedicated server would be to expose each application on port 80 through a specific sub-domain (blog.domain.com, jenkins.domain.com, gitlab.domain.com).

Using Nginx as a reverse-proxy

These requirements can be achieved using a proxy (also called reverse-proxy). Here is a diagram:

This classic architecture could be implemented using Nginx as a reverse-proxy, but this solution comes with inconvenients:

• necessity to write a configuration file per application/container to proxy
• reload Nginx each time an application or a container is added to the architecture.

Using nginx-proxy from Jason Wilder

Nginx-proxy consists in a simple Nginx server and docker-gen. Docker-gen is a small tool written in Go which can be used to generate Nginx/HAProxy configuration files using Docker containers meta-data (obtained via the Docker API).

These two applications are running as a Docker container and so are easy to get up running. Once started, nginx-proxy will act as a reverse proxy between your host and all your sub-domains (blog.domain.com, jenkins.domain.com, etc.), effectively routing incoming requests using the VIRTUAL_HOST environment variable (if set, for each Docker containers).

To proxy a Docker container, you basically have to expose the port the applications uses (for example 80 for WordPress) and add the VIRTUAL_HOST environment variable to the container:

Using docker run command:
docker run -d --expose 80 -e VIRTUAL_HOST=blog.domain.com wordpress

Via docker-compose.yml file:

wordpress:
  image: wordpress
  links:
    - db:mysql
  expose:
    - 80
  environment:
    - "VIRTUAL_HOST=blog.domain.com"
db:
  image: mariadb
  environment:
    MYSQL_ROOT_PASSWORD: example

The following configuration could be represented like this:

As you can see above, the Nginx-proxy listens on http standard port (80) and forward incoming requests to the appropriate container. We will see later how this routing is made.

Starting Nginx-proxy

To start the nginx-proxy, type the following command:

shell docker run -d -p 80:80 -v /var/run/docker.sock:/tmp/docker.sock jwilder/nginx-proxy

Using docker-compose syntax:

nginx-proxy:
  image: jwilder/nginx-proxy
  ports:
    - "80:80"
  volumes:
    - /var/run/docker.sock:/tmp/docker.sock

Update:
As Moon suggested in his comment, you could add a piece of extra security to hide Nginx server version using a custom configuration file:

server_tokens off;

To make nginx-proxy use your custom Nginx config file, launch it with this flag:

-v /path/to/my_proxy.conf:/etc/nginx/conf.d/my_proxy.conf:ro

How it works ?

As you can guess with the last command, the nginx-proxy container listens on port 80 and has access to the host Docker socket. By giving the Docker host socket, the nginx-proxy container will be able to receive Docker events (ie. container creations, shutdowns, etc.), and react to them.

At its startup, the nginx-proxy container will look for containers with the VIRTUAL_HOST environment variable set and create appropriate basic Nginx configuration file for each of them. These configuration files will tell Nginx how to forward incoming requests to the underlying containers.

Then, each time a container starts, the nginx-proxy will receive an event and create an appropriate Nginx configuration needed to serve the container application and reload Nginx.

Routing requests using VIRTUAL_HOST environment variable:

Nginx-proxy will route requests to containers according to the VIRTUAL_HOST environment variable of each container. This means that if you want a container to be served with a specific domain or subdomain, you have to launch this one with the desired VIRTUAL_HOST environment variable.

Here is an example:

# Launch WordPress (db part omitted for clarity)
docker run -d --name blog --expose 80 -e VIRTUAL_HOST=blog.domain.com wordpress

# Launch Jenkins
docker run -d --name jenkinsci --expose 8080 -e VIRTUAL_HOST=jenkins.domain.com -e VIRTUAL_PORT=8080 jenkins

Again, here is the equivalent configuration for the Jenkins instance, using docker-compose syntax:

jenkins:
  image: jenkins
  expose:
    - 8080
    - 50000
  environment:
    - "VIRTUAL_HOST=jenkins.domain.com"
    - "VIRTUAL_PORT=8080"
  volumes:
    - "/your/home:/var/jenkins_home"

Note: the port used by the application inside the container must be exposed for nginx-proxy to see it. If the application exposes multiple ports, you have to tell nginx-proxy which port to proxy using the VIRTUAL_PORT environment variable.

In this example, nginx-proxy will forward all requests matching with blog.domain.com url to the WordPress container.
However, all requests beginning by the url jenkins.domain.com will be forwarded to the Jenkins container.

This tool is really simple and gives great flexibility. It allows running multiple Docker containers in the same dedicated server, without writing much configuration.

Tip:
Map a container to multiple domains:
A common requirement is using multiple domains for a given container. To do this, simply add hosts to VIRTUAL_HOST variable like this:
VIRTUAL_HOST=domain.com,www.domain.com,home.domain.com

Further documentation can be found at the following url: https://github.com/jwilder/nginx-proxy.

Spring MVC – Access Spring profiles in JSP

This post explains how to restrict access to an area, based on active Spring profile. By reading this, you will be able to access Spring profiles in JSP in order to achieve this functionality:

<qcm:profile value="dev">
    <form action="login-as-admin" method="POST">
        <input type="submit" value="login as admin"/>
    </form>
</qcm:profile>

Spring profiles

Spring profiles allow to create and run a separate configuration per environment. A common use case is the declaration of multiple data source configuration beans according to the environment (h2 for development purposes, PostgreSQL in production).

To enable any class / method for a given profile, simply annotate the class/bean method with @Profile("..."):

@Profile(value = {QcmProfile.HEROKU, QcmProfile.PROD})
@Bean
public DataSource dataSource() {
    final HikariDataSource dataSource = new HikariDataSource();
    dataSource.setMaximumPoolSize(properties.getMaximumPoolSize());
    dataSource.setDataSourceClassName(properties.getDataSourceClassName());
    dataSource.setDataSourceProperties(dataSourceProperties());
    return dataSource;
}

@Profile(QcmProfile.TEST)
@Bean
public DataSource testDataSource() {
    final HikariDataSource dataSource = new HikariDataSource();
    dataSource.setMaximumPoolSize(properties.getMaximumPoolSize());
    dataSource.setDataSourceClassName(properties.getDataSourceClassName());
    dataSource.setDataSourceProperties(testDataSourceProperties());
    return dataSource;
}

You can see the complete configuration class here.

Any component (@Component / @Configuration) annotated with @Profile("...") will be loaded if the given profile(s) is/are enabled.
This behavior is achieved by using @Conditional(ProfileCondition.class) on the @Profile annotation itself.

As you can see, the ProfileCondition simply check the given value with current environment profile:

/**
 * {@link Condition} that matches based on the value of a {@link Profile @Profile}
 * annotation.
 *
 * @author Chris Beams
 * @author Phillip Webb
 * @author Juergen Hoeller
 * @since 4.0
 */
class ProfileCondition implements Condition {

	@Override
	public boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata) {
		if (context.getEnvironment() != null) {
			MultiValueMap<String, Object> attrs = metadata.getAllAnnotationAttributes(Profile.class.getName());
			if (attrs != null) {
				for (Object value : attrs.get("value")) {
					if (context.getEnvironment().acceptsProfiles(((String[]) value))) {
						return true;
					}
				}
				return false;
			}
		}
		return true;
	}

}

Use Spring profiles in JSP

It may be useful to display a piece of content in a JSP file, based on a specific environment profile.
In my use case, I wanted to display an admin-login button to facilitate tests, only in development phase (development profile).

I found that the best way to achieve this behavior was to develop a custom JSP tag as the Servlet class gives helpers to show/hide a piece of text.

The main concern was to find out how to access Spring profiles inside a tag. Fortunately, Spring provides with a useful tag class: RequestContextAwareTag.

Access Spring profiles in a tag

To gain access to the Spring context, you need your tag to extend RequestContextAware class, which
exposes the current “RequestContext” according to the JavaDoc:

/**
 * Superclass for all tags that require a {@link RequestContext}.
 *
 * <p>The {@code RequestContext} instance provides easy access
 * to current state like the
 * {@link org.springframework.web.context.WebApplicationContext},
 * the {@link java.util.Locale}, the
 * {@link org.springframework.ui.context.Theme}, etc.
 *
 * <p>Mainly intended for
 * {@link org.springframework.web.servlet.DispatcherServlet} requests;
 * will use fallbacks when used outside {@code DispatcherServlet}.
 *
 * @author Rod Johnson
 * @author Juergen Hoeller
 * @see org.springframework.web.servlet.support.RequestContext
 * @see org.springframework.web.servlet.DispatcherServlet
 */

This class extends the TagSupport Servlet class and override the main method doStartTag() to inject the RequestContext:

	/**
	 * Create and expose the current RequestContext.
	 * Delegates to {@link #doStartTagInternal()} for actual work.
	 * @see #REQUEST_CONTEXT_PAGE_ATTRIBUTE
	 * @see org.springframework.web.servlet.support.JspAwareRequestContext
	 */
	@Override
	public final int doStartTag() throws JspException {
		try {
			this.requestContext = (RequestContext) this.pageContext.getAttribute(REQUEST_CONTEXT_PAGE_ATTRIBUTE);
			if (this.requestContext == null) {
				this.requestContext = new JspAwareRequestContext(this.pageContext);
				this.pageContext.setAttribute(REQUEST_CONTEXT_PAGE_ATTRIBUTE, this.requestContext);
			}
			return doStartTagInternal();
		}
		catch (JspException ex) {
			logger.error(ex.getMessage(), ex);
			throw ex;
		}
		catch (RuntimeException ex) {
			logger.error(ex.getMessage(), ex);
			throw ex;
		}
		catch (Exception ex) {
			logger.error(ex.getMessage(), ex);
			throw new JspTagException(ex.getMessage());
		}
	}

By extending the Spring RequestContextAwareTag and overriding the doStartTagInternal() method, your tag will have access to the RequestContext, needed to retrieve Spring profiles.

With this context, it’s easy to retrieve environment profiles:

public class ProfileConditionTag extends RequestContextAwareTag {
    
    private String profile;

    @Override
    protected int doStartTagInternal() throws Exception {
        final Environment environment = this.getRequestContext().getWebApplicationContext().getEnvironment();
        if (environment != null) {
            final String[] profiles = environment.getActiveProfiles();
            if (ArrayUtils.contains(profiles, this.profile)) {
                return EVAL_BODY_INCLUDE;
            }
        }
        return SKIP_BODY;
    }

    public String getValue() {
        return profile;
    }

    public void setValue(String profile) {
        this.profile = profile;
    }
}

Usage

Create the taglib descriptor and place it into the WEB-INF/taglibs/ directory:

<?xml version="1.0" encoding="UTF-8" ?>
    <taglib xmlns="http://java.sun.com/xml/ns/j2ee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-jsptaglibrary_2_0.xsd" version="2.0">
    <description>Conditional profile Tag</description>
    <tlib-version>2.1</tlib-version>
    <short-name>ProfileConditionTag</short-name>
    <uri></uri>
    <tag>
        <name>profile</name>
        <tag-class>com.ingesup.java.qcm.taglib.ProfileConditionTag</tag-class>
        <body-content>scriptless</body-content>
        <attribute>
            <name>value</name>
            <required>true</required>
        </attribute>
    </tag>
</taglib>

Using this tag is pretty straightforward:

<qcm:profile value="dev">
    <form action="login-as-admin" method="POST">
        <input type="submit" value="login as admin"/>
    </form>
</qcm:profile>

You can head at this url and see that the admin-login button doesn’t appear as the active profile for the application is heroku.

Application code

You can see the whole application code in my GitHub project.