Create a Full Stack Magento Environment With gaudi

Emmanuel Quentin
Emmanuel QuentinJune 17, 2014
#php#devops#oss

Scaling Magento: A Complex Architecture

A few years ago, I used to work on a famous French e-commerce website dealing with more than 10K orders a day. This website was built on top of Magento, and we quickly hit the Magento performance limit (as soon as the first ads were broadcasted on TV).

The first Single Point Of Failure (SPOF) detected was the database, so we decided to add a master/slave replication to the MySQL storage. Magento cache keys are stored in the database by default, and that was another cause of heavy Database load. A memcached storage was set to store and retrieve these cache keys faster. Finally, in order to balance the application load over more than one CPU, we used a F5 load balancer, dispatching visitors to 2 frontend servers running apache.

Reproducing this architecture on the dev and staging environments was a huge pain for each developer. This was a time when orchestators like Chef or Puppet were not yet popular.

Gaudi To The Rescue

Version 0.2 of gaudi, released last week, introduces a lot of new components, bug fixes and examples. Let's learn to build a complex architecture like the one described above with gaudi.

Configuration File

gaudi provides a GUI to ease the creation of the YAML configuration file. Drag and drop some component to reproduce the architecture above. The result is the following gaudi.yml file:

applications:
    lb:
        type: varnish
        ports:
            8080: 80
        links: [front1, front2]
        volumes:
            .: /app
        custom:
            backends: [front1, front2]
            probe_url: /up.php

    front1:
        type: apache
        links: [app1]
        ports:
            8081: 8080
        volumes:
            .: /app
        modules: [rewrite]
        custom:
            fastCgi: app1
            fastCgiIdleTimeout: 500
            documentRoot: /app/htdocs

    front2:
        type: apache
        links: [app2]
        ports:
            8082: 8080
        volumes:
            .: /app
        modules: [rewrite]
        custom:
            fastCgi: app2
            fastCgiIdleTimeout: 500
            documentRoot: /app/htdocs

    app1:
        type: hhvm
        ports:
            9000: 9000
        volumes:
            .: /app
        apt_get: [php5-gd, php5-intl]
        links: [memcached, master, slave1, slave2]
        custom:
            memoryLimit: 512M
            maxExecutionTime: 500

    app2:
        type: php-fpm
        ports:
            9001: 9000
        volumes:
            .: /app
        apt_get: [php5-gd, php5-intl]
        links: [memcached, master, slave1, slave2]
        custom:
            memoryLimit: 512M
            maxExecutionTime: 500

    master:
        type: mysql
        ports:
            3306: 3306
        volumes:
            /var/tmp/data/master: /var/lib/mysql
        after_script:
            mysql -e "CREATE DATABASE IF NOT EXISTS magento CHARACTER SET utf8 COLLATE utf8_general_ci;"
        custom:
            repl: master

    slave1:
        type: mysql
        ports:
            3307: 3306
        volumes:
            /var/tmp/data/slave1: /var/lib/mysql
            /var/tmp/data/master: /var/lib/mysql-master
        links: [master]
        custom:
            repl: slave
            master: master

    slave2:
        type: mysql
        ports:
            3308: 3306
        volumes:
            /var/tmp/data/slave2: /var/lib/mysql
            /var/tmp/data/master: /var/lib/mysql-master
        links: [master]
        custom:
            repl: slave
            master: master

    memcached:
        type: index
        image: borja/docker-memcached
        ports:
            11211: 11211

    phpmyadmin:
        type: phpmyadmin
        ports:
            80: 80
        links: [master]

Here we create a Varnish server, load balancing requests to 2 Apache servers. Each of these servers forwards PHP requests to a PHP-FPM service, linked to a master/slave MySQL database, and a memcached server. Finally, we set up phpMyAdmin to manage our database.

MySQL master/slave replication is automatic when gaudi detects at least 2 Mysql instances.

All files will be mounted in an /app folder. Varnish will use /up.php to check the health of each server. This file is created at the root the our project.

Initializing The Project

First of all, grab a fresh copy of Magento and extract it to a htdocs folder:

mkdir htdocs
wget http://www.magentocommerce.com/downloads/assets/1.9.0.1/magento-1.9.0.1.tar.gz
tar zxvf magento-1.9.0.1.tar.gz -C htdocs --strip 1

Next, create a pretty simple up.php monitoring file. This file will be requested by Varnish to check if the server is running:

echo "<?php echo 'ok';" > htdocs/up.php

Start gaudi:

sudo gaudi

For each server, gaudi will output the IP address, and the port the server is listening to. Take note of the master IP address, as you'll need it to setup Magento.

Setting Up Magento

Open a browser and go to http://localhost:8080 (or another IP if you use Vagrant). Setup the database using the IP of the master container, use root for login and an empty password.

Magento is now up and running, but neither memcached nor the slave connection for read queries are configured yet.

Open app/etc/local.xml, and add the following lines after default_setup:

<default_read>
	<connection>
        <host><![CDATA[slave_IP]]></host>
        <username><![CDATA[root]]></username>
        <password><![CDATA[]]></password>
        <dbname><![CDATA[magento]]></dbname>
        <model><![CDATA[mysql4]]></model>
        <type><![CDATA[pdo_mysql]]></type>
        <pdoType><![CDATA[]]></pdoType>
        <active>1</active>
	</connection>
</default_read>

Replace slave_IP by the IP given by gaudi.

Add a cache section in the global node:

<cache>
	<backend>memcached</backend>
	<memcached>
		<compression/>
		<cache_dir/>
		<hashed_directory_level/>
		<hashed_directory_umask/>
		<file_name_prefix/>
		<servers>
		    <default>
		        <host>memcached_IP</host>
		        <port>11211</port>
		        <persistent>1</persistent>
		    </default>
		</servers>
	</memcached>
</cache>

Don't forget to replace the memcached_IP.

You Magento configuration file should now look like the following:

<?xml version="1.0"?>

<config>
    <global>
        <install>
            <date><![CDATA[Wed, 28 May 2014 15:55:03 +0000]]></date>
        </install>
        <crypt>
            <key><![CDATA[25bb2c3dd7ffb23fc2caa0f083431754]]></key>
        </crypt>
        <disable_local_modules>false</disable_local_modules>
        <resources>
            <db>
                <table_prefix><![CDATA[]]></table_prefix>
            </db>
            <default_setup>
                <connection>
                    <host><![CDATA[172.17.0.2]]></host>
                    <username><![CDATA[root]]></username>
                    <password><![CDATA[]]></password>
                    <dbname><![CDATA[magento]]></dbname>
                    <initStatements><![CDATA[SET NAMES utf8]]></initStatements>
                    <model><![CDATA[mysql4]]></model>
                    <type><![CDATA[pdo_mysql]]></type>
                    <pdoType><![CDATA[]]></pdoType>
                    <active>1</active>
                </connection>
            </default_setup>

            <default_read>
                <connection>
                    <host><![CDATA[172.17.0.4]]></host>
                    <username><![CDATA[root]]></username>
                    <password><![CDATA[]]></password>
                    <dbname><![CDATA[magento]]></dbname>
                    <model><![CDATA[mysql4]]></model>
                    <type><![CDATA[pdo_mysql]]></type>
                    <pdoType><![CDATA[]]></pdoType>
                    <active>1</active>
                </connection>

                <connection>
                    <host><![CDATA[172.17.0.5]]></host>
                    <username><![CDATA[root]]></username>
                    <password><![CDATA[]]></password>
                    <dbname><![CDATA[magento]]></dbname>
                    <model><![CDATA[mysql4]]></model>
                    <type><![CDATA[pdo_mysql]]></type>
                    <pdoType><![CDATA[]]></pdoType>
                    <active>1</active>
                </connection>
            </default_read>
        </resources>
        <session_save><![CDATA[files]]></session_save>

        <cache>
            <backend>memcached</backend>
            <memcached>
                <compression/>
                <cache_dir/>
                <hashed_directory_level/>
                <hashed_directory_umask/>
                <file_name_prefix/>
                <servers>
                    <default>
                        <host>172.17.0.3</host>
                        <port>11211</port>
                        <persistent>1</persistent>
                    </default>
                </servers>
            </memcached>
        </cache>
    </global>
    <admin>
        <routers>
            <adminhtml>
                <args>
                    <frontName><![CDATA[admin]]></frontName>
                </args>
            </adminhtml>
        </routers>
    </admin>
</config>

Retrieving Container IPs Dynamically

Setting up all IPs manually is not ideal. What if they change on the next build?

We have to use environment variables to retrieve the IP of all our containers. Docker injects a lot of information about linked containers in environment variables.

We can override Magento configuration to set these IP dynamically.

Copy app/code/core/Mage/Core/Model/App.php to app/code/local/Mage/Core/Model/App.php, and change the _initBaseConfig method as follows:

<?php
protected function _initBaseConfig()
{
    Varien_Profiler::start('mage::app::init::system_config');
    $this->_config->loadBase();

    // Rewrite configuration using environment variables
    $this->_config->getNode('global/resources/default_setup/connection')->setNode('host', getenv('MASTER_PORT_3306_TCP_ADDR'));
    $this->_config->getNode('global/resources/default_read/connection')->setNode('host', getenv('SLAVE_PORT_3306_TCP_ADDR'));
    $this->_config->getNode('global/cache/memcached/servers/default')->setNode('host', getenv('MEMCACHED_PORT_11211_TCP_ADDR'));

    Varien_Profiler::stop('mage::app::init::system_config');
    return $this;
}

We're now good to go. Enjoy this complete Magento environment packing memcached, load balanced web servers, PHP application servers, master/slave MySQL replication, and database administration in separate containers, all linked together automatically!

Hope this helps! Check out the gaudi repository for more examples.

Did you like this article? Share it!