The marmelab blog

Create a Full Stack Magento Environment With gaudi

Published on 17 June 2014 by Emmanuel with tags magento architecture gaudi

Magento architecture with gaudi

Scaling Magento: A Complex Architecture

A few years ago, I used to work on a famous French e-commerce website dealing with more than 10K orders a day.
This website was built on top of Magento, and we quickly hit the Magento performance limit (as soon as the first ads were broadcasted on TV).

The first Single Point Of Failure (SPOF) detected was the database, so we decided to add a master/slave replication to the MySQL storage. Magento cache keys are stored in the database by default, and that was another cause of heavy Database load. A memcached storage was set to store and retrieve these cache keys faster. Finally, in order to balance the application load over more than one CPU, we used a F5 load balancer, dispatching visitors to 2 frontend servers running apache.

Reproducing this architecture on the dev and staging environments was a huge pain for each developer. This was a time when orchestators like Chef or Puppet were not yet popular.

Gaudi To The Rescue

Version 0.2 of gaudi, released last week, introduces a lot of new components, bug fixes and examples. Let’s learn to build a complex architecture like the one described above with gaudi.

Configuration File

gaudi provides a GUI to ease the creation of the YAML configuration file. Drag and drop some component to reproduce the architecture above. The result is the following gaudi.yml file:

Here we create a Varnish server, load balancing requests to 2 Apache servers. Each of these servers forwards PHP requests to a PHP-FPM service, linked to a master/slave MySQL database, and a memcached server. Finally, we set up phpMyAdmin to manage our database.

MySQL master/slave replication is automatic when gaudi detects at least 2 Mysql instances.

All files will be mounted in an /app folder. Varnish will use /up.php to check the health of each server. This file is created at the root the our project.

Initializing The Project

First of all, grab a fresh copy of Magento and extract it to a htdocs folder:

mkdir htdocs
wget http://www.magentocommerce.com/downloads/assets/1.9.0.1/magento-1.9.0.1.tar.gz
tar zxvf magento-1.9.0.1.tar.gz -C htdocs --strip 1

Next, create a pretty simple up.php monitoring file. This file will be requested by Varnish to check if the server is running:

echo "<?php echo 'ok';" > htdocs/up.php

Start gaudi:

sudo gaudi

For each server, gaudi will output the IP address, and the port the server is listening to. Take note of the master IP address, as you’ll need it to setup Magento.

Setting Up Magento

Open a browser and go to http://localhost:8080 (or another IP if you use Vagrant).
Setup the database using the IP of the master container, use root for login and an empty password.

Magento is now up and running, but neither memcached nor the slave connection for read queries are configured yet.

Open app/etc/local.xml, and add the following lines after default_setup:

<default_read>
	<connection>
        <host><![CDATA[slave_IP]]></host>
        <username><![CDATA[root]]></username>
        <password><![CDATA[]]></password>
        <dbname><![CDATA[magento]]></dbname>
        <model><![CDATA[mysql4]]></model>
        <type><![CDATA[pdo_mysql]]></type>
        <pdoType><![CDATA[]]></pdoType>
        <active>1</active>
	</connection>
</default_read>

Replace slave_IP by the IP given by gaudi.

Add a cache section in the global node:

<cache>
	<backend>memcached</backend>
	<memcached>
		<compression/>
		<cache_dir/>
		<hashed_directory_level/>
		<hashed_directory_umask/>
		<file_name_prefix/>
		<servers>
		    <default>
		        <host>memcached_IP</host>
		        <port>11211</port>
		        <persistent>1</persistent>
		    </default>
		</servers>
	</memcached>
</cache>

Don’t forget to replace the memcached_IP.

You Magento configuration file should now look like the following:

Retrieving Container IPs Dynamically

Setting up all IPs manually is not ideal. What if they change on the next build?

We have to use environment variables to retrieve the IP of all our containers. Docker injects a lot of information about linked containers in environment variables.

We can override Magento configuration to set these IP dynamically.

Copy app/code/core/Mage/Core/Model/App.php to app/code/local/Mage/Core/Model/App.php, and change the _initBaseConfig method as follows:

<?php
protected function _initBaseConfig()
{
    Varien_Profiler::start('mage::app::init::system_config');
    $this->_config->loadBase();

    // Rewrite configuration using environment variables
    $this->_config->getNode('global/resources/default_setup/connection')->setNode('host', getenv('MASTER_PORT_3306_TCP_ADDR'));
    $this->_config->getNode('global/resources/default_read/connection')->setNode('host', getenv('SLAVE_PORT_3306_TCP_ADDR'));
    $this->_config->getNode('global/cache/memcached/servers/default')->setNode('host', getenv('MEMCACHED_PORT_11211_TCP_ADDR'));

    Varien_Profiler::stop('mage::app::init::system_config');
    return $this;
}

We’re now good to go. Enjoy this complete Magento environment packing memcached, load balanced web servers, PHP application servers, master/slave MySQL replication, and database administration in separate containers, all linked together automatically!

Hope this helps! Check out the gaudi repository for more examples.

comments powered by Disqus