Musings of using Aegir for deployment.

I have been following the Aegir projectfor some time now, almost 3 years. It’s great to see how far the project has come along, and how easy it is to get an Aegir instance up (it used to be very challenging to install). However, I haven’t really fully embraced Aegir (yet) into our current workflows at ActiveLAMP. I’m still pondering how exactly I want to use the tool. Allow me to describe our current process, before I elaborate on how I want to use Aegir for deployment.

Our current workflow.

Early in 2010 we adopted a workflow for managing and building sites using Drush Make. Several articles inspired this new workflow for us, so I won’t go into detail of why it’s a good idea to use Drush Make for doing your builds, rather than having one giant repository of code in which 80% of that code you don't touch (hack) anyway. On each of our developers machines we have one drupal core (a platform) that runs any number of sites that we may be working on at the time. We may have several platforms (different core versions i.e. 7.9, 7.10, 7.12, etc...) on our development machines. All sites that we work on are essentially a multisite within the specific platform (drupal core version) that we're working within. With every site we have an aliases.drushrc.php within its sites directory. This aliases.drushrc.php file holds meta data regarding what platform the site is running on locally, as well as where the production server is, and where the dev server is. I wrote a custom drush command that actually sets all this up, so we don't have to do too much work. When we start a new site we just type `drush ns mynewsite.com` (drush new-site) and that will fire off a number of tasks:

  • Check drupal.org to see what the latest version of core is
  • Check local platforms directory on developer machine to see if that core version is installed.
    • If the core version is not installed, it is downloaded, a vhost is setup in the apache config, and the /etc/hosts files is edited for the new platform.
  • Then it checks our git repository for a repo called newsite.com.git and checks that out into a separate sites directory (not within the platform just downloaded)
    • If the site repo doesn't exist, the drush new-site command creates a new site directory with a few files copied over from the drush new-site command for defaults (site.make, .gitignore, and rebuild.sh)
    • git init, git add, git commit, and git push origin master are all executed with the initial files at the very end of the drush command.
  • The new sites directory is symlinked to the platform
  • Two drush alias files are created, a global alias file placed in ~/.drush, and an aliases.drushrc.php in the actual site directory.
  • Finally the site is installed with `drush si`

For deployment we use Capistranoinvoked with a custom Drush command. This drush command looks at parameters set in aliases.drushrc.php within the sites directory and then fires off a capistrano command passing in arguments it finds in aliases.drushrc.php. We're only deploying the sites directory to the new environment, and capistrano takes care of symlinking it to the correct platform on the dev or production servers. I'm leaving out a lot of minor details, but in a nutshell that's how our current workflow works for developing sites and deploying.

Thoughts for using Aegir for deployment.

Most people (that I've read about in blog posts and the drupal.org issue queue) that are using Aegir for deployment, are using install profiles for their custom code, and are utilizing a recursive make file technique to handle the build of the platform, as well as the profile. The process seems to work well, and it makes sense, but I'm not sure I want to handle our deploys this way for a number of reasons:

  • A platform has to be built for every deployment, with the "new" profile.
  • Really only one site is going to run on this platform, unless you put multiple profiles in the platform make file (which then leads to more issues if you have different development cycles for the number of sites you're currently working on.)
  • A whole bunch of extraneous tasks are being fired to build this platform, when only site code could only be deployed.
  • Most important to me, you're getting away from what Aegir does well, host instances of sites on a common platform.

My initial thoughts for handling this with Aegir is just integrating Capistrano with Aegir with a few drush commands and then expose those drush commands to hostmaster. I would also add a sites make file field to the node add form for sites, so that when creating a new site, you can specify a site make file, just like you can when you build a platform. The process of deployment would still be handled by Capistrano, while still utilizing Aegir for creating, managing, and migrating sites. I'm going to start developing this functionality, but I'm curious to hear others thoughts on this, and if there any holes in how I think this could work.

Comments

mig5 (not verified)

Hi,

I haven't much to say other than I encourage you to get your hands dirty with it and let the community know your experience :) As many know, I use Fabric almost exclusively for very similar purposes (see this article and this article. But as I say there, I think Capistrano could easily be a drop-in replacement for those fab commands since at the end of the day, they are both just executing SSH commands. It's the latter article where I describe the actual commands you'll want to implement in order to save Drush contexts and migrate/import sites etc from the command line (and thus via Capistrano).

I don't quite understand the general fear people have of building a platform each time: this seems to be a common concern and I just can't see the big deal. For one thing, building platforms and removing them is *cheap* with Aegir and Drush Make. People should try it and then see what the worry was all about :) 'Extraneous tasks' - really? It takes about 5 to 10 seconds to build most platforms. Disk space is cheap and you can automate the cleanup process afterward to remove old platforms.

Secondly, if you are familiar with Capistrano already, then the concept of 'make a whole fresh build each time and switch symlink/migrate site to new build' should be a no-brainer as that's how most people do it with Cap and Drupal: that's exactly analogous to 'build a platform and migrate to it'.

Yes, Aegir can do multisite very well, but you don't *have* to, and I don't consider it heresy to have a platform per site in this model.

I don't want to discourage you from doing the 'deployment within one platform' approach though - I know for a fact that a lot of people want to do what you're about to do :) The more people that implement automation of this deployment methodology with Aegir, the better - regardless of the finer details/tools they use to do it.

I'd like to hear more about how you handle rollback in the event of failure. Another reason I prefer the 'build platform and migrate each time' approach, is that if it fails, Aegir (well, Provision, through its Drush hooks) is able to rollback the attempted build and the site goes back to running on the previous platform. I know that Capistrano has a rich logic of implementing its own rollback hooks, but I like the fact Aegir already does that part for me with the 'build a platform and migrate' method.

How will you handle rollback if your site deployment screws something up? Symlink the sites dir in the platform back to the previous build?

Keep us informed on your findings and good luck!

Tom Friedhof

Thanks Miguel, your blog is one that I often visit to keep up with what the latest trends of Aegir.

Regarding extraneous tasks, I just mean tasks that don't necessarily need to be done, especially if the change is just a one line css rule in a theme file. I don't mean to imply that it is HUGE resource hog to deploy an entire platform for each change.

The way we are doing deployments with Capistrano is deploying a new release of only the sites directory. In the platform for the site we are deploying, there is a symlink created on initial deploy that symlinks the sites/myexample.com directory to the "current" symlink that Capistrano creates for sites/myexample.com outside of the platform. The rest of the sym-linking happens within the normal workflow of Capistrano. Upon a successful build of the site directory Capistrano symlinks "current" to the latest release, otherwise Capistrano does a rollback and deletes the new release it created and never gets to the symlink stage.

A rollback will simply call `cap deploy:rollback`, which just changes the symlink of the "current" symlink to the previous release. All deployment responsibilities would be handled by Capistrano.

I appreciate your workflow of deploying on single platforms, in fact you've helped me tremendously in understanding how I can finally start using Aegir, it's just a matter of preference to not deploy singles sites on new platforms for each deploy.

I'm happy to hear that others want to have this similar workflow. Thanks for your comment! I look forward to your next blog post on the subject!

Christian Pearce (not verified)

One platform per site. We use a starter project.

https://github.com/xforty/xforty-drupal

Keep your theme separate from your makefile repo. We actually deploy the theme in the sites/default/themes directory. Managing sites aliases etc with multisite is awful. Plus our themers want to be able to do quick css changes and I can't blame them. We just commit the theme to a separate git repo. It is simple enough for everyone involved to manage.

Dean Reilly (not verified)

We follow this approach also but (just to add a third contender) we use phing instead of fab.

Christian Pearce (not verified)

We too have followed and tried to embrace Aegir for the last 3 years. It has come a long way but doesn't work for us. I can see how it might be successful in certain scenarios. We had a lot of trouble getting it to be a development environment. Not due to lack of trying. The biggest issue is dealing with permissions. And in a production setting, yikes

http://drupal.org/node/762138

Once you convinced yourself to use capistrano, who cares about aegir? We switched over to virtualmin. I hacked together a capistrano drush make module in 2 hours. Not great but it works.

https://github.com/xforty/capistrano-drush-make

I just don't see what aegir has to offer beyond a traditional control panel. Unless you are going to do something like this

http://wedful.com/

Just seems to me there isn't any added value. Thoughts?

Ilari Mäkelä (not verified)

You should bother just to avoid the hacking. You are making build functionality which is just developed by your team. With Aegir there is the whole Aegir community behind it to aid in development of new features and finding bugs. That should be a enough big reason to use Aegir. And if you are comparing Aegir to a traditional control panel I must honestly say that comparison is a total miss.

Kris Buytaert (not verified)

You write "while still utilizing Aegir for creating, managing, and migrating sites. "

not even considering the security impact this has but .. if it is creating , managing etc vhosts and other config regarding those sites and you are already using Cap I must agree with @Christian Pears on why add AEgir to the mix.

If you'd add anything to that mix to me it would be a tool to do the actual config mgmt like Chef or Puppet or even CFEngine but I wouldn't even consider looking at AEgir again ..

Tom Friedhof

We are using puppet for general server configuration management, not so much for vhosts or deployment. I have considered just rolling my own automation with our current setup and just adding Puppet to the mix for handling setting up new vhosts. However, I have other reasons for wanting to use Aegir.

  • If on a project we have multiple feature branches branched from the master branch being worked on simultaneously, I want it super simple for any of our developers (or even a non-technical person) to deploy new instances of a site on a specific branch and create a new "dev" site for testing, without me needing to update a Puppet manifest.
  • We have a Drupal distribution that we're working on similar to the wedful site (but for another industry) that Christian Pearce mentioned, so I want to use something that will help launch instances super easy so that a non-technical person can handle deploys.
  • As Ilari Mäkelä mentioned, there is already a community around Aegir. Why not embrace the spirit of open source, and work with others to improve the process of making it super simple for non-technical users to deploy sites.

I think there is still a place for Aegir, even with all the other tools we technical people tend to lean toward.

Christian Pearce (not verified)

"You should bother just to avoid the hacking. You are making build functionality which is just developed by your team. With Aegir there is the whole Aegir community behind it to aid in development of new features and finding bugs. That should be a enough big reason to use Aegir. And if you are comparing Aegir to a traditional control panel I must honestly say that comparison is a total miss." @Ilari Mäkelä

Well the hack was to initially get something out the door. Since then we have put together a great set of dev tools for local development with vagrant and deployments with drush and capistrano. Not just for our team but for anyone else who wants to use them.

http://github.com/xforty/vagrant-drupal
http://github.com/xforty/drush-deploy

You say there is a whole Aegir community. Just didn't feel like. Seems the lastest releases are mostly nginx bugfixes. Which is fine if Aegir is your thing run with it. We weren't going to kid ourselves that Aegir was meeting our needs when it wasn't. So we did something about it.

I guess I wasn't so much trying to compare Aegir to traditional panels, I was simply indicating it didn't provide additional value for us. Just more pain.

It has been 6 months since I last posted. I just turned off my aegir instances yesterday. I have to say our process has improved, and our experience is better.

I guess being a developer and systems guy more than a Drupal guys the convention tools feel a whole lot better.

Add comment

Recent Posts

6.29.2012
By: Matthew Krick | 0 Comments
6.27.2012
By: Evan Jenkins | 4 Comments
3.14.2012
By: Tom Friedhof | 1 Comment
2.28.2012
By: Tom Friedhof | 9 Comments

From this Author

Twitter icon
Facebook icon
Flickr icon