Deploying to a Fleet of App Servers Using Fabric and Elasticbeanstalk

Periklis Sakkaris, March 25 2016

Versioning

The first thing you will have to decide is how to version your deployments. In previous Java services I worked on, the common pattern was to use some variant of the Maven Versioning Strategy

I cannot express how many needless headaches this strategy has caused me. The main reason for the headaches is that the maven strategy is fine for an open source library such as the AWS Java SDK, but I think this is a bad strategy if you are trying to version a microservice.

Our service is in version 1. That means our REST api has “v1” in the url calls and that is what our service users want or need to know. That means I can can just call my warfile [service-name]-v1.war and that tells me which major version the service I am in. Simple.

Then, to distinguish states of the v1 service we use the first 5 characters in the corresponding Git commit. For example, If I wish to deploy code that has a git commit “72209e15922f1e18676ae24cbc9e3fbb81b8b453” then my version of the service is “v1-72209”

Simple and works better then maven versioning! Why? Because when I look at the deployed version I immediately know what code is running. If I had a version of “1.12.33” this tells me nothing about the latest code that is running. Also, suppose you need to branch of that version and fix some bugs? Have fun finding which git commit corresponds to version “1.12.33”

Also, we have no need for SNAPSHOT version. Code in each major version such as v1 must be backwards compatible for our microservice users. Code that is in development should be on a developer branch. When you merge to the master branch you must expect your code to be deployable right now. If it is not, keep working on your branch until you are ready for deploying.

Anyway, hopefully I have convinced you to version using the actual major version of your service and the git commit hash.

DevOps Machine

You want to setup a server which will be designated as a devops server. Assign an Elastic IP to this server and have it part of your VPC which your app servers are running on. Allow SSH access. Configure your GitHub repo to use Deploy Keys Use a passphrase when generating the SSH key for added security. When you deploy via fabric, fabric will ask for your passphrase and pass it to the devops server when a pull is performed.

Configure the devops machine to use the AWS command line tools.

Finally, create a machine image so you don’t have to do this again when the server crashes.

Elasticbeanstalk Basics

Here are some basics to get you started with Elasticbeanstalk terminology. The main characters are applications and environments. Application is your microservice, for example, it could be your REST API for your notifications microservice. Environment is a specific configuration of your Application, for example, production, staging and test. We use Elasticbeanstalk applications and environments to push our code changes.

There are many benefits. Here are our top benefits for using Elasticbeanstalk

  • Auto Scaling: we set up auto scaling triggers to automatically add App Servers with our latest service code when traffic spikes

  • Hot Deploys: we deploy new code 1 app server at a time and incrementally switch traffic to the new servers. This enables us to keep the service running while deploying our code and when there is an error to immediately rollback the change.

  • Rollbacks: Elasticbeanstalk saves application versions by pointing to S3 files, thus enabling us to use the management console to revert to specific application versions easily.

  • Containers: Elasticbeanstalk supports containers. You can use pre-built containers like Java 8 / Tomcat 8 on Linux or roll your own with Docker.

Fabric Script

   
from fabric.api import *
import fabric.utils
import json

env.user = "[your-devops-server-user]"
env.hosts = ["[your-devops-ip-address]"]
env.key_filename = ["~/.ec2/[your-key-file].pem"]

code_dir = "/home/ubuntu/[where-your-code-is]"

def local_uname():
   local('uname -a')

def remote_uname():
   run('uname -a')

@serial
def deploy(env_name="platform-stg", branch="master"):
   with cd(code_dir):
       fabric.utils.puts("PULLING FROM GIT BRANCH: %s" % branch)
       run("git checkout %s" % branch)
       run("git pull")
       fabric.utils.puts("ATTEMPT TO DEPLOY TO: %s" % env_name)
       build_label = run("git rev-parse --short=5 HEAD")
       description = run("git log -1 --pretty=oneline --no-color", pty=False)

       platform_info = get_platform_info(env_name)
       deployed_version = platform_info["VersionLabel"]
       fabric.utils.puts("deployed version: %s" % deployed_version)
       if build_label != deployed_version:
           run("mvn clean package")
           with cd("target"):
               war_file = "%s.war" % build_label
               run("aws s3 cp yumavore-platform-v1.war s3://[your-elasticbeanstalk-bucket]/platform/%s" % war_file)
               run("aws elasticbeanstalk create-application-version --application-name platform --version-label %s --description \"%s\"  --source-bundle S3Bucket=[your-bucket],S3Key=platform/%s" % (build_label, description, war_file))
               run("aws elasticbeanstalk update-environment --environment-name %s --version-label %s" % (env_name, build_label))
       else:
           fabric.utils.puts("LATEST GIT VERSION ALREADY DEPLOYED. SKIPPING")


def get_platform_info(env_name):
   json_result = run("aws elasticbeanstalk describe-environments --environment-names %s" % env_name)
   platform_info = json.loads(json_result)
   return platform_info["Environments"][0]

 

You can learn the basics of fabric here. Here is an overview of what our deploy script does.

It deploys a Git Branch to and Application Environment. The Environment defaults to staging and the Git Branch defaults to master but these can be overwritten on command line.

  • We grab the first 5 character of the latest Git commit and make that our build label, a.k.a, version

  • We grab the description of the latest Git commit and make that our build description

  • We check that the the Git commit is different from the current deployed version

  • We package our application into a war file using maven. This re-runs the unit tests to make sure everything is ok

  • We copy the resulting war file to S3

  • We create an application version on Elasticbeanstalk pointing to the S3 file we uploaded

  • We tell Elasticbeanstalk to update our environment to use the new version

This system drastically improves our devops experience and has lead to quick deploys, easy rollbacks, and auto scaling of App Servers. Note that this requires that you use a microservices methodology of architecting your platform. Each microservice team would use this method separately and modify to suit their specific needs.

Happy Scaling!