Tag: Continuous Deployment

Continuous Deployment with Codeship and WP Engine

Overview of continuous deployment pipeline

The purpose of this article is to share our approach to continuous deployment for WordPress plugins and themes on our servers at WP Engine. Continuous deployment is the automated testing and delivery of code to development and production environments. Knowing your web host’s requirements is essential to setting up a continuous delivery process. Once you know this, then you can evaluate a service and later develop a solution that meets your needs. I have implemented this for more than a dozen WordPress themes and plugins, and I believe that effort was worth the time it is saving us now.

Video Tutorial

Our Approach

Codeship automates the process of testing, building, and deploying code to your server, and was the first such product I found which would integrate with our web host (WP Engine). It watches a repository, clones an updated branch to a virtual machine, runs your commands on it, and then deploys the result to your servers. Services like Codeship allow us to automate our existing development and deployment workflow so that changes to the code in our Git repositories are automatically made on our servers.

We use three custom bash scripts in our Codeship projects: setup, build, and deploy. The setup script runs first and is mostly identical for all branches:

# Setup
# Get around restriction on shallow pushes by WP Engine
git filter-branch -- --all
git checkout branchname
# Add User Data
git config --global user.name "codeship-shortreponame"
git config --global user.email "youremail@email.com"
# Combine remote git servers
git remote add servers $SERVERNAME_ENVIRONMENT
git remote set-url --add --push servers $SERVERNAME_ENVIRONMENT
git remote set-url --add --push servers $SERVERNAME2_ENVIRONMENT
# Install needed modules
npm install -g grunt-cli
npm install -g grunt@0.4.0
npm install -g bower
npm install -g ruby
gem install compass
# Move repo files to a named folder
mkdir $FOLDERNAME
shopt -s extglob
mv !($FOLDERNAME) $FOLDERNAME
# Move repo files whose name begins with a period
mv .sass-lint.yml $FOLDERNAME/.sass-lint.yml
# Exclude development-only files from commit
rm .gitignore
mv .codeshipignore $FOLDERNAME/.gitignore
# Move named folder into a structure identical to the root directory of a WordPress server
mkdir -p $DIRECTORY
mv $FOLDERNAME $DIRECTORY
cd $DIRECTORY/$FOLDERNAME/

It adds git user data, defines a list of remote server repositories, and installs command line modules on the virtual machine. The staging servers are listed as remote git repositories in the staging branch’s setup script, and our production servers are listed as remotes in the master branch’s script. Then it replaces the repository’s gitignore file with our .codeshipignore file and moves all files into a folder structure relative to the root directory of our servers:

# RECOMMENDED BY WPENGINE
*~
.DS_Store
.svn
.cvs
*.bak
*.swp
Thumbs.db
# large/disallowed file types
# a CDN should be used for these
*.hqx
*.bin
*.exe
*.dll
*.deb
*.dmg
*.iso
*.img
*.msi
*.msp
*.msm
*.mid
*.midi
*.kar
*.mp3
*.ogg
*.m4a
*.ra
*.3gpp
*.3gp
*.mp4
*.mpeg
*.mpg
*.mov
*.webm
*.flv
*.m4v
*.mng
*.asx
*.asf
*.wmv
*.avi
# Directories that may or may not exist in repo AND should not be on the server
package.json
bower.json
.bower.json
.bowerrc
config.rb
node_modules
.sass-cache
*.md
*.txt
*.ai
*.scss
*.coffee
.gitignore
LICENSE
LICENSE-MIT
gruntfile.js
Gruntfile.js
werker.yml
.editorconfig
css/src
js/src
bower_components/**/foundation/scss/
bower_components/**/jquery/src/
bower_components/**/modernizr/*/
bower_components/**/modernizr/grunt.js
bower_components/**/picturefill/index.html
bower_components/**/picturefill/logos/*.png
bower_components/**/superfish/examples/
# Only using one file from bower_components/html5shiv
bower_components/html5shiv/
!bower_components/html5shiv/dist/html5shiv.js

Our typical build script imports Node and Bower modules and uses Grunt to compile Sass and Coffee files:

# Build
composer install
npm install
bower install
grunt develop

It uses the repo’s gruntfile (example) to perform a different set of tasks for the staging and production servers:

module.exports = (grunt) ->
  @initConfig
    pkg: @file.readJSON('package.json')
    watch:
      files: [
        'css/src/*.scss'
      ]
      tasks: ['sasslint', 'compass:dev']
    compass:
      pkg:
        options:
          config: 'config.rb'
          force: true
      dev:
        options:
          config: 'config.rb'
          force: true
          outputStyle: 'expanded'
          sourcemap: true
          noLineComments: true
    jsvalidate:
      options:
        globals:
          jQuery: true
          console: true
          module: true
          document: true
      targetName:
        files:
          src: [
            'js/*.js',
            'bower_components/foundation/js/vendor/fastclick.js',
            'bower_components/foundation/js/foundation/foundation?(.topbar).js',
            'bower_components/modernizr/modernizr.js',
            'bower_components/jquery/{dist,sizzle}/**/*.js',
            'bower_components/jquery-placeholder/*.js',
            'bower_components/jquery.cookie/jquery.cookie.js',
            'bower_components/respond/{cross-domain,dest}/*.js',
            'bower_components/html5shiv/dist/html5shiv.js'
          ]
    sasslint:
      options:
        configFile: '.sass-lint.yml'
      target: ['css/src/**/*.s+(a|c)ss']
    compress:
      main:
        options:
          archive: 'AgriFlex3.zip'
        files: [
          {src: ['AgriFlex/*.php']},
          {src: ['css/*.css']},
          {src: ['img/**']},
          {src: ['js/*.js']},
          {src: ['bower_components/fastclick/lib/fastclick.js']},
          {src: ['bower_components/foundation/{css,js}/**']},
          {src: ['bower_components/modernizr/modernizr.js']},
          {src: ['bower_components/jquery/{dist,sizzle}/**/*.js']},
          {src: ['bower_components/jquery-placeholder/*.js']},
          {src: ['bower_components/jquery.cookie/jquery.cookie.js']},
          {src: ['bower_components/respond/{cross-domain,dest}/*.js']},
          {src: ['bower_components/html5shiv/dist/html5shiv.js']},
          {src: ['vendor/**', '!vendor/composer/autoload_static.php']},
          {src: ['functions.php']},
          {src: ['README.md']},
          {src: ['rtl.css']},
          {src: ['screenshot.png']},
          {src: ['search.php']},
          {src: ['style.css']}
        ]

  @loadNpmTasks 'grunt-contrib-compass'
  @loadNpmTasks 'grunt-jsvalidate'
  @loadNpmTasks 'grunt-contrib-watch'
  @loadNpmTasks 'grunt-sass-lint'

  @registerTask 'default', ['compass:pkg']
  @registerTask 'develop', ['sasslint', 'compass:dev', 'jsvalidate']
  @registerTask 'package', ['compass:pkg', 'jsvalidate']

  @event.on 'watch', (action, filepath) =>
    @log.writeln('#{filepath} has #{action}')

Note that we use Compass’s expanded output with sourcemaps for developing Sass files and a compressed output for production. A web browser’s inspection tool can detect sourcemaps and use them to show you where a CSS rule is located in your Sass files. This is essential for troubleshooting or modifying Sass as quickly as possible.

Once the build script has finished, the deploy script commits changes to the cloned repository and pushes them to our WP Engine servers:

# Deploy
git add --all :/
git commit -m "DEPLOYMENT"
git push servers HEAD:master --force

If this is the first time you’re using the Git Push feature for a particular WP Engine install, you might need to use this git push command instead to avoid a “missing branch” error:

git push servers HEAD:refs/heads/master --force

We can only send code to WP Engine servers from Codeship using their git push feature. It replaces user input authorization with an SSH key. The .codeshipignore file allows us to exclude unneeded files from the commit and reduce upload times. Since our projects are authorized git push users with the WP Engine servers we push to, the last command pushes our final commit to the remote server repositories we defined during setup.

The video tutorial demonstrates how to implement this process. What follows are the same steps I take to connect each of our repositories to our WP Engine servers using Codeship.

Using Codeship to deploy Github repos to WP Engine

  1. Create a new Codeship project and connect it to a Git repo URL
  2. On the Codeship General Project Settings page, copy the SSH public key
  3. On each WP Engine install you want connected, use the SSH public key to add a new Git Push developer named codeship-shortreponame.
  4. Show what the variables page looks like On the Codeship project’s Environment settings page, create a new variable named SERVERNAME_PRODUCTION and SERVERNAME_STAGING for each WP Engine install you want connected.
  5. On the same page, define a FOLDERNAME variable as the name of the folder your repo’s files should be placed within. Also define a DIRECTORY variable as either wp-content/plugins or wp-content/themes.
  6. Show what the deployment pipeline page looks likeOn the Codeship project’s Deployment settings page, create one pipeline for the repo’s master branch and another for the staging branch.
  7. Create separate custom scripts in each pipeline for the setup, build (optional), and deploy portions of the process.
    • The setup script only has a few variables that need to be changed for each project: branchname, shortreponame, youremail@email.com, and SERVERNAME_ENVIRONMENT.
    • The build script contains your existing build process, such as importing NPM modules or preprocessing css and js files.
    • The deploy script has no variables.
  8. Ensure your Git repo has a master and staging branch.
  9. Checkout the staging branch of your repo and add a .codeshipignore file to its root directory using this template as a guide.
  10. Show what the build output page looks likeCommit the change and push it to your repository. This will trigger the Codeship build process, which you can watch on the project’s main page as it moves through each line of your custom scripts.
  11. Once the build has finished, it will send the code to the staging server if successful or stop on the first line that results in an error. If it is successful, it will be pushed to the server! Whether or not it is successful, you can click on any line of the script to see its output, so if it fails you can use this to troubleshoot that command.
  12. When your build is successful, you can merge the staging branch to the master branch on your computer and then push it to the repository. This will trigger the master pipeline, which will either deploy it to your production server or result in an error.

And that’s it! Now you have an automated deployment process integrated with your existing Git repository, and all of your servers can have the latest, tested version of your plugins and themes each time you push to it. If you found this article helpful, please leave a comment. You can view the public repositories of Agrilife Communications on Github for more information about their build processes. I can be reached by email at zachary.watkins@ag.tamu.edu.

Resources

Continuous Release on Github with Grunt

TL:DR; Just show me the Gruntfile!

This post will describe the release tasks I’ve written for Grunt which work seamlessly with our existing continuous deployment pipeline on Codeship. I also mention a step you can take if you want to do this directly from your command line instead.

I help develop WordPress themes and plugins for Texas A&M AgriLife, which are available on Github. Earlier this year, I was asked to develop and implement a continuous deployment pipeline for most of our repositories. I wrote an article about it here. Long story short, we use Codeship to watch our repository branches (staging and master) and run console commands to prep and push them to the staging and production versions of our servers.

I was recently asked to create a release for our latest theme AgriFlex3, and in hope I looked for articles on how to do so with Grunt. I soon found this article by Paul C. Pederson which showed me this approach was possible, but I needed something a little different. What follows are Grunt modules and commands based on those I use to package and release our open-source projects on Github with a release message that describes all of the commits made since the previous release. These tasks can be run for any project, not just WordPress!


grunt-contrib-compress

This is the simplest task. You provide the archive file name and an array of file globs to copy into it. We try to use values from package.json when we can. View the grunt-contrib-compress repository for more information.

compress:
  main:
    options:
      archive: '<%= pkg.name %>.zip'
    files: [
      {src: ['css/*.css']},
      {src: ['img/**']},
      {src: ['js/*.js']},
      {src: ['**/*.php']},
      {src: ['*.php']},
      {src: ['README.md']},
      {src: ['screenshot.png']}
    ]

grunt-gh-release

This task creates our release using properties within package.json, a Github access token available in our Codeship environment, and a few custom tasks (shown in the next section) that populate the release message beforehand. If you want to run your release task locally then use the documentation’s method for the options token instead of what is shown here. View the grunt-gh-release repository for more information.

gh_release:
  options:
    token: process.env.GITHUB_PERSONAL_ACCESS_TOKEN
    owner: 'agrilife'
    repo: '<%= pkg.name %>'
  release:
    tag_name: '<%= pkg.version %>'
    target_commitish: 'master'
    name: 'Release'
    body: 'First release'
    draft: false
    prerelease: false
    asset:
      name: '<%= pkg.name %>.zip'
      file: '<%= pkg.name %>.zip'
      'Content-Type': 'application/zip'

Custom tasks

The two custom tasks I’ve written will run console commands and handle their output. The first task gets the last release tag and uses it to set a config value as the range of commit messages we want to retrieve. The second task takes this range and uses it to populate the release message with a bulleted list of commits below their author’s name. This is necessary because the github token is linked to a person and that name shows near release messages. We want to ensure they are not seen by users as the author of all released commits.

@registerTask 'setreleasemsg', 'Set release message as range of commits', ->
  done = @async()
  grunt.util.spawn {
    cmd: 'git'
    args: [ 'tag' ]
  }, (err, result, code) ->
    if(result.stdout!='')
      # Get last tag in the results
      matches = result.stdout.match(/([^\n]+)$/)
      # Set commit message timeline
      releaserange = matches[1] + '..HEAD'
      grunt.config.set 'releaserange', releaserange
      # Run the next task
      grunt.task.run('shortlog');
    done(err)
    return
  return
@registerTask 'shortlog', 'Set gh_release body with commit messages since last release', ->
  done = @async()
  grunt.util.spawn {
    cmd: 'git'
    args: ['shortlog', grunt.config.get('releaserange'), '--no-merges']
  }, (err, result, code) ->
    if(result.stdout != '')
      # Hyphenate commit messages
      message = result.stdout.replace(/(\n)\s\s+/g, '$1- ')
      # Set release message
      grunt.config 'gh_release.release.body', message
    else
      # Just in case merges are the only commit
      grunt.config 'gh_release.release.body', 'release'
    done(err)
    return
  return

Gruntfile.coffee

module.exports = (grunt) ->
  @initConfig
    pkg: @file.readJSON('package.json')
    compress:
      main:
        options:
          archive: '<%= pkg.name %>.zip'
        files: [
          {src: ['css/*.css']},
          {src: ['img/**']},
          {src: ['js/*.js']},
          {src: ['**/*.php']},
          {src: ['*.php']},
          {src: ['README.md']},
          {src: ['screenshot.png']}
        ]
    gh_release:
      options:
        token: process.env.GITHUB_PERSONAL_ACCESS_TOKEN
        owner: 'agrilife'
        repo: '<%= pkg.name %>'
      release:
        tag_name: '<%= pkg.version %>'
        target_commitish: 'master'
        name: 'Release'
        body: 'First release'
        draft: false
        prerelease: false
        asset:
          name: '<%= pkg.name %>.zip'
          file: '<%= pkg.name %>.zip'
          'Content-Type': 'application/zip'

  @loadNpmTasks 'grunt-contrib-compress'
  @loadNpmTasks 'grunt-gh-release'

  @registerTask 'release', ['compress', 'setreleasemsg', 'gh_release']
  @registerTask 'setreleasemsg', 'Set release message as range of commits', ->
    done = @async()
    grunt.util.spawn {
      cmd: 'git'
      args: [ 'tag' ]
    }, (err, result, code) ->
      if(result.stdout!='')
        # Get last tag in the results
        matches = result.stdout.match(/([^\n]+)$/)
        # Set commit message timeline
        releaserange = matches[1] + '..HEAD'
        grunt.config.set 'releaserange', releaserange
        # Run the next task
        grunt.task.run('shortlog');
      done(err)
      return
    return
  @registerTask 'shortlog', 'Set gh_release body with commit messages since last release', ->
    done = @async()
    grunt.util.spawn {
      cmd: 'git'
      args: ['shortlog', grunt.config.get('releaserange'), '--no-merges']
    }, (err, result, code) ->
      if(result.stdout != '')
        # Hyphenate commit messages
        message = result.stdout.replace(/(\n)\s\s+/g, '$1- ')
        # Set release message
        grunt.config 'gh_release.release.body', message
      else
        # Just in case merges are the only commit
        grunt.config 'gh_release.release.body', 'release'
      done(err)
      return
    return

 

If this helps you, or if you have any suggestions for improvements, please leave a comment!