Keep Your Source Clean: Don’t Worry About Generated Files

Don’t dirty your source and don’t worry about generated files.

Working with OpenSim recently, I didn’t want to have to worry about checking in generated files by accident, or filtering my git status output in order to determine what I changed and was generated. In this post, I’ll describe how I do this and outline some tools I use to make it easy.

The basic premise in my development style is that I build the project in a separate directory than where the checked out code lives:

# checked out code lives in ~/source/opensim
$ cd ~/source
$ svn co http://opensimulator.org/svn/opensim/trunk opensim

# the workspace is where I build and run the project
$ mkdir ~/source/workspace

Then, to build and run the code:

# first I rsync the code to the workspace, skipping .svn folders
# and other unneeded files
$ rsync -avz --exclude=\.svn [...] ~/source/opensim/ ~/source/workspace/

# then I run prebuild and compile
$ cd ~/source/workspace
$ mono bin/Prebuild.exe /target nant
$ nant

This way I never have to filter through generated files when inspecting or committing my changes, and I don’t commit generated files by mistake.

Tools

I have a set of scripts I use when developing OpenSim on Linux (I also use some of the scripts on Windows with cygwin). The scripts are available on github.

All those above commands are encapsulated in os_build.sh, a single script which takes two arguments, the first is the name of the directory in ~/source/ in which the source
lives, and the second is the name of the directory in ~/source/ in which to build:

# this one command performs the rsync, prebuild and nant steps from above
$ os_build.sh opensim workspace

Another key part is the os_clean_workspace.sh script which removes all build artifacts from the workspace directory, while keeping all configuration files in
place. I use this whenever I need to do a clean build.

My typical session looks like this:

$ cd ~/source/opensim
$ git svn rebase                   # pull the latest changes from upstream
$ vim -p file1 file2 ...           # open the files I'll be working on

... edit source ...

$ os_build.sh opensim workspace    # rsync, prebuild, build

... in another terminal, run & test ...
... make more edits ...

# sync only changed files, rebuild only what's needed
$ os_build_sh opensim workspace

... make big changes requiring clean build ...

# removes everything except config files
$ os_clean_workspace.sh workspace
$ os_build.sh opensim workspace

There are also other helper scripts for cleaning the database, running unit tests, and other tasks. Have a look!

git post-receive Hook

Notes on setting up automated tasks when commits are pushed to a central git repository.

At my job we use git for version control, and I wanted to do a few things whenever we pushed changes to the main tree. It took me a little bit of effort to figure out how it works, and below are my notes.

The post-receive hook is called when the repository receives changes. Let’s assume I have a git repository on my development machine that is a clone of a public git repository on a server. Then, when I run git push in my local repository in order to push my commits to the public repository, the post-receive hook is called on the public repository on the server.

Email Notification

On Debian-based systems there is an example email notification post-receive hook in /usr/share/doc/git-core/contrib/hooks/post-receive-email. It’s also available online from the git repo on github in the contrib/hooks directory.

In the repository that will be receiving commits (the public repository on the server, in the example above), first we need to make the post-receive hook executable:

$ cd /path/to/git/repo/.git/hooks
$ chmod +x post-receive

Then we need to call the post-receive-email script from inside this post-receive hook. Open .git/hooks/post-receive with your favorite editor and make sure this line exists:

. /path/to/post-receive-email

Notice we’re not executing the post-receive-email script, we’re sourcing it.

Next, we need to tell git where to send the email to. The hooks.mailinglist property controls this:

$ cd /path/to/git/repo
$ git config hooks.mailinglist "mailinglist@example.com"

At this stage emails will be sent each time changes are pushed to the public git repository containing a summary of the new changes.

By default, emails refer to the project as “UNNAMED PROJECT”. To change this, edit .git/description.

For further customization options, look inside post-receive-email. For example:

$ cd /path/to/git/repo
# envelopesender? see `man sendmail` for the -f switch
$ git config hooks.envelopesender "git@example.com"
$ git config hooks.emailprefix "[MY PREFIX] "  # note the trailing space

Kicking Off a Buildbot Build

You may also want to have Buildbot start a build from the post-receive hook. Like the email script, the Buildbot git script expects revision hashes on standard input. The post-receive script then needs to be modified a little to pass the hashes it receives via standard input to all the scripts it invokes. Here’s a new copy of post-receive:

while read oldrev newrev refname
do
   echo $oldrev $newrev $refname | /bin/sh /path/to/post-receive-email
   echo $oldrev $newrev $refname | /usr/bin/python /path/to/git_buildbot.py
done

This script first reads oldrev, newrev, and refname from standard input, then pipes those values into post-receive-email and git_buildbot.py. We loop until there is no more input, presumably because another push may happen while we’re executing.