Paths in OSX 10.6 (Snow Leopard)

I’ve recently being doing a cleanup of my dotfiles (more on that later). In the process I commented out all of the “export PATH=...” statements in the profile and rc files in my home directory, but my path was still getting set to this:


There are three problems with this:

  1. I’d prefer my /usr/local/bin to be first
  2. I don’t want/need either of those git/x11 paths
  3. I don’t like not understanding where things are getting set!

I eventually worked out that it came down to some files in my /etc directory :


The paths.d method actually makes a lot of sense for a certain kind of application: it’s a nice clean way of adding bin files to the path nice and early (before any local modifications to the path).

Here’s the full order in which my paths were getting loaded in (I’m using zshell):

~/.zshrc (which loads in ~/.profile)

Apparently /etc/profile can come into play as well, but mine doesn’t set the path so I’m not sure what the ordering is. Indeed it looks like /etc can have it’s own set of shell-specific configs as well (mine seems to have a default bashrc), but that’s a whole other ballgame.

My solution

Now that I properly understand how my path is getting generated, I’m comfortable setting it from scratch in my local profile:

export PATH=/usr/local/bin:/usr/bin:/usr/sbin:/bin:/sbin

I prefer to have /usr/local/bin first, because this means that I can install things in /usr/local and symlink them into /usr/local/bin without conflicting with any old versions which came bundled with the operating system (and thus live in /usr/bin)

I could have achieved the same effect by editing /etc/paths, but I’d like to try and make my dotfiles as portable as possible

The following posts really helped me understand all this:


Removing duplicated rvm lines from zsh config files

I’ve just been cleaning up my zshell config files. I noticed that the following lines were at the end of both my .zshrc and .zprofile files:

# Add RVM to PATH for scripting
[[ -s "$HOME/.rvm/scripts/rvm" ]] && source "$HOME/.rvm/scripts/rvm" # Load RVM into a shell session *as a function*

I’m not sure how I got into that situation, but this duplication was an itch which had to be scratched.

Removing this from .zshrc gives the following error when trying to use rvm:

Warning! PATH is not properly set up, '/Users/duncanstuart/.rvm/gems/ruby-2.0.0-p247/bin' is not at first place,
usually this is caused by shell initialization files - check them for 'PATH=...' entries,
it might also help to re-add RVM to your dotfiles: 'rvm get stable --auto-dotfiles',
to fix temporarily in this shell session run: 'rvm use ruby-2.0.0-p247'.

This is fair enough: for reasons I don’t fully understand, rvm needs a bunch of paths to be at the very beginning of PATH – that’s why the lines in question need to be at the end of the .zshrc file. Looking at my path (echo $PATH), those paths are in the middle somewhere – presumably being added by .zprofile. No good.

On the other hand, removing this from .zprofile gave me a couple of  errors from rvm on opening a new terminal:

/Users/duncanstuart/.rvm/scripts/initialize:42: __rvm_cleanse_variables: function definition file not found

…and also something like curl (6) Couldn't resolve host '' (I’ve lost that actual error message).

A search on stackoverflow suggested the thing to do was to remove  ~/.zshcompdump

As far as I can understand from the zshell docs, this is something to do with autocompletion:

To speed up the running of compinit, it can be made to produce a dumped configuration that will be read in on future invocations; […] The dumped file is .zcompdump in the same directory as the startup files

Fine – sounds very deletable. And indeed, deleting it removes the errors, so I can happily leave those lines out of my .zprofile. Also, as expected, a fresh .zcompdump got generated.

One final thing: I noticed that opening a new terminal was taking slightly longer than before. Compiling the new .zcompdump seems to shave off a couple of milliseconds and makes it feel nice and responsive again:

% zcompile .zcompdump

Caching with Varnish on Heroku (Rails)

I’ve had a bit of trouble understanding how to get Heroku to cache pages from Swing Out London, so here’s my explanation – hopefully it’ll help someone.


Varnish Caching on Heroku IS as simple as the docs state but because Heroku has multiple Varnish servers, any given request is unlikely to return a cached copy unless your site has a lot of traffic, or you set max-age to a large number.

The Basics

Some of the Heroku stacks (currently Aspen and Bamboo) support basic caching using Varnish – a service which sits in front of your application server and serves up cached content. It essentially works in the same way as your browser cache, except that when the varnish cache serves up your content, it

Telling Varnish to cache your content is the same as telling a user’s browser to cache content for a limited time:

class WebsiteController < ApplicationController

  def index
    response.headers['Cache-Control'] = 'public, max-age=300'

In this case it says “this page won’t change in the next 5 minutes (300 seconds), so you don’t need to request it again from the App”.

For further info, check the Heroku docs.

The Problem

When I first tried this, it didn’t seem to be working at all – the response header was being set on the page, and Varnish was being used, but the request was reaching the application. Here are some example http headers:

HTTP/1.1 200 OK
Server: nginx
Date: Mon, 14 May 2012 14:59:14 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Cache-Control: public, max-age=1200
Etag: "913fba41febb1c5111ac375f09e88f46"
X-Ua-Compatible: IE=Edge,chrome=1
X-Runtime: 7.055496
Content-Length: 45804
Accept-Ranges: bytes
X-Varnish: 878887772
Age: 0
Via: 1.1 varnish

Note that Age is 0, indicating that this is a fresh page retrieved from the application rather than the cache. Also worth noting is that the X-Varnish header is the ID that Varnish assigns to the request.

After making about 10 requests I found one which was returned from the cache:

HTTP/1.1 200 OK
Server: nginx
Date: Mon, 14 May 2012 14:59:17 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Cache-Control: public, max-age=1200
Etag: "913fba41febb1c5111ac375f09e88f46"
X-Ua-Compatible: IE=Edge,chrome=1
X-Runtime: 1.824720
Content-Length: 45804
Accept-Ranges: bytes
X-Varnish: 793821349 793811181
Age: 40
Via: 1.1 varnish

This says “This page is 40 seconds old, and was originally cached as part of request 793811181”.

So it seemed like Varnish was caching, but only intermittently. A number of other people seemed to have the same problem.

The explanation

It looks like Heroku has multiple Varnish instances running independently, so when you request a page you could be hitting any of those instances. Here’s a likely scenario for a site with low traffic:

Let’s assume there are 3 varnish instances: A, B and C (Heroku maybe has 10) and represent the age of the cached pages on those instances as [a, b, c]. Now if the max-age is set to 25 seconds, and I make a number of repeated requests, I might see the following:

  1. [-, -, -] On the first request I get sent to A. A doesn’t have a cached copy so requests the page from the server and caches it:
  2. [10, -, -] Ten seconds later I make another request and get sent to B. Ditto.
  3. [20, 10 , -] Ten seconds later I make another request and I get sent to C. Ditto.
  4. [30, 20 , 30] Ten seconds later I make another request and I get sent to A again, it has a cached copy, but it’s older than the max-age so I discard it and request the page again from the server.

If on the other hand those requests were made every 5 seconds (i.e. if the site had higher traffic) then on the fourth request the cached copy would only be 15 seconds old so would be returned.

The solution

Increasing the max-age makes a big difference – I’ve set mine to 20 minutes for the time being. In slow periods not many pages will be served from the cache, but when traffic increases, the number of requests to the server will remain relatively similar.

The thing to understand is this: if the number of varnish instances is N and you’ve set the max age to A, then of the requests made in any period of A seconds, on average N of them will hit your server.

Many thanks to Garry Shulter for helping me understand.

Caveat: my understanding may not be entirely correct – if I’ve made any mistakes, please let me know in the comments!


For one week only: free analyst/developer in London


After following up a couple of possibilities I’ve decided to go and work with New Bamboo for the week to see what value I can add as a Business Analyst. I’m looking forward to working in a very fast-paced agile environment – somewhat different to the projects I’ve been on recently and I’ll report back what I’ve learned.

It’s been extremely interesting talking to the guys at GoSquaredWikiJob and Pusher and getting an insight into what their priorities are and how they work.

Original post:

Calling all London-based Charities, Green or Social enterprises, Web agencies and Ruby-on-Rails development houses, I have a question for you:

If you had access to skilled analyst for a whole week and it cost you nothing, how would you use them?

I’m leaving my current Consultancy job and have taken some time off before starting my new position. In the first week of August (1st-5th) I’m looking to have some new experiences with an interesting organisation and I’m very much open to suggestions.

So what can I offer?

I’m a Generally Clever Chap – I’m all about dealing with complexity and solving hard problems.