On the heels of my post yesterday, I want to share a strategy I’ve recently started using to manage third party dependencies with Subversion. In my code, I rely on many packages like jQuery, Dojo and FCKEditor. It’s easy to fall behind as they release new versions, especially when you have to overlay their code into your directories. Messy.
What I was doing
Up until recently, I was using svn:external to pull a release of Dojo into my code during deployment. It works well during development but it creates a major, semi-dangerous deployment dependency when rolling out your code to production servers.
Creating a “Local External”
My new approach is using a local svn:external. I’ve created a new directory in my repository called “externals” and then a subdirectory for each third-party package I’m leveraging. Under that directory I include a subset of the common directories: tags and trunk. It looks like this:
/repo
/repo/externals
/repo/externals/fckeditor/
/repo/externals/fckeditor/trunk (svn:external to http://svn.fckeditor.net/FCKeditor/tags/2.4.3/)
/repo/externals/fckeditor/tags/2.4
/repo/externals/fckeditor/tags/2.4.1
/repo/externals/fckeditor/tags/2.4.2
/repo/externals/jquery
/repo/externals/jquery/trunk (svn:external to http://jqueryjs.googlecode.com/svn/tags/1.2.1)
/repo/externals/jquery/tags
/repo/externals/jquery/tags/1.2
/repo/externals/transfer
/repo/externals/transfer/trunk (svn:external to http://svn.riaforge.org/transfer/trunk/)
/repo/externals/transfer/tags
/repo/externals/transfer/tags/0.6.3
/repo/externals/transfer/tags/0.6.3.367
/repo/core/transfer (svn:external to /repo/externals/transfer/tags/0.6.3.367)
/repo/core/mydir1
/repo/core/mydir1/jquery (svn:external to /repo/externals/jquery/tags/1.2)
/repo/core/mydir2
/repo/core/mydir2/fckeditor (svn:external to /repo/externals/fckeditor/tags/2.1)
And so on. Basically, it’s an intelligent cache between our core application and the remote subversion repositories. Now there are three ways to track remote software depending on your preference. The first two rely on Subversion access; the last is for distributed software in a zip or tar file:
- Track the trunk. Set the /trunk with svn:external to the trunk of the remote repository and update to the latest and greatest with a simple “svn update”. This is working well for new projects like Transfer which are committing important updates to source control but aren’t releasing new packages yet. I use FileSync in Eclipse to synchronize these files with my development directory for the external (like, /repo/core/transfer) so while developing I’m using the very latest. When I’m ready to deploy, I export and commit the /repo/externals/transfer/trunk to a tags dir like /repo/externals/transfer/tags/0.6.3.367 (where 367 is the version number from Transfer’s SVN repo). This is your bleeding edge trick.
- Track the release. For more stable packages my svn:external points to a specific release. For jQuery, it might point to http://jqueryjs.googlecode.com/svn/tags/1.2. As I work with the specific tag, and validate that it’s working in my environment, then I commit it to my local repository with the same tag name.
- Non-Subversion Access. If all you have is a zip or tar file of the software, you can still use this method effectively. Start by completely deleting the contents of the /trunk directory and then untar/unzip the latest, re-adding and re-committing and then copying to a new tags directory. I would do this in case the release removes files from the distribution. This way you’re always 100% in sync with the gold release.
Now with our core application using svn:externals that point to our /externals/package/tags/x.x.x directories, we never have to worry about a remote repository being unavailable when we want to deploy (not to mention our local repo is about 10x faster than the remote). It also gives us a little more flexibility in choosing between versions of the software than “svn switch” and rolling back and forward as needed.
Rolling back
Let’s say I deploy and something is screwed up (despite my thorough testing! 
); rolling back is as easy as reverting the svn:externals in my main application to the previous tag (say changing /repo/core/mydir2/fckeditor to point to /repo/externals/fckeditor/2.4.1 instead of 2.4.2). While you can roll back using subversion in many ways, this limits the scope of the roll back specifically to the third party package.
Testing new releases
Now let’s say jQuery releases version 2.0 and I want to see what happens to my application. If my jQuery external is already tracking the trunk, I just run “svn update”. Otherwise it’s a simple “svn switch” to the 2.0 tag. Let the update finish, commit and copy them to a local tag: /repo/externals/jquery/tags/2.0. That took all of thirty seconds. Now in my main application, /repo/core/mydir1/jquery, I will change the svn:external to point to my new 2.0 tag and run “svn update”. Done. I’m now working with 2.0. I can test, decide if 2.0 is compatible with my existing code (or fix it) and then either commit or revert my svn:external based on my results.
Running multiple versions side-by-side
Say you’ve built a widget using Dojo 0.4 and you want to build another widget using the latest and greatest 0.9, but, you don’t want to rebuild the first one. In this case, you might want to deploy both versions of Dojo to your production server. Here’s where those tagged externals come in handy:
/repo/core/js/dojo-0.4 (svn:external to /repo/externals/dojo/tags/0.4)
/repo/core/js/dojo-0.9 (svn:external to /repo/externals/dojo/tags/0.9)
/repo/core/tools/somepage_that_uses_0.4.html
/repo/core/tools/otherpage_that_uses_0.9.html
Sweet – automatically handled when I “svn export” my /repo/core now.
You might wonder if it’s worth it given you have just one or two dependencies. I’d say take the plunge; it requires basically nothing more than an “svn move” to get started and it makes it so much easier to integrate third party packages that you’ll probably start doing more of it. I have about 7 dependencies at this point and I’m already seeing the benefits in just the first week of using it as Transfer revved 3 times.
Conclusion
Creating a Local External repository helps eliminate your dependency on remote subversion repositories during development and deployment. However, it also enables a new set of tight controls for developers specific to their third party dependencies.
Tell me if you try it!
UPDATED 10/15/2007 – Thanks to Ryan Wood for helping sort out my “tracking the trunk” scenario.
All Categories from Feedburner
Ryan said:
on October 15, 2007 at 7:34 am
That looks like a great idea. We’re running into the same issues with external libraries (transfer and coldspring oddly enough). I’m having trouble getting the trunk of transfer set up.
Do you put the svn:external ref (svn:external to http://svn.riaforge.org/transfer/trunk/ )to trunk on /externals/transfer? Or do you create the trunk folder and put it there?
If I do the former, I can’t point my core app’s /transfer to the svn:external [repos]/externals/transfer/trunk because the the trunk folder only exists when the working copy is checked out.
But with the latter way, you’d have to create another directory under trunk in your svn:external definition (“[some_folder] http://svn.riaforge.org/transfer/trunk“).
Feel free to email me offline if you’re willing to chat about your set up. Thanks for the post.
-Ryan
Ryan said:
on October 15, 2007 at 8:01 am
Wow, after reading my comment, I’m not sure that I made any sense.
What I’m asking is simply:
1. which directories in SVN are physically created?
2. On which directory are you putting the svn:external property (for transfer specifically)?
3. What is the name of your svn:external property? i.e.
“[name] http://svn.riaforge.org/transfer/trunk/”
If your setting it up the way your post reads, then you’ve set an svn:external on repos/externals/transfer to be “trunk http://svn.riaforge.org/transfer/trunk/“. Is that correct?
If so how can you reference one external from another?
i.e. /repo/core/transfer (svn:external to /repo/externals/transfer/trunk)
The /repo/externals/transfer/trunk folder doesn’t exist in the repository, but would be created on checkout?
Any help is appreciated?
brian said:
on October 15, 2007 at 9:32 am
@Ryan – Good questions, in looking back at my repo set up, I can see that I left out some details and glossed over an error that I need to fix. You’re right – you can’t just check out the trunk in the external and then have your core app get that, at least not when deploying.
What I left out is that I use FileSync in Eclipse so whenever I svn update I get my files moved into the right place in my development environment. However, that doesn’t cover the deployment scenario and now that I’m looking for my svn:externals for you, they seem to have gone missing. I think I broke something after I got it working so I will sort that out and then get back here to re-document it.
In a nutshell though, what you’ll want to do for deployment is probably svn export/copy the trunk from the remote repo to a /tags/ directory and make your svn:external reference that.
Using the FileSync plugin, you can make your working-copy use the trunk you’re tracking remotely but when you’re happy with it working the way it is, I think you have to copy it to your tags directory and release against that. Try that and tell me if it works; I’ll do the same here.
Mike Howarth: Web Developer» Blog Archive » Third party SVN imports said:
on October 23, 2007 at 3:50 am
[...] A bit on how people are using svn:externals [...]
brian said:
on October 23, 2007 at 9:18 am
Mike’s trackback had a good link to the SVN book that talks about “Vendor Branches” as an alternate solution to what I’ve proposed above. One aspect it discusses that I haven’t dealt with yet is making code changes to the 3rd party release and maintaining those changes across revisions. Interesting read at red-bean.com.