evil.che.lu

This article was written for the ruby advent 2008, day 20. I’m reposting it here so I have a backup under my own roof.

If you already know about vendoring and why it’s good for you, feel free to skip directly directly to the braid example#.

If you still need convincing, read on.

Vendoring theory

It’s a well-known fact that every piece of code that you write should be version controlled.

Not reinventing the wheel is also a good idea that will keep you happy. In practice, not reinventing the wheel means trying to use libraries or plugins written by other happy open source developers. This means that you now have external dependencies.

Generally, you want your project to be as stable as possible. However, you will also want to update your external dependencies quite often to make use of their new and shiny features.

The simplest way is to just make a note in the documentation and hope that all our users and contributors will setup things correctly. But this is an error prone procedure and it might be too much work for some to the extent that they might not bother.

Still, if all that you require from your external dependencies is an official release, most ruby libraries are released as a gem so, you could just use rubygems and be done with it.

Using rubygems is all fine until you need to make some custom changes to a library. These might be some changes you are not ready to submit to the original projects, or even changes that you submitted but have not yet been integrated. Or, maybe they don’t even make sense for anyone else but you.

Due to ruby’s nature, in some cases, you can avoid having all of the external code in your repository with some clever monkey-patching. But, if you have changes that are not trivial, sooner or later, you will need to bundle the external dependency with your project.

A simple and quick idea is to just take all the source code of your dependency form it’s repository, copy and dump it in a directory, remove the .git or .svn directories and simply commit it. This is a good enough starting point and it’s exactly what script/plugin does in rails. You are now free to tweak it to your heart’s content and then commit it to your own project’s repository.

Let’s say that you now vendored a few libraries and happily hacked on them and your project for a while. Suddenly, a security flaw is discovered in one of the external libraries. It’s authors are quick to provide a fix and you are keen to have it.

So, if you used the “copy and dump” approach, you’re kinda screwed, since you need both the security fix and your magic tweaks. You have to either manually merge the changes, discard your changes and start from zero with the latest version or, ignore the security issue altogether. I’m not sure which one is scarier :).

Let’s just assume that you definitely need both the security fix and all your custom changes. Basically, to continue, you have two options. You could either apply the new fixes on top of your code or you could reapply your fixes on top of the new library code.

• Applying new fixes on top of your code

All you need is a patchfile from the point where you vendored the dependency up to now. You’ll most likely have to create it by yourself. Then, you can try applying it on top of your code. If you want to go this route, I hope you noted that initial revision down somewhere or else, you’d have to guess and actually be lucky.

• Applying your custom changes on top of the most recent version

All you need for this is all of your custom changes. Then, you can simply remove your vendored code, “copy and dump” the new one in, commit, and reapply your changes. If you want to go this route, I hope you kept all your commits to the vendored code in separate patchfiles and are happy manually applying patches.

Vendor everything

Besides needing major custom changes, there are two other major reasons you might want to bundle your external dependencies.

One is that if you are using a new library that’s in active development, you will most likely need to update with bugfixes quite often.

The other is that if you are developing a web application that needs to be deployed, you will not want to depend on external repositories being down and preventing your deploy.

So, most of the time you will want to bundle external dependencies anyway, regardless of any custom changes you may or may not have.

Theory conclusion

If you made it this far, you should have a pretty good picture about your vendoring needs. I’ll just make this easy to skim by recapping things.

You don’t need vendoring if:

• You are writing a library that depends on another library. Just use rubygems. Examples of this are most gems like capistrano, braid, haml, etc.
• You are writing a library that depends on another library and needs minor tweaks. Just use rubygems and a small monkey-patch.
• You are writing an application that will be deployed and you don’t care if you can’t deploy :). Example: websites that fail to deploy.

You need vendoring if:

• You need close integration with an external library. Example: gitnub uses grit internally. So, it’s bundled.
• You need to update to new releases of the external library relatively often. Example: all rails applications that use plugins such as will_paginate, acts_as_list, etc
• You need to be able to deploy even if everything is down, except the production server :). If have vendored everything, you can still deploy form your local directory.
• You want other people working on the project to be able to quickly get started

So, basically, you need vendoring most of the time. There are several tools out there to help with this. Braid is the first one that was written especially to take advantage of git’s power. Keep reading to see how easy braid makes things for you.

Using braid to manage dependencies in git

Let’s go through a simple scenario to see how easy braid makes dependency management by using a completely silly example. You will notice that braid commands are very simple and scarcely used. To make them stand out, they will be highlighted.

The requirements are simple: we need to make an app that makes it clear that the hoff should not be hassled. Since this is a simple app, let’s use sinatra.

Okay, let’s start by initializing an empty git repository:

$ mkdir donthasslethehoff $ cd donthasslethehoff $ git init Initialized empty Git repository in /Users/chelu/tmp/donthasslethehoff/.git/ $ echo donthasslethehoff > README $ git add README $ git commit -m “initial commit” Created initial commit c6eeba4: initial commit 1 files changed, 1 insertions(+), 0 deletions(-) create mode 100644 README

To start, we’ll vendor sinatra using braid:

$ braid add git://github.com/bmizerany/sinatra.git

This will simply create a normal commit in your repository with all that is needed to make braid work. If someone clones your repository, they don’t have to do anything. They don’t need braid installed and don’t even need to know you’re using braid.

You will now have sinatra in a directory inside your app. Let’s add the code to display the message.

$ cat > hoff.rb $:.unshift File.dirname(FILE) + ‘/sinatra/lib’ require ‘sinatra’ get ‘/’ do “don’t hassle the hoff” end ^D

And, we’re done. Let’s try it out:

$ ruby hoff.rb

Load up http://localhost:4567/ in a browser, and see that it worked. All ok? Go ahead and commit our changes:

$ git add . $ git commit -m “not hassling the hoff”

Now, the site looks good but, our app must make it very clear that the hoff should not be hassled! So, let’s go ahead and change sinatra’s console messages to be more clear about this. This is one of those have to tweak a dependency situations I mentioned earlier.

So, go to sinatra/lib/sinatra.rb and search for the line that says: “has taken the stage”. Add a new line just below it:

puts “Sinatra says: Remember folks. Don’t hassle the Hoff!”

Now we’re talking. Everytime you start your sinatra app, you will be reminded not to hassle he hoff. Restart to see the new message and commit the change:

$ git add . $ git commit -m “not hassling the hoff hack”

You can now also use the braid diff command to see all of your custom changes to sinatra:

$ braid diff sinatra

After a while, you will want to use braid to update your sinatra mirror to get bugfixes and new features. Just make sure you don’t have any uncommitted local changes and run:

$ braid update sinatra

This will bring you sinatra goodies and still leave your modifications in place. Under the hood, braid will do some tricks so that the power of git is leveraged to merge the changes in smoothly. Again, the end result will be just a normal git commit.

Thanks

I hope this made it clear how easy and simple using braid is. You can find more examples on the braid wiki.

You can file bugs and feature requests at the braid project on lighthouse. And there is also a google group for discussions and community support: braid-gem google group.

Braid is hosted on github at http://github.com/evilchelu/braid/tree/master and we are always glad to receive patches and suggestions.

I just wrote an article for the ruby advent 2008 series.

It has more introductory details about vendoring and why you should do it. So, if you were not sure whether to braid or not, check out the Easy Dependency Management for Git with Braid article.

Braid is a simple tool to help track git and svn vendor branches in a git repository.

This is a major release that brings two main features (no more braid/track branch and better merging support) and lots of usability changes, bugfixes and tweaks.

The braid project page has now moved to github, mainly because we wanted to use the wiki for documentation.

Braid now also has a google group. If you have questions or ideas about braid, please subscribe to http://groups.google.com/group/braid-gem.

Installing

gem install braid

Important changes from 0.4

• removed braid/track branch
• use merge-recursive to allow changes from upstream to be automatically merged in
• added local cache to workaround git fetch limitations

Other changes from 0.4

• improved messages
• refactoring all around
• moved to test/spec from rspec
• basic integration tests

Coming soon in braid 0.6

• braid switch
• braid format-patch
• braid import
• braid outdated
• braid list
• bash completion
• possibly, windows compatibility

Braid is a simple tool to help track git and svn vendor branches in a git repository.

This release fixes a bunch of bugs and adds important missing features which allow braid to actually be useful to people working in teams and it also stops polluting the git history with messages from the imported mirrors.

This release is possible because of the work of Norbert Crombach, who totally rocks!

Important changes

• all new mirrors are created squashed, which means that you’ll only get one commit when adding a mirror
  • if you already have braid mirrors you need to update to squashed format, please read the relevant section on the Usage and examples page
• braid now can be used by all developers on a project. There’s a guide for this on the wiki.

Other changes

• braid diff
• braid update can now lock to a revision
• commands are only executed if there are no local changes and the git version is above 1.5.4.5
• bunch of bugfixes and cleanups

Install with rubygems

Unfortunately the github gem server is not working for everyone yet so please install from the git repository if this doesn’t work

gem sources -a http://gems.github.com/ # only need to do this once gem install evilchelu-braid

Get it from the git repository

Get a clone of the git repository using:

git clone git://github.com/evilchelu/braid.git cd braid rake install_gem braid —help # see usage

Open source works!

This release is very special for me because it showed me, in a personal way, that open source really works.

I was about to give up on braid at the end of March mainly because I had no idea on how to fix the history polluting and being able to work in teams. And there were also bugs and the code looked a bit ugly.

Fortunately, I didn’t shut it down because after about a month where I didn’t do any updates to braid, Norbert Crombach (github) told me he forked the project on github and fixed all the major outstanding issues! I was shocked and awed by his effort and, since all the major stuff is now fixed, I found new motivation to work on braid again. Thanks, dude! :)

We’ve been tweaking and cleaning up things for the last week to get this handy release out. We have ideas for a number of other features for the next releases so, head over to the braid project page on github and watch it to get updates. Github is great and it makes patching and merging so easy that we hope you’ll consider forking the project and contributing with code :).

If you got braid from the git repository before 12:40GMT, please pull the latest changes.

The 0.3.0 version has a bug where it will clone the mirrored svn repositories entierly. braid 0.3.1 just grabs the HEAD revision and this is significantly faster.

Changes

BUGFIXES

• braid was fetching entire repositories for svn mirros. changed to fetch only HEAD
• workaround for stupid git svn behaviour. it is transforming all _ (underscores) into . (periods) in git svn remote branch names
  • braid now is replacing all _ with – preemtively

BUGS

• braid only works with a quite recent git (I’m using latest from the git git repo). I know that git 1.5.2 definitely doesn’t work (git fetch is acting up). Not sure about versions in between.

Updating

cd braid git pull rake install_gem

Get it

git clone git://github.com/evilchelu/braid.git cd braid rake install_gem braid —help # see usage

Sighseeing

Mihai Anca has written a nice article showing some real world braid usage: Installing rspec with braid

He also helped track down the stupid bugs in the 0.3.0 release. I have beers already prepared.

Update

braid 0.3.2 was released to fix some major bugs

Braid is a simple tool to help track svn and git vendor branches in your git repository. It used to be called giston and was more hackish.

There is no direct way to update your giston mirrors to braid mirros, sorry :(. You know, life on the edge and stuff, heh.

Get it

git clone git://github.com/evilchelu/braid.git cd braid rake install_gem braid —help # see usage

giston 0.3.0 changes from giston 0.2.0

• using YAML::Store to save config
• using git merge-subtree with git and git svn remotes to track repositories

Rubyforge project on the way. I’ll update this page when it’s ready.

Have fun and please let me know of any bugs, complaints, ideas or patches you might have.

Usage

From the README:

create a git repo

git init moo cd moo

ideally you will also do these, but they are just good practices

git config —global merge.summary true git checkout -b localwork

adding mirrors

braid add git://blah braid add svn://muh gitk braid/track git merge braid/track

updating mirrors

braid update muh braid update git merge braid/track

removing mirrors

braid remove blah braid remove muh git merge braid/track

Update

This article is outdated: Please read about braid 0.3.0

Giston is a simple tool to help track svn vendor branches in your git repository. It’s loosely based on the idea of piston, but it’s more simplistic and it does less.

Get it

git clone git://git.che.lu/giston.git cd giston sudo ruby setup.rb giston —help # see usage

giston 0.2.0 changes

• now using the “main” gem for command line arguments parsing
• rewrote the command handling and cleaned up the code
• broke “giston update” when used without a mirror name

Please be aware that giston is, still, very immature and might(read: will) unexpectedly break at points. Make sure to commit any local changes before using giston.

I’ll create a rubyforge project for issue tracking and gem releasing next week. At the moment I can’t seem to login to rubyforge. Not to mention that I have to chase some beers around :).

Have fun and please let me know of any bugs, complaints, ideas or patches you might have.

Update

This article is outdated: Please read about braid 0.3.0

Apparently, quite a few people got interested in giston lately. I am actually in the middle of rewriting some of the ugliness and didn’t bother to let people know about it. My bad.

I’ll be committing some changes on Friday noon-ish gmt so please don’t patch against the current version as things most likely won’t apply anymore. If you already have a patch, please email it to me and I’ll try to apply your changes manually.

Sorry about the lack of transparency, I thought noone was using it, heh :).

Update

This article is outdated: Please read about braid 0.3.0

Motivation

A few months ago I started to use git for some of my projects instead of svk. I won’t go in what git is, plenty of people have done that better than I could.

Giston, like piston, aims to solve the problem of tracking svn vendor branches in your repository. Piston does a lot of work to be nice and safe and works great with svn. Giston is mostly a hack around some shell commands to do something similar for git, but in a more hackish and naive way. At least for now.

Giston’s code, as you will see, is pretty bad. But, on the bright side, it’s tested with rspec so, theoretically, you’d be able to at least understand what’s going on.

I’ve been trying to get back to it for a while and rewrite the ugly parts but never got around to do that. I’m putting it out there to have something visible in case anyone has the same needs and to motivate myself to fix it, by means of public shame :).

Giston

Giston is a simple tool to help track svn vendor branches in your git repository. It’s loosely based on the idea of piston, but it’s more simplistic and it does less.

Installing

git clone git://git.che.lu/giston.git cd giston sudo ruby setup.rb giston —help # see usage

Thanks

I would like to thank Chris Sturm for using giston and finding some bugs :).

Future

I plan to fix the bugs in the TODO file in the following weeks to have a more decent version till Christmas.

If you have ideas or wishes or, if you’d like to send a patch (YAY!), just leave a comment with a pastie or email me.

The ruby hit squad just released vlad 1.0, a merciless deployment tool, very similar to capistrano. I just new I had to install it when I saw the name, with me being Romanian and all. Thanks for not calling it Dracula!

As opposed to capistrano, vlad doesn’t come with it’s own runner. You have to manually require it in your Rakefile and it’ll add a bunch of deployment related tasks in the vlad namespace. It also has very few dependencies and uses your current ssh settings to connect.

Coming from capistrano, there are a few things you have to do manually for now to get a similar setup.

Vlad doesn’t have after or before tasks (yet), so you’ll have to add new tasks and wrap existing ones to get the same functionality:

desc “chelu’s setup” remote_task :mysetup, :roles => :app do Rake::Task[“vlad:setup”].invoke run “mkdir -p #{shared_path}/assets” end

It also doesn’t define a deploy task, to allow people to stick things together the way they need it. There’s a sample task in vlad’s bugtracker if you need an idea.

It also assumes you’re using mongrel_cluster as your server if you use the built in start, stop, restart tasks. I use monit so I just added my own and ignored the default ones.

So, it does the same things as capistrano and I just tried it without a real goal. But I was able to look in the source and find answers almost instantly. Simpler code is always nice.

Even if that doesn’t matter to you, It may be worth it to use just for the sheer coolness of being able to put in your resume:
“used vlad the impaler to deploy applications using sphincter”. :)

ps: There’s a bug with running sudo commands in 1.0, so you might want to wait for a new release if you don’t want to manually authenticate with sudo.

Prerequisites

svn and perl bindings

You might have to recompile svn if you don’t already have the perl bindings. I’m on OS X and I got these svn binaries with perl bindings

git

Git is being actively developed so various bug fixes and features are usually not in the stable release. You’ll need to get the source and compile it.

Compiling is just make && make install if you want it just for your user. You can grab a recent tgz from this git-snapshots page

Setup

You’re going to need a svn repository where you can commit. You can create an empty repository and or just use your current mephisto customization one, if you have it.

Tip: If you create an empty repository, make sure you create a trunk directory in it. It will help if you later decide to use some svn branches:

$ svn mkdir svn://server/blah/trunk

1. Create local git repositories

To clone a svn repository without grabbing all the revisions, you need to know a commit number that actually exists on branch you want to mirror. Many times this is not the number you get when you do svn update.

The easiest way to find it is:

$ svn info http://svn.techno-weenie.net/projects/mephisto/trunk | grep “Last Changed Rev:” | sed -e “s/Last Changed Rev: //”

On to the clones…

For mephisto, we’re going to start from a revision earlier than the latest so we can see how updates from mephisto will get into our repository:

$ git-svn clone -r 2939 http://svn.techno-weenie.net/projects/mephisto/trunk mephisto

For our svn repository we want all the history so we don’t pass a revision:

$ git-svn clone svn://server/blah/trunk blah

Now, let’s create a branch for mephisto into our blah git repository.

$ cd blah $ git remote add mephisto ../mephisto $ git fetch mephisto

We used git remote add to create a name for our mephisto repository for easier access.

Let’s check what branches we have so far. Your output should be the same.

$ git-branch -a

• master
  git-svn
  mephisto/master

At this point, it depends on wether you are using an empty repository or not. Use the next step (2a) if you’re starting on a new repository.

Jump to 2b. if you already have customizations in a svn repository.

2a. Empty svn repository

Merge the mephisto/master branch in the current master:

$ git merge mephisto/master

Then push the changes to our svn repository:

$ git-svn dcommit

Note: The git-svn documentation says to do a git-svn rebase before doing dcommits. Just this once, don’t do that.

2b. Existing mephisto customization

If you did the step above(2a), ignore this and skip to step 3.

Things are a bit more complicated if you’re trying to reuse a svn repository and want git to properly handle merges. This is just the way I’ve been able to do this, I’m sure there must be a better way. If you know of a better solution, please let me know about it.

First, I assume you have no local changes in your git repository. Use git-status to check and git-reset to undo any local changes. If you really really want to lose all local changes do git-reset —hard HEAD.

Now, let’s try a merge:

$ git merge mephisto/master

The result should be a bunch of conflicts for each change that happened on the mephisto tree since you last updated your customization. I’m thinking this is because there’s no common ancestor for the master and mephisto/master branches.

You could now go through all the changed files, remove the conflict markers, commit and it would work. But that kinda sucks. Let’s try something else..

Undo the local changes form the attempted merge with:

$ git-reset —hard HEAD

We’re going to manually apply the changes so we don’t get conflicts. We’ll use git-log to find the most recent update we did from mephisto. Scroll down until you find it and grab the commit hash.

$ git-log

Once you got that, we’ll use it to grab and apply the changes from mephisto:

$ git-diff 6c9f4058f13ad3e37cb8bd3d43315c914cdaf396 mephisto/master > changes.diff $ git-apply changes.diff

You might have some conflicts now so go ahead and fix them. Use git-status to see which files have conflicts. After you fix a conflict, use git-add blah to tell git about it.

After doing the conflicts, add the new files that we got form the merge to the index:

$ git-add .

Remove the changes.diff form the index:

$ git-rm —cached changes.diff

We also need to tell git about any deleted files (let me know if there’s a better way to do this):

$ git ls-files -z —deleted | git update-index -z —remove —stdin

Let’s see what we’ll commit:

$ git-status

If everything looks ok, go ahead and commit to git and push changes to our svn:

$ git-commit $ git-svn dcommit

3. Daily usage

Once you got the initial dcommit in your svn, things are the same regardless of wether you started with an existing mephisto customization or not. You can happily use git branch form master, do a bunch of commits, merge back, etc.

Getting changes from mephisto

Make sure you don’t have any uncommited changes first and do:

$ git fetch mephisto $ git merge mephisto/master

Pushing new changes in svn (and pulling changes from svn)

Make sure you don’t have any uncommited changes first and do:

$ git-svn rebase $ git-svn dcommit

Final notes

I spent a lot of time working this things out and I hope this helps some people to get up to speed faster. Please let me know if things can be done better or simples.

Resources

• git tutorial
• everyday git
• git man pages
• An introduction to git-svn for Subversion/SVK users and deserters

Note: If you don’t care about why I moved to git and just want the practical bits, you can go directly to Using git to track changes between svn repositories

¿Por que?

I often find myself having several patches to open source projects piling up in my local checkouts. It’s a pain to keep them together until they get applied upstream, if that ever happens.

Being able to commit without internet access is also required. I’m not usually on a fuckin plane, but I like working from some nice places where I can’t get online.

So, I definitely need something else than subversion that could talk nicely with subversion, to keep me happy.

Besides svn tracking and offline commits, I’d like to have piston like functionality out of the box, but I can live without that.

svk

A while ago I started to use svk to mirror some of the svn repositories I work with on a regular basis. I based my setup on octopod’s svk tutorial.

I didn’t quite enjoy the smerge and creating of a bunch of directories. Each time I wanted to add a new project, or even worse, a new project for a new mirrored repository I had to be very careful about a bunch of things. Not fun!

I also tried to import a local branch of a svn repository in a directory of a local branch of another one and tried to push that to my svn own repository. FAIL! “thingy is from another repository”. Bummer…

That last paragraph might not have made too much sense if you’re not familiar with octopod’s tutorial and svk so feel free to ignore it.

git

Im not a mad brancher so I mainly used svk as a fancy svn client. Also, my svk setup is a bit of a mess. It was easier for me to ignore the svk repository, recreate mirrors and reapply some patches in git.

However, if you want to import your whole svk into git, have a look at the introduction to git-svn for Subversion/svk users and deserters

In the next article I’ll show how to do a setup similar to the one in octopod’s tutorial. Go to Using git to track changes between svn repositories

Servarr!)!

While trying to get your skittlish fix on, you may have noticed that this site has been down for quite a bit. Turns out that the mini this site was hosted on got disconnected from the world (collocation lease expired) and it took a bit for the backups to show up.

Things are looking up though. Me main man, Jamie, was just about to start a hosting company and got a shiny new server. Fortunately, for both of us, he found himself snatched up by the funky Engine Yard people. So, thanks to his kindness, this site is now hosted on a slice of cheese somewhere in the UK.

I also moved the svn for skittlish to googlecode, “just in cases”. The old svn backups couldn’t be recovered in useful time so you have to recheckout from the new location and reapply your local changes. Sorry about that.

The new svn location is:

http://evilchelu.googlecode.com/svn/skittlish/trunk/

And you can checkout like this:

                
                svn co http://evilchelu.googlecode.com/svn/skittlish/trunk /path/to/mephisto/themes/site-1/skittlish
                
                

Hey, long time no zip!

Since the last release, mephisto’s interface got a bit lot more stable and I also got around to fixing some ugly IE bugs. So, head over to the skittlish project page and grab the latest version or just svn up if you’re living dangerously.

In other news, Amos Griffin did a smashing job by porting skittlish to wordpress (yay!) and Steve Ehrenberg made a trac theme based on skittlish. Great job, guys!

I totally forgot it was today so I’m starting a bit late. Just assume I was on a different timezone. :)

What happened to the design?

To learn more about why styles are disabled on this website visit the Annual CSS Naked Day website for more information.