Holovaty.com

Moving Django to GitHub: the postmortem

Adrian Holovaty — Sat, 28 Apr 2012 19:12:54 -0500

We finally moved Django to GitHub late yesterday. Here's a postmortem, to keep the community updated and for the benefit of any projects that take this leap in the future.

Background

We've used Subversion to manage our code since originally open-sourcing in July 2005. Over the last few years, we started to feel Subversion's limitations, namely:

The difficulty of branching. We used tools like svnmerge to keep track of which parts of branches had been updated from trunk, and some of us on the core team used Git/Mercurial on top of Subversion, but this was all unnecessarily complicated -- to the point of being stifling.
Lack of decentralization. When I would hack on Django on an airplane, for example, I couldn't make a bunch of commits locally, then push all of those to the master repository; I'd have to put everything in a single commit. With Subversion, it's all or nothing -- you push everything to a centralized server as you do it (or you use a branch, but that's painful, as noted above).
Slowness. After you use Git for a while, Subversion feels sluggish. This is due to a bunch of design and implementation differences.

(Of course, it's 2012 now, and these are all obvious, well-documented points. To the people who responded to our GitHub news by saying "finally!" -- I totally agree.)

Aside from that, we had set up a GitHub mirror (now called django-old) a few years ago, and lots of people were getting code and forking it there anyway.

Why Git/GitHub, as opposed to Mercurial/Bitbucket or some other system? Because it's very well-made, and it's where the people are. Clearly GitHub has won the majority of open-source developers' mindshare. John Lennon said: "If I'd lived in Roman times, I'd have lived in Rome. Where else?" GitHub is Rome.

The authors file

The first thing we considered was to simply start using our existing GitHub mirror -- turn off the Subversion stuff and start committing there directly. But the problem there was that we'd never set up an authors file.

Basically, an authors file maps Subversion committer names to standard names and email addresses, so that GitHub knows that a commit by "adrian" in Subversion maps to the adrianholovaty GitHub account. With that mapping established, you get niceties like GitHub commits linking to appropriate GitHub user pages and displaying proper user avatar images. More importantly, it gives all of our contributors proper credit within the GitHub ecosystem for the full history of their work on Django -- which has value these days, considering companies are looking at GitHub involvement for job applicants, etc.

So the first step was creating that authors file, which Brian Rosner organized, with the help of several other people. We ended up accounting for every one of the 58 people who have ever committed to Django, except for somebody named "cell" who was given temporary commit access during a sprint six years ago.

One crucial detail is that we couldn't simply change the commit data retroactively in the existing GitHub repository. That's because Git uses the committer data in creating hashes. Changing the commit data would change the hashes, which would break all existing forks of that repository. (We ended up breaking existing forks anyway, of course, but it was cleaner to do it from scratch.)

Nuts and bolts of the process

Once we finalized the authors file, doing the migration was actually kind of easy, thanks to git-svn. I took many missteps along the way, got a lot of help from people in #django-dev on IRC and ended up doing three dry runs. Here are the final steps I ended up taking:

1. Copied the Subversion repository from code.djangoproject.com to my laptop, to make the migration faster.

# On the server:
svnadmin dump /home/svn/django | gzip > svndump.gz

# On my laptop:
scp djangoproject.com:svndump.gz .
gunzip svndump.gz
svnadmin create /Users/adrian/code/django-svn
svnadmin load /Users/adrian/code/django-svn < svndump

On my first run of git-svn, I ran it from my laptop and pointed it at code.djangoproject.com, and it took 3.5 hours! After I copied the repo to my laptop and tried it again, it took a little over an hour. But the caveat here is that I also changed the git-svn command between those two runs, so I'm not sure how much of the speed improvement was because of the local SVN repo.

2. Ran git-svn (with the correct arguments!).

git svn --authors-file=authors.txt --trunk=trunk clone file:///Users/adrian/code/django-svn/django/ django-dry-run

This took a little over an hour, and it created a Git repository called django-dry-run. Note that authors.txt is the authors file, as explained above.

The trickiest thing about this was determining the correct arguments to use -- specifically, whether to use --branches explicitly or --stdlayout. As you can see, I ended up using neither.

Originally, the plan was to migrate all of the branches from our Subversion history -- classics such as magic-removal, new-admin, newforms-admin, unicode, queryset-refactor and multidb -- so that the branches' commit histories (which have all since been merged to trunk) could be preserved in our new Git history. Many of those branches were very involved, with a lot of commits, and there's a lot of value in being able to isolate specific commits in the branch, rather than one large merge commit. (Imagine you're investigating the original reason we added a line of code, for example.)

But as we discussed this over IRC, we decided it wasn't worth the effort, we could always do it later and git-svn wouldn't actually do it the way we wanted. Ideally, I'd like these branches' histories to be migrated such that they're treated like merged branches in Git -- a merge commit that knows the individual commits on the branch. If you know how to pull this off, and it can be done without altering the Git hashes, please let me know.

3. Changed git-svn-id to point at code.djangoproject.com instead of my laptop.

git filter-branch --msg-filter "sed \"s|^git-svn-id: file:///Users/adrian/code/django-svn/django/trunk|git-svn-id: http://code.djangoproject.com/svn/django/trunk|g\"" -- master

git-svn adds a "git-svn-id" section to each commit message in the resulting Git repository. It includes a URL pointing to the commit in the original Subversion repository, which is very useful.

But, because I did the import from a local repository, the git-svn-id's were all pointing at my laptop. So I ran git filter-branch to clean it up.

4. Renamed old GitHub django repository to django-old.

(Done via the GitHub Web site.) This was the scary part, because it meant there was no turning back. :-)

Originally we'd talked about deleting the repository outright, but that would have deleted all pull requests and likely would have broken some other things. So I just renamed it to django-old. Not sure how long we'll keep this around.

4. Imported the new repository into GitHub.

git remote add origin git@github.com:django/django.git
git push -u origin master

I spotted an error in the repository after the first time I did it, so I had to delete it -- which I thought made for a rare and amusing screenshot:

Then I cleaned up the repository and did it again. I mistakenly created it as a private repository, so I marked it as public, which led GitHub to believe I had just open-sourced Django. :-)

And that's it!

Stats

For posterity:

Final number of commits in our Subversion repository: 17,942.
Size of Subversion repository: 339 MB gzipped. (That's for the dump file as generated by svnadmin dump.)
Number of commits created in Git by git-svn: 11,883. (This is less than 17,942 because we only migrated trunk. Any commit to our repository that didn't touch Django trunk -- such as commits to the django_website project or commits to branches -- did not get migrated.)
Number of forks of the old (mirror) GitHub repository, as of this writing: 783.

Going forward

The old Subversion repository will remain indefinitely, for the benefit of scripts out there that do automatic updates, and general stability of the Django world. There won't be any more commits there, obviously.
If we ever need to dive into the history of one of the big merged branches -- such as magic-removal -- we can do so in the Subversion history. Or we can consider copying the branch history into Git somehow (see above).
I'd like us to provide some documentation on how to convert your previous Django fork (from the django-old repository) to track the new repository. Any volunteers?
We still have a bunch of work to do fixing places in our documentation and code.djangoproject.com that refer to Subversion. Bear with us.

Filing bugs / pull requests / the ticket system

GitHub's ticket system is a bit too simple for our needs, given the Django triage process, so we're sticking with our Trac installation, at least for the time being.

But, of course, we want to take advantage of GitHub pull requests at the same time. So we'll need to figure out the right balance between pull requests and Trac tickets, such that we maintain our sanity, we don't make people jump through hoops, and we optimize for contributor and committer productivity.

Personally, I want to avoid a situation (and culture) where we force contributors to use Trac if they post pull requests, especially ones that contain trivial changes. But at the same time, it'll likely become a maintenance nightmare if we have lots of tickets in two places, with no coordination. So, this is an open issue we'll be working to figure out. Jacob has been working on a technological solution.

Thanks to all the people who helped with this transition, and I look forward to the much happier development and collaboration experiences we get with GitHub. The commits and pull requests I've already handled have been a pleasure.

Back in the Django saddle

Adrian Holovaty — Mon, 28 Nov 2011 14:19:11 -0500

I've been a bad BDFL.

As a co-creator and one of the two Benevolent Dictators for Life of Django, my responsibility is to guide the open-source project forward. I did a great job of this for several years: from our open-sourcing in July 2005 (and for about two years prior to that), I was rather obsessed with it, contributing thousands of commits, answering thousands of support questions, participating in hundreds of design discussions, co-writing two editions of a book, writing week in review blog posts and evangelizing the framework at events. Making Django's code and community better is how I spent a significant chunk of my free time.

It grew wonderfully and brought me some of the proudest moments of my life, such as Google's launch of App Engine (with Django support built in!), a full-blown conference devoted to the framework and the consistently mind-blowing experience of being handed a book about Django written in a language other than English: Japanese! Chinese! Italian! Not to mention hearing about all of the incredible applications people around the world are building with the framework.

But then, in July 2007, EveryBlock happened. Running a startup took over my life, and even though EveryBlock is itself a rather complicated Django application, meaning I use the framework daily in non-trivial ways, my contributions to the framework itself basically stopped. And though I'm just one of more than a dozen core contributors, it's fair to say the project has suffered because of my absence. The development process got slower and too bureaucratic, contributors got frustrated by our lack of decision-making, and changes were made to the framework that I would've prevented had I been more involved.

I've felt horribly guilty about this for the past several years, and every few months I'd say to myself, "OK, I'm going to get back into Django development. For real this time." I'd fix a few bugs, make some documentation tweaks or whatever, but then I'd inevitably get distracted by my day job again.

But today, everything changes for the better. I'm excited to share the great news that Adrian's Lost Weekend is over. I've gotten the OK to spend a full workday a week on Django!

This is made possible by the growth at EveryBlock. I recently hired my own boss, after running the show for four years, and we've expanded our development team significantly. We do have some specific needs at EveryBlock that I'll be addressing in Django -- making speed improvements and fixing various pain points -- but I've made it clear that not everything I do on Weekly Django Day will be directly applicable to EveryBlock. Django, and my involvement therein, has yielded great dividends for EveryBlock, and it's time to formally give back.

The one-day-a-week regimen is exactly the kind of structure that will help me get back in the groove. I plan to work on these priorities, to start:

Release version 1.4.
Move to Git/GitHub. I want us to move from Subversion to Git, hosted on GitHub. Obviously, this is a large project with many repercussions, not the least of which is how our bug tracker and development process will change. We'll likely end up with some sort of hybrid system with our existing Trac installation, given how much we have invested in it. I predict some pain during this process, but it'll be worth doing for a whole bunch of reasons.
Reinstate "This Week in Django" blog posts. These were a great, concise way for Django users to stay updated with developments in the framework and community. I'm going to try to get these going again on the official Django blog.
Implement easy Pjax support. I have some design ideas on how we can easily add intelligent Pjax support to any Django app, with minimal code changes. I'm really excited about this one in particular.
Speed up template rendering. Armin has repeatedly talked about how much slower Django's template rendering is than Jinja's, and I want to cut out our inefficiencies to bridge the gap.
Fix transaction management. Not sure yet whether this only affects PostgreSQL, but I'm tired of the "idle in transaction" problem.
Fix the way .exclude() works. This one, described here, is getting in the weeds a bit, but it's an important thing to fix.

Look for more discussion on these issues on the django-developers mailing list. (My blog's comment section below is not the place to talk this stuff over. :-) )

I'm officially starting the one-day-a-week thing next week. It's good to be back. Let's get to work.

Introducing the YouTube Insult Generator

Adrian Holovaty — Sun, 16 Oct 2011 22:50:24 -0500

(It has been way too long since I launched a side project. Time to get back into the game. This blog post is adapted from a five-minute talk I gave today at Ignite Chicago.)

Everybody knows YouTube comments are atrocious. This is referenced all over the place, from a bunch of blogs to xkcd (twice!). One guy even took the time to make YouTube Comment Snob, a brilliant browser extension that hides poor quality comments.

If only we could extract some value from all that crap.

The YouTube community is so huge at this point that "meta" comments are very common. One particular "meta" comment stems from the fact that YouTube changed its rating system in early 2010. It used to use a five-star rating system but moved to a simpler "thumbs up" or "thumbs down" model, citing the fact that most people gave either one- or five-star ratings anyway.

In this new rating system, each video displays how many people have liked -- and disliked -- it. In a classic example of interfaces influencing behavior, this has encouraged users to make insulting remarks about the dislikers. I'm sure you've seen the type: it's comments like "439 people own a zune" on the 2001 Steve Jobs iPod presentation. (439 people had disliked the video at the time the comment was posted.)

After groaning about these types of comments for a while, I realized they're a sort of semistructured information -- it's always a number, followed by some sort of insult. And, besides, some of them are actually kind of witty. Hence my new project: The YouTube Insult Generator.

This is a basically a "search engine for insults." Type in a search term, and it'll give you insults you can use against a person who doesn't like that term.

For example, enter "the godfather," and it'll give you "You sleep with the fishes," "You sleeps with horsehead in bed" and "You will get an offer you can't refuse." Enter "alfred hitchcock" and it'll say "You had your eyes plucked out by crows" and "You have Vertigo." Enter "mario brothers" and it'll say "You aren't Super enough for Mario," "You can't beat world 1-1" and "You are bowser." You get the idea.

It finds stuff only about 50% of the time, but it works surprisingly well when it does work. Try general terms ("car") and pop culture ("michael jordan", "i love lucy"). Each insult includes a link to its source YouTube video.

How does this work? It uses the YouTube API to search for the top 50 most relevant videos for your search term. For each of those videos, it grabs the latest 50 comments. Then it looks through all that for comments starting with a number followed by a word such as "people," "youtubers" or "nincompoops." (View source for the full list, a regular expression that would make Alex Gaynor proud.) Finally, it just replaces the number and the word "people" with "You."

It doesn't do anything fancy like caching or giving users a way to mark searches as particularly funny, but it's not bad for a quick hack. Enjoy, and please use the insults wisely.

UPDATE: Wired wrote about it. Also, a few people have been confused about what the point of this is. It's not intended to be literal -- I mean, I don't expect people to use this, at face value, to create insults. The larger point is that it's a demonstration of finding structured data in unexpected places. I think we should experiment more with this sort of "poor man's data mining" on things like YouTube comments. Thanks for checking it out!

Crash tags

Adrian Holovaty — Thu, 14 Jul 2011 23:38:18 -0500

Chris Messina, the guy who started Twitter hashtags, proposed the use of hashtags on Google Plus earlier this evening.

There's some major cognitive dissonance going on here, seeing Twitter hashtags on Google Plus.

It really hits you over the head with how limiting (and, frankly, silly) Twitter is. I think in a few years people will look back at Twitter and feel like they were, well, duped in a way -- millions of grown men and women, adults, typing in obscure abbreviations and making up messy syntaxes just to fit a semi-arbitrary character limit; large, important companies falling over themselves to play the game and Build Their Brands around these tiny messages; opportunists positioning themselves as gurus of this Sophisticated New Communication Platform™.

It feels like the Twitter founders are sitting in a room somewhere, laughing at how they've gotten millions of otherwise-sane people to jump through these absurd hoops -- and that, deep down, even the most guru-y of the Social Media Gurus ask themselves from time to time, "Seriously, I'm doing this?"

Anyway, it feels like G+ should have something much more sophisticated and user-friendly.

Job opportunity: Web/mobile developer at EveryBlock

Adrian Holovaty — Wed, 06 Jul 2011 13:56:42 -0500

We're looking to hire another programmer to join our small and effective development team at EveryBlock. This is your chance to work on a wide variety of interesting problems, helping improve a great product with a passionate and growing user base. You should expect to have a hand in all aspects of the site, and your contributions will have an immediate, direct impact on the awesomeness of our service.

When it comes to developers, we like to hire Jacks (or Jills) of all trades. The ideal candidate will be equally comfortable developing back-end Python code, working with databases (we use PostgreSQL and PostGIS), writing efficient front-end JavaScript (we use jQuery), writing screen-scrapers and other various data-import scripts (we use Python), helping optimize our infrastructure, and working on EveryBlock mobile applications (iPhone, Android). There will be a focus on mobile applications, at least at the beginning, but your responsibilities will be wider and more varied than that over time.

Here is the requisite bulleted list of requirements:

Significant experience using Python.
Experience building mobile applications for iPhone and/or Android.
Experience using Django to build Web applications.
Experience developing non-trivial JavaScript applications.
Experience deploying code on Linux.
Experience (and preference for) working on a small team.
Clear communication skills, both verbal and written.
Impeccable work ethic and ability to manage your own workload effectively.
Ability to iterate quickly.
Ability to write high-quality code, with a passion to the point where poorly written code makes you slightly nauseous.
A passion for computer programming/hacking to the point where you'd be doing it even if you weren't getting paid for it.

And here are some nice-to-haves:

Experience building geo apps and using PostGIS.
Experience writing screen-scrapers and working with data (cleaning it up, importing it).
Passion about improving neighborhoods through information and enabling community conversation.
Sense of humor.

This is your chance to work directly with two Django committers, including one of the creators of the framework (hey, that's me!). We have three developers at the moment; you'd be the fourth. It's worth mentioning that the entire team right now is seven people, and you have a chance to make a huge impact, beyond strictly development -- suggesting marketing ideas, helping prioritize various strategies and tactics, etc.

We have a low-stress environment and a culture of getting things done with as little corporate BS as possible. Though we're technically no longer an independent startup -- we were acquired by msnbc.com in 2009 -- we're culturally very much still a startup, given that our product is still in early stages and we're still figuring things out. It's a very nice combination of startup culture with the financial security of working for a big company. (We have great benefits -- salary, health care, 401(k), bonuses, etc.) And msnbc.com is taking the long-view on us, investing in us over time and giving us years to develop a large audience and become profitable. It's a great company to work for.

Now is a fantastic time to be joining our team. We just hired a new President, we recently launched a major redesign that's been almost universally loved, and, most importantly, we're getting momentum and critical mass in more and more neighborhoods.

You'll need to live in the Chicago area for this position; telecommuters need not apply. Our office is in a comfortable loft space on the north side of Chicago, near a bunch of other startups. Several of us walk or bike to work; we're also equidistant from the Montrose and Irving Park el stations on the Brown Line, and several bus lines.

Apply through the official msnbc.com Jobvite site here. Thanks for checking this out.