Holovaty.com

EveryBlock acquisition and me

I'm excited to announce some huge news. EveryBlock, the project I've led for the last two years, has been acquired by MSNBC.com.

The main details are at the EveryBlock blog, but I wanted to mention a few other things here on my personal site, to try to anticipate questions.

First, this has no impact whatsoever on my involvement with Django. MSNBC.com is half-owned by Microsoft (although it's a separate company), so I expect plenty of Microsoft jokes -- but that's all they are: jokes. I'm not going to start developing things with Microsoft technologies; EveryBlock is not going to be converted from Django/Python to .NET or any other framework, MS or otherwise; I'm going to keep using a Mac as my work and personal computer, although I'm tempted from time to time to return to Linux. All of this is well-established with the MSNBC.com guys; they're cool.

If anything, this acquisition will give me some more time to work on Django, because I'm no longer spending a lot of time negotiating, talking with lawyers and worrying about how EveryBlock will continue operation. :-)

Second, yes, I'm staying in Chicago. The team's main headquarters will continue to be here, except for our coastal EveryBlockers, Wilson Miner (San Francisco) and Paul Smith (Baltimore). I'm hoping I can still participate in the local Web startup scene.

Third, this has no effect on the EveryBlock open-source code (ebcode). The code as released on June 30 will continue to be available.

I'm happy to answer any other questions in the comments. Looking forward to a great next chapter of EveryBlock!

The definitive, two-part answer to "is data journalism?"

It's a hot topic among journalists right now: Is data journalism? Is it journalism to publish a raw database? Here, at last, is the definitive, two-part answer:

1. Who cares?

2. I hope my competitors waste their time arguing about this as long as possible.

Django tip: Caching and two-phased template rendering

We've launched user accounts at EveryBlock, and we faced the interesting problem of needing to cache entire pages except for the "You're logged in as [username]" bit at the top of the page. For example, the Chicago homepage takes a nontrivial amount of time to generate and doesn't change often -- which means we want to cache it -- but at the same time, we need to display the dynamic bit in the upper right:

One solution would be to pull in the username info dynamically via Ajax. This way, you could cache the entire page and rely on the client to pull in the username bits. The downsides are that it relies on JavaScript and it requires two hits to the application for each page view.

Another solution would be to use Django's low-level cache API to cache the results of the queries directly in our view function. The downsides are that it's kind of messy to manage all of that caching, plus each page view still incurs the overhead of template rendering (which isn't horrible, but it's unnecessary overhead).

The solution we ended up using is two-phased template rendering. Credit for this concept goes to my friend Honza Kral, who suggested the idea to me during PyCon earlier this year.

The way it works is to split the page rendering into two steps:

At cache reset time, render everything except the "You're logged in as" bit, which should remain unrendered Django template code. Cache the result as a Django template. (This is the clever part!)
At page view time, render that cached template by passing it the current user. This is super fast because, at this point, the template only has two or three template tags. (The rest of the page is already rendered.)

It's a clever solution because you end up defining what doesn't get cached instead of what does get cached. It's a sideways way of looking at the problem -- sort of like how Django's template inheritance system defines which parts of the page change instead of defining server-side includes of the common bits.

In order to make this work, we had to write two parts of infrastructure: a template tag and a middleware class that does the cache-checking and rendering. The template tag looks like this:

# Copyright 2009, EveryBlock
# This code is released under the GPL.

from django import template
register = template.Library()

def raw(parser, token):
    # Whatever is between {% raw %} and {% endraw %} will be preserved as
    # raw, unrendered template code.
    text = []
    parse_until = 'endraw'
    tag_mapping = {
        template.TOKEN_TEXT: ('', ''),
        template.TOKEN_VAR: ('{{', '}}'),
        template.TOKEN_BLOCK: ('{%', '%}'),
        template.TOKEN_COMMENT: ('{#', '#}'),
    }
    # By the time this template tag is called, the template system has already
    # lexed the template into tokens. Here, we loop over the tokens until
    # {% endraw %} and parse them to TextNodes. We have to add the start and
    # end bits (e.g. "{{" for variables) because those have already been
    # stripped off in a previous part of the template-parsing process.
    while parser.tokens:
        token = parser.next_token()
        if token.token_type == template.TOKEN_BLOCK and token.contents == parse_until:
            return template.TextNode(u''.join(text))
        start, end = tag_mapping[token.token_type]
        text.append(u'%s%s%s' % (start, token.contents, end))
    parser.unclosed_block_tag(parse_until)
raw = register.tag(raw)

This template tag merely treats everything between {% raw %} and {% endraw %} as unrendered template code.

Then, in our base EveryBlock template, we wrap the appropriate bit of code in the {% raw %} tag, like this:

{% raw %}
	{% if USER %}
		<p>Logged in as {{ USER.email }}</p>
	{% else %}
		<p>Sign in / register.</p>
	{% endif %}
{% endraw %}

The final part is to write some middleware that renders every text/html response through the template system:

# Copyright 2009, EveryBlock
# This code is released under the GPL.

from django.core.cache import cache
from django.template import Template
from django.template.context import RequestContext
import urllib

class CachedTemplateMiddleware(object):
    def process_view(self, request, view_func, view_args, view_kwargs):
        response = None
        if request.method == 'GET' and 'magicflag' not in request.GET:
            cache_key = urllib.quote(request.path)
            response = cache.get(cache_key, None)

        if response is None:
            response = view_func(request, *view_args, **view_kwargs)

        if 'magicflag' not in request.GET and response['content-type'].startswith('text/html'):
            t = Template(response.content)
            response.content = t.render(RequestContext(request))

        return response

One thing to note here is that there's a backdoor for an external process (say, our script that resets the cache) to retrieve the halfway-rendered template code for any page -- magicflag in the query string. (We actually use something different on EveryBlock; I've changed this example.) So that means the only thing the cache-resetting script has to do is make a request to the appropriate page, with that query string, and save the result in the cache. Pretty slick.

There's also a potential gotcha/limitation here: anything within {% raw %} and {% endraw %} will only have access to a template context with the default RequestContext stuff -- which, in our case, will be user-specific stuff.

Thanks again to Honza for telling me about this concept. It's a great idea, and it's serving us well.

Looking toward EveryBlock’s future

It's been a year and a half now since I've started working on EveryBlock, and I'm still having the time of my life. Starting from scratch in July 2007, our team of six has built a one-of-a-kind local news site that now serves 11 cities and makes more than a hundred distinct types of local information useful to people. By all measures, from passionate user feedback to press coverage to traffic numbers to influence on other projects, the site is a success, and we're incredibly proud of our work.

Thanks to our out-of-the-ordinary funding — a generous grant from Knight Foundation — our team has been given free rein to invent a new form of news, and, more importantly, iterate on the concept. (Check out our recent one-year anniversary blog post to see how much we've changed.) This sort of experimentation is sorely needed by the news industry, and we're very lucky to have this opportunity — particularly considering the economy and the trend for news organizations to cut staff rather than invest in innovation.

But now we've reached an interesting point in our project's growth: our grant ends on June 30, and, under the terms of our grant, we're open-sourcing the EveryBlock publishing system so that anybody will be able to take the code to create similar sites. That's a Good Thing, in that EveryBlock's philosophies and tools will have the opportunity to spread around the world much faster than we could have done on our own, but it puts the six of us EveryBlockers in an odd spot. How do we sustain our project if our code is free to the world?

We have a number of ideas for sustaining our project beyond a dependency on grants, like building a local advertising engine and/or selling hosted versions of the open-source software, but we're sure there are other ways for EveryBlock to be a successful business. That brings me to the reason I'm posting this — we're looking for ideas and partners who would be interested in helping us figure this out. If you have any ideas or suggestions, get in touch with me. I'm confident we'll make something happen; it's just a matter of how.

Announcing the Django Book, second edition

I'm excited to announce that I'm working on a second edition of the Django Book.

The first edition, which I cowrote with Jacob Kaplan-Moss, was published in print by Apress more than a year ago, and, sadly, it's become out of date. It covers Django version 0.96, and many of the examples don't work with the current version, 1.0.

Fortunately, now that Django has reached 1.0 and is committed to backwards compatibility, this book will have a much longer shelf life. :-)

At this point, I've rewritten/edited the first three chapters and published the drafts for free online, as we did the first time around. I plan to add chapters incrementally over the coming weeks. Instead of posting blog entries here, I'll make those "new chapter posted" announcements on my Twitter account.

What's new in this edition? I'm not only updating examples to work with 1.0; I'm also doing a full rethinking and rewriting of many of the sections and will probably add a couple of chapters (testing Django apps, anyone?).

As before, we've got a slick granular comment system so that you can leave feedback on particular paragraphs. These were invaluable for the first edition, so I hope folks submit lots of great comments (without, I hope, finding too many errors!).

Apress has signed on again to publish the second edition in print. Also as before, I'm happy to report that the book will remain free online. The first edition is still available in its entirety, too, but I've added some much-needed caveats that it's out of date.

Jacob is sitting this one out due to time constraints, so I'll be doing the writing/rewriting, and the fine editors at Apress will be editing. It's easier to rewrite than write from scratch, so it shouldn't take too long.

Dig in, enjoy, and leave plenty of comments!