Holovaty.com

EveryBlock hiring a Python screen-scraping expert

Attention Python screen-scraping experts! We're looking to hire another full-time developer at EveryBlock. Our site, which just launched a few weeks ago, compiles a wealth of granular geographic data and publishes it on a block-by-block basis. We offer a distinct Web page (plus an RSS feed and e-mail alerts) for every city block in Chicago, New York and San Francisco. We're expanding to more cities and more data sources. And we have a ton of fun features and projects up our sleeves.

This position involves contributions to all of our site's technology and data, with a concentration on screen-scraping public data from government Web sites. Some specifics we're looking for are:

Mastery of screen-scraping
Experience programming in Python
Experience with geographic data

Experience with Django is a nice-to-have.

For more on EveryBlock, check out our launch announcement and this recent interview.

This is an opportunity to work on an exciting and important project with a talented and experienced Web development team. We're currently only four people, so you'll have a lot of freedom and opportunities to make a difference.

This is a full-time, salaried position, on-location in our modest downtown Chicago office. We're a startup, funded by a grant, trying to make the world a better place. Please contact me if you're interested or have any questions. Tell me about the gnarliest site you've ever scraped.

A couple of EveryBlock interviews

Back in 2006, I had a very enjoyable interview with Robert Niles at Online Journalism Review. Now, Robert and I have gotten back together for another e-mail conversation about my latest project, EveryBlock: check it out.

And there's more! Earlier today, Rex Sorgatz published an interview with me about EveryBlock, with more of a technology focus.

Thanks to both Robert and Rex for the great questions.

In memory of chicagocrime.org

It's with mixed feelings that I announce the end of one of my projects, chicagocrime.org. This site has been serving Chicago residents since May 2005. I hope you'll indulge me in a brief retrospective.

Chicagocrime.org was one of the original map mashups, combining crime data from the Chicago Police Department with Google Maps. It offered a page and RSS feed for every city block in Chicago and a multitude of ways to browse crime data — by type, by location type (e.g., sidewalk or apartment), by ZIP code, by street/address, by date, and even by an arbitrary route. The New York Times Magazine featured it in its 2005 "Year in Ideas" issue, and it won the 2005 Batten Award for Innovations in Journalism.

It's been a fun ride. When I launched the site, Google Maps hadn't yet released the mapping API that's so common — even passé? — today. I can't help but feel like an old-timer: "Back in my day, we had to reverse-engineer Google's obfuscated JavaScript just to get maps embedded on our own sites!" Now it seems like every other Web site finds an excuse to use those familiar, bubbly, yellow-white-blue-pastel map tiles.

Chicagocrime.org wasn't the first Google Maps mashup. That honor belongs to Paul Rademacher's HousingMaps, which, at that time, was modestly titled "Craigslist + Google Maps." The straightforwardness of that original title illustrates the excitement of it all: just the mere fact that somebody had mixed Craigslist data with Google's maps was new and remarkable. Kudos to Paul for keeping the site up and running for all these years. Not only was it a groundbreaking technical achievement; it remains genuinely useful.

A lot of good has come out of chicagocrime.org. At the local level, countless Chicago residents have contacted me to express their thanks for the public service. Community groups have brought print-outs of the site to their police-beat meetings, and passionate citizens have taken the site's reports to their aldermen to point out troublesome intersections where the city might consider installing brighter street lights.

It's done some good on a larger scale, too. The site helped influence Google to open up its mapping API for all to use. It inspired at least a dozen "spin-off" sites in other cities, from Berkeley to New Haven to Houston — most of whose designs were very similar to Wilson's beautiful chicagocrime.org design. And the site's slashdotting forced me to write parts of Django's cache system. (Django itself was released open-source two months later; chicagocrime.org was the first public Django-powered site not run by the Lawrence Journal-World.)

A few weeks ago, I received an e-mail from the folks at Amazon EC2, where the crime site is hosted, saying the server instance that houses the site will be terminated on February 15 — and that it will no longer be accessible after January 31. This is happening because I was an early user of EC2 and their network has gone through some changes that require all customers of a certain tenure to rebuild their servers. Instead of going through the hassle of upgrading my server instance, I'll let the Amazon staff shut it down on Thursday. All pages will redirect to the appropriate pages on my newest project, EveryBlock.

In many ways, EveryBlock is the next generation of chicagocrime.org. I've often described it to people as "chicagocrime.org on steroids — more than just crime, and more than just Chicago." It's brought to you by the same people (Wilson and me from chicagocrime.org, plus Paul and Dan, who've worked on similar projects), and it has the same philosophies. As we developed EveryBlock, we kept chicagocrime.org firmly in our minds — this new thing we were making had to be a superset, an expansion, a significant step forward. So there's almost nothing you could do on the old chicagocrime.org that you can't do on EveryBlock. And, unlike chicagocrime.org, which was always a side project, EveryBlock has a team of four people improving it full-time, meaning we have the resources to add features, such as e-mail alerts (just added yesterday), that chicagocrime.org never had. We hope EveryBlock is a worthy successor.

This story has a fitting epilogue. In just a few weeks after chicagocrime.org goes offline, the site will be featured in an exhibition at New York's Museum of Modern Art, called Design and the Elastic Mind. Chicagocrime.org will have ended its life and become a museum piece.

Interview at akitaonrails.com

Yesterday, I was interviewed over IM by Fabio Akita, a Web developer from Brazil. We had a good conversation about Python, Django and other various things, and Fabio has posted the transcript. There's also a Portuguese translation.

Django Book has shipped -- and, thoughts on the next book

It's here! At long last, the print copy of the Django Book has shipped. I received my author copies late last week and am still poking at them to make sure that, yes, a tangible book with my name on the cover has actually been printed, on real paper, by a real publisher.

Early drafts of the book have been available free at djangobook.com for more than a year, and co-author Jacob and I are grateful to all of the readers who submitted corrections and suggestions. Jacob is going to update the site soon with the final text of the book (which will be available free under an open-source license), and we plan to revise the online text over time with corrections and additions. There's something nice about having a paper copy, of course.

The book is available on Amazon, and I'm told the big brick-and-mortar bookstores should begin stocking it soon. Looks like it's gotten some buzz already, as it was the number one best-selling "Software Development" book and number four best-selling "Internet" book. Not bad at all! What I'm most proud of is not the fact that the book itself is doing well, but the larger fact that demand for information about the framework is high.

Now that the Django Book is finally in the can, I'm mulling the idea of writing another book -- this time, a book about online journalism. In the past two years, I've been to (way too) many journalism-related events and conferences trying to spread the good word about "journalism via computer programming," and I've detected a strong, I daresay furious, demand, from journalists at all levels in the org chart, for information about this new form of journalism. Higher-ups want to know why they should employ programmers; middle managers want to know how to find them and how to treat them; and working journalists want to learn these skills and strategies. The problem is that I can't point them anywhere for in-depth information. This book would attempt to solve that.

I want to take a shot at writing a manual, a manifesto, a practical guidebook to this emerging discipline of database-driven Web journalism. It would be a combination of high-level strategy and low-level technique, probably split cleanly into two parts (one for the suits, one for the non-suits).

That's about all the thought I've given to this idea. What do you think? If you're a journalist (or even not), is this something you'd be interested in?