Holovaty.com

Request: Headless HTML rendering engine?

Warning: Seriously geeky request ahead!

I'm looking for a way to render arbitrary Web pages -- including CSS and JavaScript -- and access the resulting DOM tree programatically, i.e., in an automated/headless fashion. I want to be able to ask the following questions of the resulting DOM tree:

For a given element, what font family, size, and color is the text?
How tall and wide (in pixels) is a given <div>, <table>, etc.?
What are the x/y coordinates of a given element (from the upper-left corner of the page, or lower-left, or wherever)?
For a given element, what is its text content?

The rendering must be state-of-the-art, handling advanced CSS that Firefox, Safari and IE handle. It should work on Linux. Bonus points if there's a Python API for this magical DOM tree.

This is all stuff that standard in-page JavaScript could accomplish, but the catch with me is that I need to be able to do it in a completely automated way, on arbitrary pages, on a headless server.

I know Gecko and Webkit provide this, but I'm not sure where to start with them. The docs and articles I've read seem to be focused more on embedding the full browser window in a GUI application than embedding the rendering engine itself and manipulating the resulting pages.

Help! If you have any clues, I'd be grateful if you left a comment or got in touch with me.

In memory of chicagocrime.org

It's with mixed feelings that I announce the end of one of my projects, chicagocrime.org. This site has been serving Chicago residents since May 2005. I hope you'll indulge me in a brief retrospective.

Chicagocrime.org was one of the original map mashups, combining crime data from the Chicago Police Department with Google Maps. It offered a page and RSS feed for every city block in Chicago and a multitude of ways to browse crime data — by type, by location type (e.g., sidewalk or apartment), by ZIP code, by street/address, by date, and even by an arbitrary route. The New York Times Magazine featured it in its 2005 "Year in Ideas" issue, and it won the 2005 Batten Award for Innovations in Journalism.

It's been a fun ride. When I launched the site, Google Maps hadn't yet released the mapping API that's so common — even passé? — today. I can't help but feel like an old-timer: "Back in my day, we had to reverse-engineer Google's obfuscated JavaScript just to get maps embedded on our own sites!" Now it seems like every other Web site finds an excuse to use those familiar, bubbly, yellow-white-blue-pastel map tiles.

Chicagocrime.org wasn't the first Google Maps mashup. That honor belongs to Paul Rademacher's HousingMaps, which, at that time, was modestly titled "Craigslist + Google Maps." The straightforwardness of that original title illustrates the excitement of it all: just the mere fact that somebody had mixed Craigslist data with Google's maps was new and remarkable. Kudos to Paul for keeping the site up and running for all these years. Not only was it a groundbreaking technical achievement; it remains genuinely useful.

A lot of good has come out of chicagocrime.org. At the local level, countless Chicago residents have contacted me to express their thanks for the public service. Community groups have brought print-outs of the site to their police-beat meetings, and passionate citizens have taken the site's reports to their aldermen to point out troublesome intersections where the city might consider installing brighter street lights.

It's done some good on a larger scale, too. The site helped influence Google to open up its mapping API for all to use. It inspired at least a dozen "spin-off" sites in other cities, from Berkeley to New Haven to Houston — most of whose designs were very similar to Wilson's beautiful chicagocrime.org design. And the site's slashdotting forced me to write parts of Django's cache system. (Django itself was released open-source two months later; chicagocrime.org was the first public Django-powered site not run by the Lawrence Journal-World.)

A few weeks ago, I received an e-mail from the folks at Amazon EC2, where the crime site is hosted, saying the server instance that houses the site will be terminated on February 15 — and that it will no longer be accessible after January 31. This is happening because I was an early user of EC2 and their network has gone through some changes that require all customers of a certain tenure to rebuild their servers. Instead of going through the hassle of upgrading my server instance, I'll let the Amazon staff shut it down on Thursday. All pages will redirect to the appropriate pages on my newest project, EveryBlock.

In many ways, EveryBlock is the next generation of chicagocrime.org. I've often described it to people as "chicagocrime.org on steroids — more than just crime, and more than just Chicago." It's brought to you by the same people (Wilson and me from chicagocrime.org, plus Paul and Dan, who've worked on similar projects), and it has the same philosophies. As we developed EveryBlock, we kept chicagocrime.org firmly in our minds — this new thing we were making had to be a superset, an expansion, a significant step forward. So there's almost nothing you could do on the old chicagocrime.org that you can't do on EveryBlock. And, unlike chicagocrime.org, which was always a side project, EveryBlock has a team of four people improving it full-time, meaning we have the resources to add features, such as e-mail alerts (just added yesterday), that chicagocrime.org never had. We hope EveryBlock is a worthy successor.

This story has a fitting epilogue. In just a few weeks after chicagocrime.org goes offline, the site will be featured in an exhibition at New York's Museum of Modern Art, called Design and the Elastic Mind. Chicagocrime.org will have ended its life and become a museum piece.

Django Book has shipped -- and, thoughts on the next book

It's here! At long last, the print copy of the Django Book has shipped. I received my author copies late last week and am still poking at them to make sure that, yes, a tangible book with my name on the cover has actually been printed, on real paper, by a real publisher.

Early drafts of the book have been available free at djangobook.com for more than a year, and co-author Jacob and I are grateful to all of the readers who submitted corrections and suggestions. Jacob is going to update the site soon with the final text of the book (which will be available free under an open-source license), and we plan to revise the online text over time with corrections and additions. There's something nice about having a paper copy, of course.

The book is available on Amazon, and I'm told the big brick-and-mortar bookstores should begin stocking it soon. Looks like it's gotten some buzz already, as it was the number one best-selling "Software Development" book and number four best-selling "Internet" book. Not bad at all! What I'm most proud of is not the fact that the book itself is doing well, but the larger fact that demand for information about the framework is high.

Now that the Django Book is finally in the can, I'm mulling the idea of writing another book -- this time, a book about online journalism. In the past two years, I've been to (way too) many journalism-related events and conferences trying to spread the good word about "journalism via computer programming," and I've detected a strong, I daresay furious, demand, from journalists at all levels in the org chart, for information about this new form of journalism. Higher-ups want to know why they should employ programmers; middle managers want to know how to find them and how to treat them; and working journalists want to learn these skills and strategies. The problem is that I can't point them anywhere for in-depth information. This book would attempt to solve that.

I want to take a shot at writing a manual, a manifesto, a practical guidebook to this emerging discipline of database-driven Web journalism. It would be a combination of high-level strategy and low-level technique, probably split cleanly into two parts (one for the suits, one for the non-suits).

That's about all the thought I've given to this idea. What do you think? If you're a journalist (or even not), is this something you'd be interested in?

Lead video on YouTube

Wow, somehow my MacGyver theme song video was selected to be the top featured video on YouTube's home page right now. A screenshot, for posterity:

I'm not sure what amuses me more -- the fact that the YouTube editors decided to feature this, or the immediate, endless stream of gratuitous "Please watch my video" comments that the video is getting (and I'm deleting).

Work with me at washingtonpost.com

Attention, Web developers! We're hiring somebody to work with me at washingtonpost.com.

We're looking for somebody who is really good at making dynamic Web applications, on deadline. You're a great candidate if:

You have significant experience building database-driven Web sites.
You pick up new technologies very quickly, enjoy learning new things and enjoy opportunities to apply your new knowledge.
You're great at cleaning digital information -- parsing data feeds, screen scraping, etc.
You enjoy automating things to save people time.
You have experience using Django. Ruby on Rails experience is fine, too, if you're willing to unlearn all that black magic. ;-)
You have a solid understanding of relational databases and experience with open-source databases, particularly PostgreSQL. (MySQL experience is fine, too.)
You are experienced using (X)HTML, CSS, JavaScript, Ajax...yadda yadda yadda.

You get bonus points if:

You've contributed to open-source projects.
You've launched a side project (or two) on the Web.
You have a weblog.
You have journalism experience.
You are passionate about improving the world through information.

In no particular order, here are some examples of the types of sites you'll be building:

It's a mix of short-deadline projects, long-term projects and general site improvements. There's enough variety to keep it interesting. In most cases, you'll be expected to build a site in a matter of hours or days, not weeks or months. It's an exciting, fast-paced environment.

Why should you take this job?

Fun and freedom -- Building Web apps with Django is fun, and you'll have significant say in what your apps should do and how they should work. You won't be a cog in the wheel; in many cases, the development team will be you, or you and I. No requirements documents, if I can help it.
Visibility -- Your work will be seen by hundreds of thousands of people -- maybe more -- around the world.
Cool tools -- You get to use open-source technologies such as Python, Django and PostgreSQL, and get paid for it.
Great people -- Since Day One, I've been continually impressed with the talent and dedication of Washington Post employees. This is the cream of the crop.
Great company -- C'mon, it's the Washington Post, one of the most highly reputable news organizations in the world. The Post is, hands down, the most innovative large newspaper company around. You won't find our killer combination -- dedication to quality journalism and willingness to innovate -- at any other company of our size in this industry.

Journalism experience is not required. A formal computer-science background is not required. I'm much more interested in seeing your work than reading bullets on a resume.

Also, you don't necessarily have to be a designer. Our design team makes things look good.

The job is located in the Washington, D.C., area -- technically, Arlington, Virginia. The washingtonpost.com office is near the Court House Metro stop on the Orange Line.

If you think you're a good fit, contact me. Send some links to work you've done, along with a resume.