HTML is my API

In August 2012, I wrote a quick script to stream front-page Hackernews stories to an IRC channel on Freenode (##hackernews in case you're interested) so that I could quickly glance at popular stories there instead of needing to load Hackernews. Since IRC is my feed reader, I've always tried to pipe as much there as possible.

Parsing the Hackernews front page HTML

In 2012 there was no API for Hackernews, so my only option was to use one of the unofficial ones, or read the HTML from the front page myself. I opted to parse the HTML directly since it wasn't super complex. I know you shouldn't parse HTML with a regex, but I did anyway. It worked great, from August 2012 up until April 2015.

Finally this month I stopped seeing updates in the channel, and went to take a look. It appears they made a slight change to the HTML which of course broke my regex, since I was matching against some specific markup. I also learned from someone else in the channel, that Hackernews had launched an official JSON API in October. I decided to rewrite the code to work with that API rather than try to update my regex.

Switching to the JSON API

It took only a few minutes to rewrite the script using two endpoints, topstories.json and fetching an item. However, within two days, I had already encountered a problem. The IRC bot started spitting out empty lines in IRC, one for each front-page story. My code was by no means foolproof, and I realized that this would happen if the API call to fetch story details returned an empty result.

So in 2.5 years of parsing the HTML, I never had any problems. In 2 days of parsing the JSON API, I hit a glitch where all the stories were empty.

2015-04-26 19:43:03     Loqi    [hackernews]  ( points) https://news.ycombinator.com/item?id=
2015-04-26 19:43:06     Loqi    [hackernews]  ( points) https://news.ycombinator.com/item?id=
2015-04-26 19:43:08     Loqi    [hackernews]  ( points) https://news.ycombinator.com/item?id=
2015-04-26 19:43:11     Loqi    [hackernews]  ( points) https://news.ycombinator.com/item?id=
2015-04-26 19:43:14     Loqi    [hackernews]  ( points) https://news.ycombinator.com/item?id=
2015-04-26 19:43:17     Loqi    [hackernews]  ( points) https://news.ycombinator.com/item?id=
2015-04-26 19:43:20     Loqi    [hackernews]  ( points) https://news.ycombinator.com/item?id=

Make the visible data machine readable

Since more people and programs see the HTML than use the API, the HTML ends up being more reliable. Luckily there's a simple solution to avoid nasty regex parsing hacks in order to be able to generate a machine-readable (and JSON!) version of an HTML page. It requires the author of the HTML to add a few classes to their existing markup. We'll use the Hackernews front page markup as an example.

Here's a snippet of HTML from the current Hackernews front page for a single story.

<tr class="athing">
  <td align="right" valign="top" class="title">
    <span class="rank">19.</span>
  </td>
  <td>
    <center><a id="up_9443241" onclick="return vote(this)" href="vote?for=9443241&amp;dir=up&amp;auth=4f617c803a44687927291cd133ac825ca9e61c4b&amp;goto=news"><div class="votearrow" title="upvote"></div></a></center>
  </td>
  <td class="title">
    <span class="deadmark"></span>
    <a href="http://www.dmiller.io/blog/2015/4/26/comparing-the-php7-and-hack-type-systems">Comparing the PHP 7 and Hack Type Systems</a>
    <span class="sitebit comhead"> (dmiller.io)</span>
  </td>
</tr>
<tr>
  <td colspan="2"></td>
  <td class="subtext">
    <span class="score" id="score_9443241">40 points</span>
    by <a href="user?id=jazzdan">jazzdan</a>
    <a href="item?id=9443241">7 hours ago</a>
    | <a href="flag?id=9443241&amp;on=t&amp;dir=down&amp;auth=4f617c803a44687927291cd133ac825ca9e61c4b&amp;goto=news">flag</a>
    | <a href="item?id=9443241">20 comments</a>
  </td>
</tr>

If this looks like a chunk of HTML you don't want to touch with a 10-foot pole, I don't blame you. In order to make this easily machine-readable, without requiring custom parsing of HTML, we can add a few Microformats classes to turn these into h-entry posts.

First, wrap both table rows with a <tbody> tag with a class of h-entry in order to group these rows into a single element. (And yes, it's okay for a table to have multiple tbody tags.)

<tbody class="h-entry">
...
</tbody>

Find the line that contains the permalink to the story as well as the name, and add the u-url and p-name classes:

<a class="u-url p-name" href="http://www.dmiller.io/blog/2015/4/26/comparing-the-php7-and-hack-type-systems">Comparing the PHP 7 and Hack Type Systems</a>

Find the link to the person who submitted the post, and add the p-author class:

by <a class="p-author h-card" href="user?id=jazzdan">jazzdan</a>

Lastly, indicate the time the post was created as well as the post's permalink on news.ycombinator.com:

<a class="u-url" href="item?id=9443241"><time datetime="2015-04-26T14:15:00-0700" class="dt-published">7 hours ago</time></a>

Here's a summary of the changes in colorized diff format. You can see the additions are extremely minor, and didn't involve changing the structure of the HTML with the exception of adding the <tbody> tag.

de blob-code-context blob-code-deletion">- <a href="item?id=9443241">7 hours ago</a>





+ by <a class="p-author h-card" href="user?id=jazzdan">jazzdan</a>





+ <a class="u-url" href="item?id=9443241"><time datetime="2015-04-26T14:15:00-0700" class="dt-published">7 hours ago</time></a>





| <a href="flag?id=9443241&amp;on=t&amp;dir=down&amp;auth=4f617c803a44687927291cd133ac825ca9e61c4b&amp;goto=news">flag</a>





| <a href="item?id=9443241">20 comments</a>





</td>





-</tr>








+</tr>





+</tbody>