Towards real-time yearbooks

We’ve been working hard for well over a year here at AllYearbooks on our new yearbook-creation system, which will completely replace our current site over the coming months. We’ve been keeping it pretty much secret until now but are bursting with excitement to take a few minutes to let you in on what’s changing and why. Oh, and of course we’re hiring (Cambridge, UK) so if this stuff excites you as much as it excites us then do get in touch.

Why we needed something new

Our current yearbook system has been used to create real, hard-copy yearbooks for approximately 150,000 happy customers but is starting to show its age. It’s a fairly traditional LAMP web application, with simple HTML pages and a sprinkling of AJAX to make it more snappy.

Two years ago we worked with UX specialists Clearleft with the remit of fixing the low-hanging fruit in our system without making any fundamental changes. Whilst their changes certainly helped make things easier for users, our greatest lesson was that we gave the wrong remit: our system required fundamental rather than just incremental changes. Some of the lingering issues that people still had were:

  • Members of books only see a textual copy of their profile which will differ in layout from the printed version — this also makes it hard for them to know how much to write
  • Editors have to click a button and wait up to around a minute to see changes rendered, then have to browse low-resolution JPEGs of pages or download a sometimes huge PDF
  • Profile page templates are limited — images are stuck to a 2×3 ratio and few design options are available
  • If something needed to be changed on a page there wasn’t always an intuitive way for the editor to make the change

These problems all stem from a similar cause, which is a lack of quick and easy feedback for users when they save changes. We made a choice with our old system that profiles should be automatically generated online so that editors could make changes quickly without having to wait hours or days for a designer, but without quick feedback to users on changes our profile designs became very limited. To alleviate issues where members wrote too much text for their entries we made text automatically resize down to fit, but this hurts the look of the pages.

Our solution

So, what are we changing? Everything! We realised we needed a solution that enabled the client to be in constant communication with the server, with changes being pushed out as and when they happened. PHP just wasn’t going to cut it for this task. As a result, our new system uses none of our existing code, and is built from the ground up in CoffeeScript using Node.js on the servers. The primary view of our new system is that of browsing a book, flicking through the pages and making changes without having to go to some random other part of the interface. If someone changes their profile picture or answers their questions then all users logged into that book see those changes almost instantly, without having to even refresh the page.

These changes free us from many concerns that forced our hand in the old system. We couldn’t make templates too custom, because we had no way to show a user what their profile would look like quick enough to be friendly. This is now fixed by rendering the page the moment the user saves changes, sending the client a new render of their page within just 2-3 seconds of the user hitting save. The old 2×3 ratio for profile photos is gone as our templates can now deal with images of any dimension, and after uploading an image the user can instantly see how it will be cropped on the page. Automatic text resizing is also gone, because the user can now see how their text fits and wraps within seconds of saving it.

Architecture

The biggest change for the new system is that the front-end is now a single-page web-app, using Backbone.js which talks to the server and listens over WebSockets (or socket.io fallbacks) for updates. The socket connections are made to a cluster of Node.js processes, synchronising state using Redis. We use PostgreSQL for the database, fronted with a lightweight API upon which the rest of the system is built.

Rendering pages has also changed completely. We used to generate entire PDFs using PDFlib and then rasterise them into JPGs so that editors could preview them. Any time anything changed in the book these files would have to be regenerated, often taking over a minute. Scaling PDFLib to multiple servers would be costly due to PDFLib licensing fees, and we wanted to go straight from code to raster images as well as just PDF. We now use node-canvas (a Node.js module which uses the HTML5 Canvas API) which can take the same code to write either to PDF or images. This lets us generate images directly, and then generate the PDF in full only when required. A future advantage of node-canvas is that our render code is compatible with the HTML5 Canvas element – we did some initial testing with this, but the text rendering was still quite different between browsers, so for now we are keeping page rendering server-side. In future we plan to have previews of changes rendered client-side, and then generate a server-side reference version once the change is saved.

Because we’re rendering pages every time a change is made by any user, rather than just when an editor requests a fresh render, we had to be really smart to keep costs down and performance up. To limit the work needed to render changes, we have a differential approach: we send a JSON representation of both the new yearbook as well as the previously rendered yearbook to the renderer. The renderer then only needs to generate those pages which have changed since the last render. This automatically aggregates things when multiple people make changes to the book in quick succession, and means each yearbook only ever needs one renderer working for it, which helps keep the renderers responsive. The front-end is decoupled from the render jobs (no-one is ever frozen waiting for a render to complete), so they can be delayed when the site is busy, and extra renderers can easily be brought online for busy periods.

Coming Soon

We’re hard at work on the new system, and are excited about getting it launched and providing a brilliant new service for our customers in the coming academic year and beyond. If this work seems interesting, we’re currently hiring a developer, so get in touch!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s