Preparing for the 2013 busy season

Preparing for the 2013 busy season

At the end of August 2012, and after nearly two years of development, we launched our new and exciting yearbook creation system along with a raft of changes to our product and pricing. Feedback on these changes has been overhwelmingly positive, however there have also been some comments about features we used to have that were missing.

We’ve been rolling out new features and improvements since launching in August, but unfortunately the speed of the roll out hasn’t been fast enough and there was no easy way to fix this. Our yearbook business is extremely seasonal and it’s crucial that we’re able to fully support all of our customers during the busy period that kicks off this month.

At the end of November I took the decision to take the best bits of our new system and the best bits of our old system and fuse them together. This enables our customers to keep our fantastic new page designs whilst also benefiting from a quicker roll out of features that are required for the yearbook busy season. What’s lost is our new system’s book-centric interface, but we’ve worked hard to ensure this shouldn’t cause any problems. We’ve also lost some aspects of the real-time nature of the new system, but these should gradually reappear over the coming weeks.

Regards,

Jake Gordon
Managing Director
AllYearbooks Limited

Here are some answers to questions you might have about our changes:

Help! Everything looks different!

The interface has changed but all your data remains intact. Please take a few minutes to explore the new interface and do get in touch if you get stuck.

Do I need to create a new yearbook?

No – we’ve copied all your data across to our new system.

Will my or my members’ log in details still work?

Yes.

What’s the new web address for my yearbook?

Use the same web address you were using before.

How do I upload pages I’ve made myself now?

We will relaunch this feature over the coming days.

How does this affect the price of my yearbooks?

It doesn’t.

How will this affect the design of my yearbook?

It doesn’t.

Why isn’t uploading photos as quick and easy any more?

We will be makiing improvements to uploads over the coming days.

As a member, how do I preview what my profile will actually look like now?

Right now, you can’t – you can only see an HTML representation of your entry. However, on saving any answers to profile questions you will be notified how much space you have left to write in, or if you’ve gone over your limit. We hope to add a feature for you to be able preview your profile again in the coming weeks.

Towards real-time yearbooks

We’ve been working hard for well over a year here at AllYearbooks on our new yearbook-creation system, which will completely replace our current site over the coming months. We’ve been keeping it pretty much secret until now but are bursting with excitement to take a few minutes to let you in on what’s changing and why. Oh, and of course we’re hiring (Cambridge, UK) so if this stuff excites you as much as it excites us then do get in touch.

Why we needed something new

Our current yearbook system has been used to create real, hard-copy yearbooks for approximately 150,000 happy customers but is starting to show its age. It’s a fairly traditional LAMP web application, with simple HTML pages and a sprinkling of AJAX to make it more snappy.

Two years ago we worked with UX specialists Clearleft with the remit of fixing the low-hanging fruit in our system without making any fundamental changes. Whilst their changes certainly helped make things easier for users, our greatest lesson was that we gave the wrong remit: our system required fundamental rather than just incremental changes. Some of the lingering issues that people still had were:

  • Members of books only see a textual copy of their profile which will differ in layout from the printed version β€” this also makes it hard for them to know how much to write
  • Editors have to click a button and wait up to around a minute to see changes rendered, then have to browse low-resolution JPEGs of pages or download a sometimes huge PDF
  • Profile page templates are limited β€” images are stuck to a 2×3 ratio and few design options are available
  • If something needed to be changed on a page there wasn’t always an intuitive way for the editor to make the change

These problems all stem from a similar cause, which is a lack of quick and easy feedback for users when they save changes. We made a choice with our old system that profiles should be automatically generated online so that editors could make changes quickly without having to wait hours or days for a designer, but without quick feedback to users on changes our profile designs became very limited. To alleviate issues where members wrote too much text for their entries we made text automatically resize down to fit, but this hurts the look of the pages.

Our solution

So, what are we changing? Everything! We realised we needed a solution that enabled the client to be in constant communication with the server, with changes being pushed out as and when they happened. PHP just wasn’t going to cut it for this task. As a result, our new system uses none of our existing code, and is built from the ground up in CoffeeScript using Node.js on the servers. The primary view of our new system is that of browsing a book, flicking through the pages and making changes without having to go to some random other part of the interface. If someone changes their profile picture or answers their questions then all users logged into that book see those changes almost instantly, without having to even refresh the page.

These changes free us from many concerns that forced our hand in the old system. We couldn’t make templates too custom, because we had no way to show a user what their profile would look like quick enough to be friendly. This is now fixed by rendering the page the moment the user saves changes, sending the client a new render of their page within just 2-3 seconds of the user hitting save. The old 2Γ—3 ratio for profile photos is gone as our templates can now deal with images of any dimension, and after uploading an image the user can instantly see how it will be cropped on the page. Automatic text resizing is also gone, because the user can now see how their text fits and wraps within seconds of saving it.

Architecture

The biggest change for the new system is that the front-end is now a single-page web-app, using Backbone.js which talks to the server and listens over WebSockets (or socket.io fallbacks) for updates. The socket connections are made to a cluster of Node.js processes, synchronising state using Redis. We use PostgreSQL for the database, fronted with a lightweight API upon which the rest of the system is built.

Rendering pages has also changed completely. We used to generate entire PDFs using PDFlib and then rasterise them into JPGs so that editors could preview them. Any time anything changed in the book these files would have to be regenerated, often taking over a minute. Scaling PDFLib to multiple servers would be costly due to PDFLib licensing fees, and we wanted to go straight from code to raster images as well as just PDF. We now use node-canvas (a Node.js module which uses the HTML5 Canvas API) which can take the same code to write either to PDF or images. This lets us generate images directly, and then generate the PDF in full only when required. A future advantage of node-canvas is that our render code is compatible with the HTML5 Canvas element – we did some initial testing with this, but the text rendering was still quite different between browsers, so for now we are keeping page rendering server-side. In future we plan to have previews of changes rendered client-side, and then generate a server-side reference version once the change is saved.

Because we’re rendering pages every time a change is made by any user, rather than just when an editor requests a fresh render, we had to be really smart to keep costs down and performance up. To limit the work needed to render changes, we have a differential approach: we send a JSON representation of both the new yearbook as well as the previously rendered yearbook to the renderer. The renderer then only needs to generate those pages which have changed since the last render. This automatically aggregates things when multiple people make changes to the book in quick succession, and means each yearbook only ever needs one renderer working for it, which helps keep the renderers responsive. The front-end is decoupled from the render jobs (no-one is ever frozen waiting for a render to complete), so they can be delayed when the site is busy, and extra renderers can easily be brought online for busy periods.

Coming Soon

We’re hard at work on the new system, and are excited about getting it launched and providing a brilliant new service for our customers in the coming academic year and beyond. If this work seems interesting, we’re currently hiring a developer, so get in touch!

New bad-ass servers!

If you’re an avid and observant user of our awesome online service, you may notice bump-up in speed and reliability of our website from today onwards. We’ve migrated to some lovely new hardware with more speed, memory and overall grunt β€” meaning faster and easier creation of your yearbook!

We’ve also brought online further servers to help us with two important tasks: creating previews of your books (the ‘Update PDF’ feature) and sending your books to the printing press when all is ready-to-go.

Right now everyone here at AllYearbooks is incredibly busy helping our customers finish the content of their yearbooks off, creating collages, awards pages, covers and lots more. Big boss man Jake is sending dozens of yearbooks off to print each and every day πŸ™‚

Custom backgrounds, messages/signature pages and more (oh my!)

It’s been a fair while since I’d last put my proverbial pen to paper and so I’ve decided to tell you all about our latest enhancements to our site that are almost guaranteed to bring joy to your day. Firstly, we’ve reinstated in the much-needed messages and signatures pages – there are currently twenty to choose from and we’ll likely be adding more as the year progresses:

Adding a messages or signatures page from our collection

Adding a messages or signatures page (when logged into yearbook: 'All pages', 'Add new pages…' and then 'Signature/messages', the fourth option)

Many of our customers choose to have a few pages at the back of their book like these, it makes it feel more professional and complete to have somewhere for your members to sign each others’ books and leave memento messages and farewells. Of course, if you have your own design you can simply upload this as a JPEG or PDF file directly into your book.

We also have a plethora of custom page backgrounds that can be assigned to particular groups of profile pages. If you have a particular background that you want to use, make sure it’s in JPEG format and exactly 2,554 x 3,583 pixels in dimension, then send it as an attachment with an email to our email address (find this under ‘Contact helpdesk’, last link on the menu, in your yearbook).

And finally, we’ve cleaned up the photo gallery area of the yearbook site. It should now be easier to add new photo folders and see an overview of your existing ones:

Cleaned up photo gallery area on yearbook site

Cleaned up photo gallery area on yearbook site

Based on feedback from visitors and customers, we are currently on a mission to improve the ease of use and usability of our website. You should see many gradual improvements of the next month-or-so which will make both your and our lives easier. If you are having any troubles, problems or have any questions about your yearbook or the site, don’t hesitate to have a chat with us about it πŸ™‚

It’s busy season – and next year

It is currently our ‘very busy’ period in the AllYearbooks office; Nic and David’s workload has shot through the roof, there are over two hundred collage pages to create seemingly all the time, lots of deadlines for many of our customers’ yearbooks are coming up soon and monies are flying in like nobody’s business πŸ™‚

Normally I’m busy coding new features, fixing bugs and generally improving our website alongside Jake. But recently I was helping out for a brief period also designing the photo collages and so forth. We’ve hired more help – so far, in the form of Pratik (aka ‘the Master’, ‘Mr P.’ and dozens of other pseudonyms). He’s been beavering away helping Nic and David clear out some of the collage work and doing so remarkably quickly!

Now, on to the exciting news… next year’s website! We’re planning Big Thingsβ„’ for next year; some of the ideas we’ve had so far are:

  • Completely custom templates: This year we’ve been able to help quite a few of our customers out with having their own page design. However, next year a sophisticated, flexible system is currently being built that will allow you to have just about any design you can conjure up! But more importantly: we’ll be able to create and show you the design in breakneck time! πŸ˜€
  • Clean redesign: Jake has been busy prototyping a clean, streamlined and ‘Web 2.0’-feel redesign for our main website (the one you see when you’re not in logged into a yearbook)
  • Going international: we’re thinking about expanding to allow yearbooks to be made from other countries, not just the UK, and hence language support on the website and the final printed yearbooks
  • Facebook app: So many of our current users have an account on Facebook, so we think we should have the ability to complete and manage your yearbook entirely through a Facebook app πŸ™‚
  • iPhone and iPod Touch app: Continuing the trend of making AllYearbooks available whenever and wherever you are, we think an app for these popular devices would be very useful
  • Chat rooms: the ability for all of the hundreds of members in all of our yearbooks being able to chat with each other, their editors and us – the trusty AllYearbooks team

Finally, here’s a sneak preview of what Jake has been working on and constantly refining to make sure that our website is very easy-to-use, modern and accessible:

A screenshot of the work-in-progress prototype design Jake has been creating for next year's website

A screenshot of the work-in-progress prototype design Jake has been creating for next year

Our new updated chat system

We launched a major new feature on the 14th of October last year: a built-in chat system. Since then, we have chatted to a remarkable 144 different customers who have yearbooks with us and in that time over 12,000 lines of chat have been sent back-and-forth between the staff here and our wonderful customers.

When we launched the new features last week, the chat system let us get immediate feedback from you, our customers, and the new site updates were well praised πŸ™‚

Today, we upgraded the chat system to iron out some bugs, and add support for a certain less-capable browser that is still installed in a large proportion of schools and colleges: Internet Explorer 6. Our chat system now works fully on Internet Explorer (version 6 and later), Mozilla Firefox, Safari, Chrome, Opera and more!

So, if you’ve got any questions about your yearbook, the website or anything else, simply login and start chatting to us. During working hours (and often outside of them!) someone will reply within a few minutes, sometimes seconds πŸ˜€

Automated backups

After far too much time and effort, Jake and I have managed to make an automated backup system for the AllYearbooks office. Our main network shared drive will (hopefully) now be backed up on a daily basis to two portable 2.5in SATA hard drives, both which will be stored off-site.

Our main office server which hosts our network shares, backup system for the live website and a plethora of other things now detects whenever one of our USB backup drives are plugged in. The logic is: we plug in one of our backup drives at the end of the day, wait a few minutes for any changes to copy across (thanks to rsync), unplug and store them at home.

More importantly, all of our drives are fully 256-bit encrypted (not just password protected!) meaning that if one of these drives was lost or stolen, it be almost impossible for someone to crack the encryption and gain access to our customers’ data.

You can rest assured that here at AllYearbooks we take data privacy and security very seriously. πŸ™‚

Site Development – Part 2: Yearbook PDFs

As David pointed out in the last blog post, we are currently under heavy development here at the AllYearbooks’ offices busy developing new and exciting features for the coming academic yearbooks. Apart from the chat system (which is already working) we’re well on the way to implementing a new way of generating the final PDF file for each yearbook.

Our system is quite unique in that as you progress building your yearbook β€” inviting new members, uploading photos, adding custom pages and so forth β€” you are given the opportunity to preview exactly how your book will look once it’s gone to print. πŸ™‚ We’re keeping this useful feature but making important changes:

  • Allowing members to upload two profile photos each
  • Making our templates accommodate this second photo
  • Text that wraps and flows professionally around objects on the page
  • Rounded corners on borders and photos

The greatest change, though, will be that because of the new programming of our PDF system, we will soon be able to create unique, custom templates when one of our lovely customers requests it. For example, if the selection of templates provided doesn’t suit you or you like the look of one but would prefer it with a few tweaks here-and-there, we will be able to modify an existing template or create a brand new one for your yearbook.

Compared to our competition, I think this is getting the “best of both worlds” β€” everyone uploads their entries and photos online, the editors choose or requests a template and the final yearbook feels impressive and gleams with professionalism. Plus, unlike some of our competitors, you won’t need to wait a week or more for your template to be generated for you πŸ™‚

Updating the AllYearbooks website

Well, I’m finally doing a monster chore that I’ve been putting off and putting off – updating the main ‘brochureware’ website with all the latest text and images.

The text and images are already written and taken – they’ve all been used in our brilliant information packs. The problem is there’s just been soooo much to do and this is such a long and boring task that I’ve been constantly pushing it back. Well, it can wait no more so I’m trudging through it now. Expect to see lots of new text and images around the website over the next couple of days.

The image below is the dreaded ‘todo’ card. Its all part of the ‘agile’ system of software development. Whenever something pops up that needs to be done, we jot it down onto an index card and add it to the todo pile. Then once every Tuesday morning Jamey and I have a meeting where we go over all the current todo cards and figure out which ones we’ll both be tackling that week, and in what priority. This particular card has been sitting there staring me in the face for weeks!

Whenever our customers come up with ideas for the website, we put them on index cards too. We love getting feedback and ideas from you – so keep them coming on the contact page.

Using less Drupal

Lately, I’ve had various frustrations with Drupal which have moved me away from using it for various things. I’d like to go through where I’ve moved away from Drupal, why I’ve made those changes, and my future Drupal decisions.

WordPress rather than Drupal blog

To begin with, this blog is now on WordPress rather than Drupal – and I have to say that I’m loving it… and so are my non-geeky colleagues. It ticks all the right boxes. Its *really* user friendly. Its much easier to add photos (and videos) to posts. And it’s hardly taken long at all to setup.

So where did Drupal go wrong with this? Well, I guess its the ‘kitchen sink’ approach back-firing. In trying to be something to everyone, Drupal runs the risk of being less than perfect to to any one specific task too. WordPress, on the other hand, has the ability to focus on being the very best blogging software out there, and nothing else gets in the way of that or deters it from its ultimate goal.

But also, we needed to seaparate the business end of our website (the creation of yearbooks, using tens of thousands of nodes) from the nodes and users to do with the blog, partly because around this time every year we flush out the old site and start a new. So we were either going to have a separate Drupal install for the blog or use WordPress. We chose WordPress.

Form theming frustrations

Have a look at the form here and let me know how you’d create it using FAPI (we’re on Drupal 6.4). I’m talking specifically about the theming of the form. Yes, its a very simple form. A search form. But let’s have a look at what’s going on with it and discuss the Drupal way versus the way we ended up doing it.

In FAPI, you’d probably have a ‘textfield’ element for the search box and a ‘submit’ element for the ‘Go’ button. Easy enough, one minute of code. But what about the title ‘Name of your school or group’? Probably a title for the textfield element, no? But then how do we get it centred above both the textfield and the submit button? And what about the text under the two fields? A description, right? Again… how do we get it to appear *exactly* where we want it? The look and feel of the form are to me absolutely crucial. I don’t just want a textfield’s title (with annoying colon after it), a textfield, a description, and then a Go button all one on top of the other.

The Drupal solution? Using a theme hook. We define the form in our implementation of hook_form() and then theme that form separately. The ‘programmer’ cares about the functionality of the form but not the visual design of the form. The ‘designer’ doesn’t care about the functionality and instead works on how it looks. But I’m both the programmer and the designer here, and I want my work to be as easy as possible! So, let’s say i go down this route (I tried, I really did). I need to register my theme function in hook_themes(). Okay, I know why, it saves extra code running on every page load. But its still annoying. Now I create my theme function… urgh. You’ve got to really know your FAPI stuff to get this to work. I try for a while but then I give up. It just feels so messy with some code somewhere, some in another place, and then when I ask my colleague to have a look so he can learn how to do it he’s disgusted… doesn’t know what’s going on… starts bad-mouthing Drupal. So we build our own massively simple FAPI instead in about half an hour that does just what we want it to do.

So now we’re using our own super-basic FAPI for this form. Not all forms, just the ones we want complete control over, visually. Rather than using hook_form() and defining a form array, we just hard-code the HTML for the form. Some of you may be in complete horror now thinking about this but its just by far and away the easiest way to get forms to do exactly what you want. Like a forename and surname field side by side rather than one on top of the other, sharing the title ‘Name’ which is a label for the forename field.

We’re sticking with the idea of a validate() and submit() hook though. I like that one. But we’re doing it slighly differently and more simply, so that any new coders we might hire can quickly and easily pick it up.

Going nodeless

I don’t always like nodes. I really, really don’t. I don’t need revisions, and if I did I’d do them in my own way, just the right way for me, rather than a way that kind of works for everyone but not quite perfectly for anyone in specific. I don’t like the way that as uid=1 I get all the extra bits like ‘Authoring information’ which I never touch. Node hooks and the nodeapi which I once loved are now a higgeldy-piggledy mess that’s a real pain for my new hire. I try to explain to him what’s going on when I save a node. “So this function deals with the submission. But not the core node stuff, Drupal deals with that. And if we want something used for all nodes, we put it in here instead. And we can also override this specific bit here.”… he looks on in amazement, totally baffled by what’s going on and why. It would be so much easier for him (and me) to understand if everything’s just in one place.

So what do we gain from nodes? Umm… not much really. We don’t use contributed modules any more because they never do exactly what I want and always do stuff which I don’t want them doing which just make them less efficient. We put all our code in our own one module instead. A massive, hefty module with a dozen or so include files.

We gain the ability to always do $node->nid and use node_save() and other handy things. But we don’t really need nodes, and it frustrates me having to do the extra INNER JOINs on node_revisions etc. So we’re trialing not using nodes at all for one of our content types – our customers. We just have a simple ‘id’ field now in one single table. We no longer need to INNER JOIN node and node_revisions. We haven’t had any problems so far, but the new hire is finding it much easier to code now, without the ‘baggage’ of Drupal.

The future

Our current plan is to gently migrate away from Drupal, perhaps altogether. We like the idea of building our own framework again, one that does exactly what it needs to do for our site. Its not something we can do overnight. Ours is a yearly cycle, following the academic year, and the current plan is to fork the codebase in around January/February and that would mark the beginning of our own framework if we still feel that way then.

In the meantime, we’ll continue using nodes for most of our content types (if simply because migrating away from them would be a long and arduos task with little reward) and we’ll continue to use FAPI for most our forms. But I see us using our own simple FAPI for more and more forms where we need complete control over them, and I see us extending this FAPI to help us reduce using the same code multiiple times.

I think I still like Drupal. I definitely appreciate the vibrant community. But sometimes I thoroughly hate Drupal and get massively frustrated by it. But I still like it in theory at least. One framework for all my websites. But whilst I just work on one massive website it just has so much less use to me.

You’re more than welcome to urge me to stay with Drupal. In fact, I highly hope someone can manage this. I’ve put a lot of time over the last few years into learning Drupal, and it would be a great waste and a shame to lose all that.