Tuesday, August 10, 2010

On Javascript Dependencies

I've been reading some posts about large-scale, or "enterprise", applications involving Javascript. As a brief aside I'll note that to me, to call software "enterprise" is to pronounce a terrible condemnation upon it, to evoke my memories of awful intranet applications, sprawling old distributed client-server systems, horrible user interface designs obsolete at their debut, and tremendous, sometimes astronomical costs of purchase and deployment. Though I'm still basically doing "enterprise software development" I never say those words to myself. But that's just a personal problem. It's funny how words get poisoned.

Anyway, here's an interesting article I just read this morning. I'm really impressed at some of the apparent capabilities of those tools the article discusses. However, the "standard folder structure" described therein evinced a distinctly different approach to the management of client-side code (well, client-side everything, by extension) than what I've come to think of as the One True Way. It's always interesting to read something like that and have one's own notions illuminated by glaring contrast with someone else's. I'm not going to say my ideas are better of course, but I think they're fundamentally different. It's going to be a little hard to describe why, because I don't think I'm equipped with the terminology necessary to discuss the subject; I'm not even sure I know exactly what the subject is. Nevertheless, as I've been preparing to do a quick presentation about the little wallflower jQuery plugin that I wrote a few weeks ago, I've been forced to think not only about the way I actually go about designing and arranging and managing my application as it grows, but also about the way I think I should be doing it.

Back a really long time ago, in the early '90s, one of the trendy must-read business books of the time was Peter Senge's The Fifth Discipline. I guess it says something about how business/management improvement trends go that you can pick up copies of that book now for the cost of shipping. In other words, it's almost literally worthless. But it's a pretty good book, as things like that go. Senge's an interesting guy, an engineer by training who got all fired up about the topic of "organizational learning", and that's what he's been doing at MIT's Sloane School of Management for a couple decades. The book is basically about that: what it means for an organization to learn.

Much of the material in The Fifth Discipline is sourced from the work of two other Sloane School researchers, Chris Argyris and Donald Schön. Since the 1950's, those two have been doing some really interesting work on the theory and mechanics of learning, both individual and organizational. Check out Dr. Argyris's Wikipedia page for a brief taste of what he's written about. One of the lasting concepts I took away from one of his books (which, you'll note, is not considered worthless) is an idea that Senge also latched onto, that of recognizing and contemplating the distinction between theory in use and espoused theory. An espoused theory is something you type into a blog post about how everyone should architect their enterprise application. A theory in use is how a development team actually does it. (Seriously, that book is one of those eye-opening, mind-opening books that you just can't forget, providing enormous insight into all the stupid things that go on in organizations large and small. If everybody read and believed in the stuff Argyris and Schön have written about, the world would be a better place.)

Now I'm not going to pontificate about how you should architect your web applications. My point in bringing up those books and those ideas is that writing about how one does things can be pretty humbling. I know that the way I actually work isn't as close as I'd like it to be to the grandiose fantasies I entertain about The Perfect Application. More than that, however, what I learned from those books is that I should not only be aware of the ways my actual work doesn't live up to my ideals, but also to be aware of the reasons I think those ideals are ideal. What do I want from my application design, and why do I want it? If I were able to fully realize my ambitions for perfect design, would I perform some sort of test to see if I actually was getting what I wanted? That's something that's generally left out of most process improvement activities — and I mean process improvement activities as actually done, not as they're described in books and blog posts. How many teams have you worked on that, after taking up some trendy activity like stand-up meetings, have actually applied an even vaguely scientific testing method to see whether the new activity was helping? How many really analyze the goals for starting their process improvement in the first place, and subsequently test to see if achieving the goals helps out their business bottom line? I think the same rigor should apply to technical processes like application construction and code management, but in my experience that's not so common.

One of the difficulties I'm having with this subject is that I don't know what to call the central activity. That activity is the construction of the application code behind an application, and the design for how that construction should proceed. I use the term "construction" because I've been driving through a bit of it for several months now on my way into and out of town. Civil engineering projects involve an outward architecture: somebody designs what a bridge should end up looking like, the shape of the arch, the look of embankments, the decorations on railings and lane dividers, and all those other things that'll be mostly ignored by the public when the project is complete. However, I suspect (knowing essentially nothing at all about what civil engineers do) that a whole lot of work goes into the design of the construction effort itself. I'm sure that there's a tremendous amalgamation of experience that instructs thousands of planning decisions for a big roadway project. Those must be planned with a rigor that I've never seen even distantly approached in any software project I've experienced. Maybe I've just been unlucky, but I don't really think so.

What do I call, then, the design and planning of software implementation, and by extension the theory of that design? The term "application design" seems to be understood as being more about what an interface looks like and what application users should experience. "Coding practices" to me means "tabs or spaces" and details like that. I'm going to use the term "application implementation design" because I can't think of anything else. I'm absolutely not an expert, of course, as should be clear by this admission that I literally don't know what it is that I'm talking about, or at least what to call it.

Now back to Javascript. What is it that I want from the way I type in Javascript code? To put it another way, what is it that I want to be true at some point in the future when the surrounding application is, if not "finished", something large and functional and under heavy use? Of course it's clear that I want it to work and to work well. That means it doesn't break in small browser windows, it doesn't slow down pages, it doesn't annoy users, and so on, satisfying all those little de rigueur stipulations we twitter about constantly. But I also think there are aspects to the application that are of great importance to the developer experience. What should it be like to add a new feature? To revise an old feature? To fix bugs? To cope with off-the-shelf framework and tool upgrades? And once I've answered those questions, why do I think those are good goals? In other words, why would it be good for the process of fixing a bug to play out in some particular way? Thought of that way, it's a pretty big topic, and though I've been doing it for almost 30 years now it's challenging to tease apart good experiences I recall from the bad ones.

Please Either Shut Up Or Talk About Javascript

Here's the point I'm trying to make. My own goals for the shape of my application code — and not just where the files are in folders, and what the folders look like, and how they're arranged, but the underlying ways in which the code all fits together — are mostly summed up like this:
I want to minimize the number of dimensions of code expansion that result from a given increment in functionality.
In other words, I want it to be the case that when new functionality is called for, a developer should have to type in as little new code as possible. OK well that's pretty obvious; that's what everybody wants. The tough part, however, is being sensitive about what sort of new code we consider to be "matter of course" new code — code that we think of as having to be written due to laws of nature. That's tough because it's easy to grow numb to some things that really aren't laws of nature, but consequences of application implementation choices. For example, in my current application, when I need a new database table or changes to an existing one, I type in a little SQL. To a lot of people nowadays, that seems horrible (or at least that's an impression I've formed). The framework should do that! Well, maybe; I'm aware of how much that part of my world deviates from some current fashion, but in that particular case I actually prefer it that way. In other words, I'm aware of the fact that I've chosen a scheme that requires me to type in some SQL manually, but I know what my reasons are and I'm OK with that.

In a former position, I worked on a really large web application: a huge pile of Java on the server side, almost a thousand separate .jsp files, and lots and lots of Javascript. The application implementation design in that situation was such that implementing new pages meant, more-or-less as a matter of course, writing Javascript code for all the widgets on each page. Though there was some re-use of some components, like confirmation dialogs for example, the Javascript required to integrate a confirmation dialog with a particular new page was pretty much unique to the page. With relatively little experience dealing with an application of that magnitude (and at the time the development effort was really ramping up, back in 2003 or so, there probably weren't a lot of people around anywhere who had much perspective on what a huge AJAX-heavy application would be like for the developers involved), all of that per-page plumbing seemed pretty natural to me. I got pretty good at sort-of automatically typing in the stuff I needed, knowing how to think ahead for what a particular sort of page was likely to need. Lots of the script code was directly in the pages because it was often reliant on bits of information that'd be dropped into the code by the server-side template engine (that is, via JSP JSTL/EL expressions).

How else could such code possibly be written? Well I didn't actually ponder that question very often, at least at first. Probably sometime around the hundredth new page, however, I started to get this nagging feeling that something was wrong. At that point, I started to see the value (now so glaringly apparent) in common modular Javascript components. And as I started writing those, I started to also see the need for some sort of mechanical way that the chains of component inter-dependencies could be dealt with automatically. If there were some way that the component dependencies could be declared, that'd factor out a big headache from page construction and maintenance. We never got there, but it sure seemed like one of those "ideals".

When I started work on my current application, I spent a lot of time looking around at various dependency management tools and modularization techniques. Many of them looked great, but I didn't really dive in because I was also immersed in the problems of getting an application concept started from scratch. I'd never really done that, so I had to spend a lot of time thinking both about the outward-facing design as well as the application implementation design; the first problem generally got the lion's share of my anxiety. I was also spending a large amount of time thinking about server-side application implementation, so the Javascript module problem didn't get a lot of attention.

All this time, another "ideal" banging around in my head was a desire to really get away from in-page Javascript, and to get much closer to a universally unobtrusive way of creating pages. My server-side investigations had led me to the Stripes framework which seemed like some magical heaven-sent gift written by people who shared most of my opinions about that problem space, and were a million times smarter than me too. One of the nice things Stripes provides on the page construction side of things is a simple but powerful layout templating system. Once I started using that, it became clear that if we were careful the pages we'd be writing for application features could be really succinct. The layout templates could handle just about all the boilerplate page code that, in my previous position, had definitely seemed like a law of nature.

Well stuffing all the boilerplate into layout templates brought an implication to the domain of managing the Javascript code that supported fancy page behaviors. If I were going to have an implementation design based on the idea that every page would somehow grab the components it needs, then dependency management or not there'd have to be some communication of those needs between pure page code and the layout templates; otherwise, how would the Javascript end up at the browser? For all their convenience, the mechanisms available for pages to "talk to" the Stripes layout template code are a little crude. On top of that, of course, the templates themselves are JSP code, and I really don't like writing code that has to "think" very much in JSP; it's hideous. (Is it a law of nature that such page-template communication should be hideous? Probably not ...)

This situation drove me down the path I'm on now. The more I tinkered with it, and the more that some early deadlines loomed in getting the application put together, the more it looked like the simplest thing to do would be to just stick a common block of script tags in the layout template, and have every single page just pull in essentially all the Javascript I had. I had concerns about performance of course, naturally imagining the worst about what that could lead to in the future. However, I had to get some pages working and make it possible for other people on the team to write pages too, so it was the simplest place to start.

The implementation of my unobtrusive behaviors proceeded, generally on an as-needed basis. A new page would need something, and I'd cobble together some way for the HTML to announce that need (a new class name, or grouping under a particular element id, or whatever) and write the Javascript to deal with it. The code evolved along the lines of little stanzas like this: $where.find('.dialogLink').each(function() { var $link = $(this); // ... }); Each page feature, in other words, had a chunk of code that would look for the tell-tale mark in the HTML indicating the need for some activity. The "$where" variable was the DOM root under which I needed the work done, and was generally either the page body as a whole, or else a piece of the page into which new content had just been dropped. I was lumping all this code into a single routine that would be applied upon page load, and also when page fragments were loaded dynamically. As you can imagine, this function became ridiculously big after only a short while. I knew it was awful, but it did work after all and I had other things to worry about.

To address the problem of the insane size of my "do all the behaviors" script, I eventually broke up the code into lots of separate files, basically organized around the page features they supported. That helped a lot with keeping the code base manageable and maintainable; fixing a behavior meant a change only to a small file, not touching any other Javascript in the repository. However, I couldn't possibly stand the idea of extending my layout template to include thirty or forty separate script tags, even if they were small.

To solve the script tag problem, I leveraged the fact that as a Java application I had a build system anyway. I added an Ant task to the build that leveraged another tool we were also incidentally using, the Freemarker template engine. The Ant task processes a small Freemarker template file that is a Javascript skeleton. Freemarker directives in the template call out to routines in the Ant task that find those little script files in the source tree and supply them for inclusion in the Freemarker output. The result is a horrible-looking yet syntactically correct monolithic Javascript file that gets cranked through YUI compressor (giving the unanticipated but welcome benefit of a Javascript syntax check at build time) and dropped into the final .war file. I even concatenate the jQuery and jQueryUI libraries into my script, so ultimately what resulted was something that I would have considered completely insane just a few months previous: every single one of the pages of this application import exactly one script, that being the sum total of all the Javascript I've got.

That's obviously a disaster, right? One huge (about 250K at present) script applied to every single page? All those jQuery selector searches eating up time when 90% of them aren't going to actually do anything? Well, yes, early on we did have some performance problems. Part of that was foolish ignorance, mostly around my failure to use smart event handling (bubble-based handlers with jQuery "live" and "delegate"). Part of it was badly-written selectors, like that ".dialogLink" above. So we did spend time worrying about that and tuning the script, and I experienced a lot of anxiety at the prospect of having to break it all up. Eventually, however, we crossed a line due to good ideas from my co-workers and good ideas from blogs and forums, and I realized that my gigantic monolith of code really wasn't posing a performance problem after all, at least not simply due to its bulk. I've now got good selectors for everything, and dealing with all that Javascript takes only a very short time as my pages are painted by the browser. One of the benefits of having just one file, of course, is that it's the only script a client browser has to cache.

The mechanism supporting all my unobtrusive page features was still a mess, however, and that's what led me to type in the wallflower code. I wanted to see what a more organized approach might look like. If and when I actually try and convert my real code to use wallflower, I'll probably stick with the idea of the build-time "assembly" step, so that my individual wallflower features will live on as separate source files. But it's been a really long time since I've spent any time at all thinking about a component dependency manager. Why should I? I've "solved" the dependency problem (without explicitly planning to do so, and in fact without really being aware that I was doing it) by having exactly one common dependency for every page on the site. That's just hard-coded into the template, and I'm done. New page with new needs? Sure, I'll just add a new little behavior script file, and it works. Given how fast things currently run (and while it's truly breathtaking how fast modern Javascript interpreters are, this stuff works fine even in IE7), I doubt I'd see problems if I had 3 times as many behavioral features to support. I doubt I'll get there, of course, mostly because a site with that many whiz-bang page features would be terribly annoying (I'm looking at you, mint.com).

Another unanticipated benefit — at least, I think it's a benefit — of all this is that it's led me to having a much more disciplined approach to adding fancy behaviors to the site. In my previous job, because the way code was written was almost certain to involve so much specific new code to be created for everything, having unique behaviors on individual pages for no really good reason was pretty common. There was nothing in the process standing in the way of that, after all. Now, when thinking of new application features, I tend to think in terms of things I can already make pages do. Yes, there are two or three "special" parts of my app that do things no other pages do, and I confess to a few pages with on-page Javascript (I'm working on it, really), but keeping things under control I think really helps with maintaining consistency, and that's good not just for the development team but also for the end user.

So that brings me to this morning. I've been typing in this ridiculously long ramble for a while, but not until I saw a couple of posts about "enterprise" components did a unifying theme really click in my head. I'm delighted that people are finding success with smart dependency management schemes, and I'm sure I've got a lot to learn about large-scale web applications from other people approaching the problem in more well-informed ways than I did. (That'd probably be just about anybody doing it.) However, there's a lot I like about the world I've made for myself. What are the "laws of nature" that I'm assuming? Where are all my unnoticed wastes of time? Would one giant script file really work out in an application of the scale that my previous position involved? I'm not really sure at the moment. I also don't know whether what I'm doing is something that lots of people do, that's been written about elsewhere, possibly extensively, and possibly with well-reasoned heavy criticism. If you know, and you're not too exhausted after reading all this stuff, leave me a comment or two.


  1. I think you make some very salient points, and give some great insight into what making applications on the web in 2003 must have been like.

    I am a firm believer in putting together the right tools for the job. I think you hint at that a few times in this post. That sometimes, a single integrated solution has too many unnecessary parts, or too-general of implementations. I think a balance has to be struck somewhere between "full integrated framework" and "custom everything." Sometimes that can be a full featured toolkit like Dojo, or perhaps even a Cappuccino or Sencha type app.

    More often than not, though, we like to think our applications are running at a scale they will never achieve, and will have to be modified by future developers that will never exist. Obsessive planning for these things slows down and confuses development at some point. I'm not claiming to know where the line is, but I'd venture to guess, for your tango health stuff, even though $.wallflower isn't some hugely integrated solution that everyone uses, etc, it's a perfectly valid organizational choice. It makes sense for the application size, etc, and any programmer that can spend 5-10 minutes looking at it could grok it and start using it.

    As far as your dependency resolution stuff goes, it certainly depends on the app. I actually help with an extremely large java/freemarker app with a build system at work and they used to do the "chunk everything into a giant file" thing. We recently switched to RequireJS, to manage dependencies, because the scale that we had reached no longer made sense with the other way.

    TL;DR: There's a ton of value in being able to choose the correct tools for a job. Sometimes that solution is popular and cool and crazy-integrated, and sometimes its your own plugin. That's cool. People who understand JavaScript will be able to figure it out.

  2. Great post and, in a post-modern fuzzy sorta way, echoing my own concerns about developing and scaling apps/systems.

    A key but unrecognized problem in code-creep can be the fact that it's so easy to change anything. Even if it's just me developing, it's too easy to change anything anywhere and bolt on functionality where it doesn't belong. Solution: compile functions into (e.g.) DLLs. Sounds crazy but it keeps you from breaking your own code and it makes you define small, flexible interfaces which, I think, are a key to good SW.