A common issue with web software is cross-site scripting attacks — the ability for a third party to inject HTML elements into pages displayed to other users, using scripts contained in those elements to capture user cookies or perform operations on their behalf.

The technical challenge in solving this is that whenever data is being output through a HTML page, it should be escaped — any special HTML characters should be turned into their non-special versions in order to be displayed verbatim. This is an ongoing effort: each new page and each new variable on a page involve the same amount of effort to be done.

Of course, the solution would be to decide that escaping string output should be a default behavior that must be explicitly overriden. This does create issues where HTML is escaped when it should not have been, but:

• These issues cannot be used to perform attacks.
• They are usually easier to reproduce and consequently to solve.
• HTML usually comes from template files, which can be handled with a different default.

Indeed, I can guarantee that my software has zero vulnerabilities related to escaped HTML, because I have built into the type system the fact that HTML always comes from templates, and the method that injects variables into templates escapes them. If I try to use a string as if it were HTML, I get a compiler error.

Even without a type system, one can guarantee that the system would rather break at runtime than allow an injection, using the exact same design, with incompatible data structures for templates and strings that blow up when a string is used as a template:

class FilledTemplate {
  function __construct($html) {
    $this->_html = $html;
  }
  function html() {
    return $this->_html;
  }
}

class Template {
  function __construct($file) {
    $this->_template = file_get_contents($file);
  }
  function fill($values) {
    $replace = array();
    $with    = array();
    foreach ($values as $key => $value) {
      $replace[] = '{'.$key.'}';
      if ($value instanceof FilledTemplate) 
        $with[] = $value->html();
      else 
        $with[] = htmlspecialchars($value);
    }
    return new FilledTemplate(
      str_replace($replace,$with,$this->_template)
    );
  } 
}

Obviously, many languages and frameworks use non-escaped string output as the default behavior. This, in my opinion, is pure, broken insanity — I can certainly see that designing a safe way of constructing HTML is harder than just following the «HTML is strings, just use string functions» approach and telling the programmer to «always escape your variables, kid» but I still find it quite irresponsible for self-proclaimed Web Languages to rely on such a primitive and dangerous paradigm. The stupid kind of irresponsible. Yes, PHP, I’m looking at you.

Article Image © Freedom II Andres — Flickr

Everyone agrees that CSS is an overly verbose stream of boiling pain, but not everyone agrees on how it should be solved. Sass is one such solution — a language extension that supports standard CSS code natively, but provides several tricks that make the maintenance of stylesheets easier, not least of which being the ability to nest selectors:

.profile {
  color: black;
  .name { font-weight: bold; }
}

A compiler turns the above into standard CSS:

.profile { color: black; }
.profile .name { font-weight: bold; }

Typical usage of Sass is to write the former by hand, run the compiler, and move the resulting CSS file to wherever the browser can see it.

I successfully set up Sass on my development machine and uploaded the generated CSS to source control. This worked correctly for a few weeks.

Then came the time when I had to set up Sass on the deployment server, because not everyone on the team had the time or the operating system to set up Sass on their computer. It burst into flames almost immediately with an arcane stack trace:

.../rubygems/defaults.rb:40:in `exist?': Insecure operation - exist? (SecurityError)
    from .../rubygems/defaults.rb:40:in `default_path'
    from .../rubygems.rb:752:in `path'
    from .../rubygems/source_index.rb:59:in `installed_spec_directories'
    from .../rubygems.rb:1051:in `source_index'
    from .../rubygems.rb:243:in `activate_dep'
    from .../rubygems.rb:236:in `activate'
    from .../rubygems.rb:1307:in `gem'
    from /usr/local/bin/sass:18

I posted an issue to the GitHub issues tracker for Sass. The answer was underwhelming :

This looks like a Rubygems issue, not a Sass issue.

And the issue was closed.

Let me set things straight here: when a Sass user cannot use Sass, it is a Sass issue. Yes, the underlying technical cause of the issue is probably somewhere inside that «Rubygems» thing I sincerely know nothing about, or maybe it lies in the Debian package for it, or maybe I’m just setting up my environment variables badly and just need a clean diagnosis to fix that up, but Sass is not working. Even if you have absolutely no personal responsibility for the issue, pointers such as «ask about this on channel X or mailing list Y» are most welcome.

Now, I don’t expect nex3 to provide me with free support. This is an open source project, and there’s nothing unusual or wrong with having no technical support on open source projects. In fact, given the size of the technology stack sitting under Sass, this was probably the best thing to do in terms of project strategy — only cater to users who can get the underlying stack working on their own.

This is not a matter of fairness or ethics, but one of incentives. I have no need to keep debugging Sass — the path of least resistance is precisely to replace Sass with an equivalent working tool. Rewriting a handful of Sass rules over to Less CSS is easier for me than tracking down bugs in the Debian packaging of «Rubygems». And the deployment server runs Less without a hitch.

I shudder to think what would have happened had I locked myself into more esoteric Sass language features.

Article image © yimmy149 — Flickr

$('input').datepicker().attr('id','the-input');

This will cause no error, and clicking on the input will correctly summon the date picking dialog, but clicking on a date in that dialog will fail with the following error:

missing instance data for this datepicker

The diagnosis is quite simple: the jQuery UI datepicker stores additional “instance data” based on the id attribute of the element, so changing the id attribute manually causes that instance data to be lost. This unexpected brittleness forced me to spend some time hacking my code so that the identifier is attributed before the datepicker is enabled, but at least this solved the problem.

Two related problems would be:

• If you have several input elements with the same identifier, and apply the datepicker on the second element, the search-by-id will return the first element and cause the same error as above.
• If you apply the hasDatepicker CSS class on an element, and then apply the datepicker plugin, it will assume that the instance data has already been initialized, and will fail.

While I agree with these, I believe there’s something far more interesting about immutable values — what you cannot do with them, and the consequences this has on feeling secure about your code.

In my current project, I use a coding style that absolutely forbids any form of side-effects with the exception of 1° a clearly delimited set of create/update functions that affect the database, 2° any side-effects that affect local variables and 3° during initialization. These are of course exceptions — while allowed, they are avoided in the vast majority of the code.

Reordering code

One simple consequence: the order in which calls are made is only constrained by things that the compiler can detect — the function that uses x is called after the function that returns x — instead of subtle «open must be called before send» errors, simply because there’s no way for «open» to affect «send» unless the former returns a value used by the latter. So, I can rearrange lines freely because the compiler will warn me when I make a mistake.

Protecting against invalid states

A corrolary of the above is that it becomes possible to design your types so that only valid states can be represented. For instance, consider a list that must contain at least one element — in a mutable context, there is no way to represent this within the type system because I could always write this:

for (i = 0 ; i < 10 ; i++)
  list.pop();

There is no way here for the pop function to indicate that it might cause a problem, because it returns nothing — the only way out is an exception, and this happens outside the type system in most languages. On the other hand, if we keep the original list unmodified and return the new list, this opens the opportunity for indicating whether additional pops are available:

type 'a one_or_more =
  | One of 'a
  | More of 'a two_or_more

and 'a two_or_more = 'a * 'a one_or_more

let pop : 'a two_or_more -> 'a one_or_more = function (_,rest) -> rest

By design, the pop function will never attempt to pop from a one-element list because it can only be called on a two_or_more value, and it returns a value which may or may not be a two_or_more — the code needs to check this, and the compiler will detect if you forget it:

let pop_n n list =
  if n = 0 then list else pop_n (n - 1) (pop list)

Here, the type of the list variable is inferred to be two_or_more (because it’s passed as an argument to pop) which means that pop list (of type one_or_more) is not a valid argument to pop_n. An appropriate solution (popping up to n, if available) takes this into account:

let pop_n n = function
  | (One _) as keep -> keep
  | more -> if n = 0 then more else pop_n (n - 1) (pop more)

Let’s summarize what happened here: I used a different representation for one-element lists and more-than-one-element lists, which let me selectively define a pop operation only for more-than-one-element lists, and this allowed the compiler to detect when the «must contain at least one element» constraint might be violated. Have fun achieving this in your non-functional language of choice! Using a different representation in this way was made possible by the fact that pop returns a new list instead of altering its argument — so the new list can be of a different type than the old one.

This is a fundamental principle in compiled functional languages: if a given state is not valid, design your types so that it may not be represented. When this is done correctly, not only does it become harder to write code that tries to create an invalid state, but when that code is written, the compiler rejects it.

Memoization

If you are certain that a given function is pure but involves computations that you would rather not repeat too often (such as database reads), you can resort to memoization: wrap the function in a piece of code that memorizes the result of every call. In my current project, this is as simple as:

let get_pic_url = Util.memoize Picture.get_url

From then on, asking ten times for the URL of a given picture will result in a single database request. Had the function involved a side-effect, memoization would have changed the program behavior: knowledge that the function is pure lets me apply memoization without any risk of creating a bug.

Retry until successful

Yet another situation is when dealing with CouchDB transactions, which involve the following process:

1. Fetch current version of the document from the database
2. Modify the document
3. Write the document back to the database
4. If a collision happened, repeat from 1.

In short, the application must repeat steps 1-2-3 until a write succeeds. Correctness of these algorithms is difficult to test because development servers seldom have enough active users to actually cause conflicts (and when conflicts do happen, they are hard to trace). By deciding that steps 1-2-3 are handled by pure functional code, I have the guarantee that the loop can be executed any number of times and yield the expected result. Right now, a transaction looks like this:

let accept id = DB.transaction id
  begin function
  | None -> (* No object in database *) DB.Keep
  | Some post ->
    if post # accepted then
      (* Post is already accepted *) DB.Keep
    else
      (* Accept post *)
      DB.Put (object
        method accepted = true
        method author   = post # author
        method text     = post # text
      end)
  end

By applying a “never use side-effects” convention, I know without even looking at the function contents that there are no side-effects inside.

These four examples are original in that they don’t try to showcase the power of immutable values and how they can be used. Instead, they simply illustrate how a program without mutable values has interesting properties that make refactoring easier, help the compiler detect many more errors, and allows the application of patterns that can only be safely applied in the absence of side-effects. Like the monks of old, I write my software without the mundane facilities of mutable values, but I derive significant benefits out of it.

Now, of course, nicollet.net is not what I would call a production server. On a real-life production server where the disk cannot be written to, sirens would start wailing, system administrators would be paged in the middle of the night (on a week-end), monitoring software would send cryptic apocalypse prophecies in their summary e-mails, and statues would cry blood. For my blog, everything continued to work in read-only mode for what appears to be half a day, until I decided to log in to the server and found out I couldn’t.

So, I rebooted it (from my hosting company’s online administration tool) and waited for it to come back up so I could investigate.

It didn’t. For half an hour. I kept refreshing the web page that shows the server status in my hosting company’s server rooms (mine is in 08H06) until I found out the page refreshed itself automatically.

And when it came back online, I stumbled upon the geek’s second worst nightmare: the private key of my server had been changed. In non-geek terms, this means that the server had been replaced by another server, possibly as a consequence of a hostile take-over by a random hacker (I’ve had this happen, and it’s very annoying) or an unexpected hard drive re-format.

After a few minutes of frantic searching, it appeared that my server had been replaced by a special rescue server that I was allowed to use to salvage whatever I could from the smoldering heap of its former self. That server, of course, had a special password that was sent to me on my e-mail address. All of you who guessed victor@nicollet.net, you get a cookie. All my attempts to have that password sent to another address failed miserably, until I finally gave up (because I have, you know, a Start-Up to start up).

It took about 12 hours for my hosting company to find out what happened, revert the changes and reboot my server. No data was lost (except for that e-mail that Steve Jobs sent me about our secret project together, which I guess I’ll never get), and everything works like a charm.

The Problem

Consider the following table, containing arbitrary text with an «alive» boolean flag:

CREATE TABLE tested (
  txt CHAR(32) NOT NULL,
  alive BOOLEAN NOT NULL,
  PRIMARY KEY(txt),
  KEY(alive)
);

INSERT INTO tested (txt,alive) VALUES
( MD5(1), FALSE ),
( MD5(2), FALSE ),
( MD5(3), FALSE ),
( MD5(4), TRUE ),
( MD5(5), TRUE );

I want to display all the lines that are marked as alive, sorted by their text field. What is the difference between these two requests?

SELECT txt FROM tested WHERE alive ORDER BY txt;

SELECT txt FROM tested WHERE alive = TRUE ORDER BY txt;

And the answer is… both queries will return the same result set! But let’s EXPLAIN them, just in case.

The first query will scan through the entire table, whereas the second query will use the index to only run through lines that are still alive. If your table consists of 99% dead elements, the first query will be a hundred times slower than the second one!

The Reason

The fundamental reason for this behavior can be found in the MySQL documentation:

These types are synonyms for TINYINT(1). A value of zero is considered false. Nonzero values are considered true:

However, the values TRUE and FALSE are merely aliases for 1 and 0, respectively, as shown here:

In short, that boolean column is not actually a boolean value, but actually an integer. This means it can contain values that are neither TRUE nor FALSE, such as 2. Such a value would be returned by the first query, but not the second, so the query optimizer is not allowed to turn the first one into the second one. And «this column evaluates to true in a boolean context» is not easily expressed as a key constraint, whereas «this column equals one» is the textbook definition of a key constraint. This explains why the second query is faster.

It also means that the second query might start behaving incorrectly if a non-TRUE, non-FALSE value finds its way into that column.

The Solution

The good news is that NOT foo is mathematically equivalent to foo = FALSE, so that the constraint can be easily rewritten by turning the «alive» property into a «dead» property. Both queries become equivalent, so the second query is a faster yet functionally identical alternative:

CREATE TABLE tested (
  txt CHAR(32) NOT NULL,
  dead BOOLEAN NOT NULL,
  PRIMARY KEY(txt),
  KEY(dead)
);

INSERT INTO tested (txt,dead) VALUES
( MD5(1), TRUE ),
( MD5(2), TRUE ),
( MD5(3), TRUE ),
( MD5(4), FALSE ),
( MD5(5), FALSE );

SELECT txt FROM tested WHERE dead = FALSE ORDER BY txt;

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

This approach to filler text is superior in many ways to most alternatives:

• It’s long, certainly longer than “Test” or “AAA”, so it can fill several lines (and test whether there is an unexpected length limit when saving the data).
• Unlike random strings of characters copy-pasted several times, it is split into words of uneven length, spaces between words do not align horizontally or vertically.
• It is readily recognizable as random text by any typesetter (or developer) worth its salt.

A typical testing strategy, when filling forms by hand, is to copy-paste one or two Lorem Ipsum paragraph to test such things as how the text area reacts, whether it is saved correctly, and so on.

Lorem Ipsum does have some limitations:

• It’s written in latin, so it fits nicely in the ASCII range of characters. As such, it does not test for Unicode support.
• It contains no quotes of any kind, so no testing of database escaping processing either.
• It contains no HTML-specific characters like < or &, so HTML character escaping is not tested either.
• For that matter, it does not contain exceedingly long words that would overflow a single line, so you cannot test for this kind of overflow either.
• Sometimes, you want to auto-linkify links and URLs.
• Sometimes, Skype turns numbers into … clickable numbers.

I need to test these things on my web applications, so I’ve developed my own version of a “Modern Lorem Ipsum“:

Lorem <a href=”javascript:document.write(”)”>ipsum</a> dòlor sit àmet, consectetur adipisicing élit, sèd do eiusmod tempor incididunt ut labore & dolore magna aliqua. <hr/>Ut enim@minim.com veniam, quis nostrud exercitation `”ullamco laboris nisi & aliquip ex æ commodo consequat. Duis aute irure dolor 01 23 45 67 89 in reprehenderit in voluptate velit esse cillum dolore `eu fugiat https://nulla.biz/pariatur. aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

Excepteur [u]sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit ‘anim id est laborum.[u] Lorem” ipsum dòlor sit àmet, consectetur adipisicing élit, sèd do eiusmod tempor http://incididunt.ut.com/labore & dolore magna aliqua. <b>Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex æ commodo consequat.</b> “Duis aute irure dolor in ‘reprehenderit in voluptate velit — esse cillum dolore eu fugiat nulla pariatur.

Feel free to copy-paste it away. WordPress certainly does seem to have a hard time with these long lines in post bodies—I wonder if it happens in comments as well

I then gathered from several sources, such as Stack Overflow, that this is a jQuery issue (or rather, feature) and not a JavaScript one.

The underlying cause is explained by Douglas Crockford:

When a DOM object contains a reference to a JavaScript object (such an event handling function), and when that JavaScript object contains a reference to that DOM object, then a cyclic structure is formed. This is not in itself a problem. At such time as there are no other references to the DOM object and the event handler, then the garbage collector (an automatic memory resource manager) will reclaim them both, allowing their space to be reallocated. The JavaScript garbage collector understands about cycles and is not confused by them. Unfortunately, IE’s DOM is not managed by JScript. It has its own memory manager that does not understand about cycles and so gets very confused. As a result, when cycles occur, memory reclamation does not occur.

A common solution to this problem is to remove the cycles when the element is removed from the DOM. Since a major source of cycles in your average jQuery program is the presence of event handlers, then removing the event handlers when an element is removed from the DOM solves the problem most of the time.

With the release of jQuery 1.4, the new documentation for .remove() makes mention of this fact:

In addition to the elements themselves, all bound events and jQuery data associated with the elements are removed.

The documentation for .html() still makes no mention of this. If you want to remove an element and keep all the goodies you bound to it, jQuery 1.4 provides you with .detach():

The .detach() method is the same as .remove(), except that .detach() keeps all jQuery data associated with the removed elements. This method is useful when removed elements are to be reinserted into the DOM at a later time.

There is an interesting duality between two categories of expectations, depending on whether they are the responsibility of the user, or of the programmer.

User errors are classic mistakes involving incorrect input, such as attempting to load a file that does not have the right format, or visiting a web site that does not exist, or entering an incorrect email address. A program is expected to, at the very least, gracefully handle these situations (because nobody likes errors) and the best programs are actively designed to reduce the possibility of error though appropriate user interface choices.

Programmer errors are the most frequent ones, but most of there are luckily caught by a compiler (or, in the case of the less lucky interpreted languages, the parser). The basic idea is that if you expect a function parameter to be an integer, and you tell your compiler, then static analysis will determine that you will receive a string argument, and the universe will collapse build will fail.

Static Analysis

Static analysis can be very smart. It can prove beyond any doubts complex properties about complex software written in obscenely low-level code (such as C with inline assembly). The problem if that working with a static analysis tool can add unusual constraints on the developers themselves: the halting problem dictates that no tool can safely predict the behavior of a program, so any given tool will either have false negatives (undetected bugs) or false positives (safe code reported as dangerous) and the general trend for static analysis tools is to avoid any false negatives at the cost of false positives.

The quality of a static analysis tool is determined by how hard it is to write code without false positives (usually done by manually coding around the blind spots of the tool).

Static analysis tools have two problems. One, they’re not available for every single language and platform out there. Some of use are still using languages with eval(), throwing Java exception-safety out the window because we find it too constraining, doing without those pesky type systems and generally making a childish fuss about those “warning” thingies. Two, static analysis tools can only check constraints that are described by the developer in some form, such as assertions, preconditions, postconditions, type annotations or some other kind of attribute added to the code.

So, if you forget to “assert” it, nobody is going to check it for you. For instance, no tool is going to warn you that you unwittingly leak a credit card number to a third party.

The Elephant Statue

In a sense, predicting user errors is the mirror activity of gathering specifications. Both force you to think about all possible situations your software will face, and decide what should happen: maybe you have to display an error, maybe you will have to tread the input in a clever but predictable way, or maybe you will have to rework your process to prevent that situation from happening.

This is akin to creating an elephant statue by starting with a block of stone and carving out everything but the elephant. Deciding what your users can do implicitly defines what your users cannot do. Depending on the situation, you may guide your design with either approach.

var button =
  $('<button>Click me</button>')
  .click(function(){alert('Clicked!')})
  .appendTo('body')

In case you were wondering, this creates a brand new button, causes it to display a “Clicked!” message box when it’s clicked, and appends it to the document you are viewing.

Click on the button that just appeared : the message box appears. Not very surprising.

Now, run the following code on the same page :

$('body').html('');
button.appendTo('body')

As expected, everything on the page, including the button, disappears. However, the button is still referenced by the button variable, so it sticks around and we can append it back to the document. And indeed, it does appear on the page.

Click on the button again. This time, no message box appears.

I honestly have no idea why.