Tag Archive for 'Psychology'

Shouldn’t Happen…

Published
by
Victor Nicollet
on January 18, 2010
in Imperative
. 0 Comments

Design and development is turning the great unknown chaos into tiny bits of controlled functionality with promises about what the result will be, and expectations about what the input should be.

There is an interesting duality between two categories of expectations, depending on whether they are the responsibility of the user, or of the programmer.

User errors are classic mistakes involving incorrect input, such as attempting to load a file that does not have the right format, or visiting a web site that does not exist, or entering an incorrect email address. A program is expected to, at the very least, gracefully handle these situations (because nobody likes errors) and the best programs are actively designed to reduce the possibility of error though appropriate user interface choices.

Programmer errors are the most frequent ones, but most of there are luckily caught by a compiler (or, in the case of the less lucky interpreted languages, the parser). The basic idea is that if you expect a function parameter to be an integer, and you tell your compiler, then static analysis will determine that you will receive a string argument, and the universe will collapse build will fail.

Static Analysis

Static analysis can be very smart. It can prove beyond any doubts complex properties about complex software written in obscenely low-level code (such as C with inline assembly). The problem if that working with a static analysis tool can add unusual constraints on the developers themselves: the halting problem dictates that no tool can safely predict the behavior of a program, so any given tool will either have false negatives (undetected bugs) or false positives (safe code reported as dangerous) and the general trend for static analysis tools is to avoid any false negatives at the cost of false positives.

The quality of a static analysis tool is determined by how hard it is to write code without false positives (usually done by manually coding around the blind spots of the tool).

Static analysis tools have two problems. One, they’re not available for every single language and platform out there. Some of use are still using languages with eval(), throwing Java exception-safety out the window because we find it too constraining, doing without those pesky type systems and generally making a childish fuss about those “warning” thingies. Two, static analysis tools can only check constraints that are described by the developer in some form, such as assertions, preconditions, postconditions, type annotations or some other kind of attribute added to the code.

So, if you forget to “assert” it, nobody is going to check it for you. For instance, no tool is going to warn you that you unwittingly leak a credit card number to a third party.

The Elephant Statue

In a sense, predicting user errors is the mirror activity of gathering specifications. Both force you to think about all possible situations your software will face, and decide what should happen: maybe you have to display an error, maybe you will have to tread the input in a clever but predictable way, or maybe you will have to rework your process to prevent that situation from happening.

This is akin to creating an elephant statue by starting with a block of stone and carving out everything but the elephant. Deciding what your users can do implicitly defines what your users cannot do. Depending on the situation, you may guide your design with either approach.

Interest(ing) rates

Published
by
Victor Nicollet
on January 6, 2010
in Functional
. 0 Comments

The most common way of investing money is putting it in a savings account. You lend a fixed amount of money to someone, and they pay interest over that money at a predetermined rate. Let’s say you lend 1,000 € at an interest rate of 3%, paid every year: at the end of the year, you would receive 30 € as payment for your lending. You would spend these on fine wine or nice clothes and wait until the next year to get another 30 €, and so on.

Savings accounts work on the basis of simple interest : what you get paid is a linear function of both time and money. Lend for half a year? 3% ÷ 2 = 1.5% Lend for two years? 3% ×2 = 6%

An important thing to bear in mind is that interest is paid at fixed intervals, for instance at the beginning of January. You don’t have to spend those 30 € : you can them on the savings account and earn simple interest on them after a year (3% of 30 € is 0.90 €).

Using this strategy, lending for two years is done at a 6.09% rate instead of 6%, because you get interest on interest. This is known as compound interest : what you get paid is an exponential function of time. Lend for two years ? (+3%)² = +6.09% Lend for three years ? (+3%)³ = +9,27%

The mathematical justification is that, with a 3% interest, your total amount of money is multiplied by 1.03 every year:

1,000 + 30 = 1,000 + 3% of 1,000 = 1,000 + 0.03 × 1,000 = 1.03 × 1,000

So, after two years, the amount is multiplied by 1.03 two times, and so on.

1,060.90 = 1.03 × 1,030 = 1.03 × 1.03 × 1,000

In short, percentages have a multiplicative effect.

And now, pop quiz : I’ve gained +5% weight over the winter holidays. What percentage of my weight do I have to lose to be back to normal ?

If you answered -5%, you missed the point. Multiplicative effect means the total change of weight would be +5% × -5% = 1.05 × 0.95 = 0.9975 = -0.25%. I would be losing too much weight !

The correct answer was 1 ÷ 1.05 = -4.76%.

Similarly, if the number of graduates of a given school increases by +10% on year one and +25% on year two, the total increase is +37.5% and not +35%.

Duality

This is where mathematicians (and computer scientists) use an interesting little concept called duality. Percentages are numbers that are easy to understand, but hard to combine. We can transform them into something that is a little bit harder to understand, but easier to combine.

The traditional way to transform multiplication into addition is to exponentiate, due to an interesting property of the exponential function:

exp(a) ×exp(b) = exp(a + b)

So, I wish to find a percentage operator (§) such that:

  • we conserve some values, 0§ = 0% and 100§ = 100%
  • applying A§, then B§, is equivalent to applying (A+B)§

Then this uniquely defines an operator which is called exponential percentage:

A§ = B%  ↔  A = 100 × log(1 + B ÷ 100) ÷ log(2)

Some common values:

0% = 0§ +100% = +100§ -100% = -∞§ 200% = 158.4§
+1% = +1.4§ +99% = +99.2§ -1% = -1.4§ -99% = -664§
+10% = +13.7§ +90% = +92.6§ -10% = -15.2§ -90% = -332§
+25% = +32.2§ +75% = +80.7§ +50% = +58.4§ -50% = -100§

percent

So, if I gained +5§ weight over the holidays, I can lose -5§ weight and be back to where I started, and if a number increases by 10§, then by 25§, it increases by 35§ overall.

And of course, a yearly interest rate of 4.2§ = 3% compounded over ten years is 42§ = 34%.

No Free Lunch

Normal percentage rules make compounding hard, but it’s reasonably easy to estimate a percentage based on a fraction. Exponential percentage rules make compounding easy, but evaluating a percentage based on real figures is harder.

In practice, compounding happens less often than evaluating, so humans use normal percentage rules. And computers are good at compounding through multiplication, so they don’t need exponentiation.

Duality does have some other uses, though. For instance, there’s the duality between two representations of complex numbers:

a + ib = r exp iθ

The cartesian (a,b) notation makes it easier to add numbers, but multiplication is harder:

a + ib + c + id = (a+c) + i(b+d)

The polar (r,θ) notation makes it easier to multiply numbers, but addition is harder:

r exp iθ × s exp iφ = (r × s) exp i(θ+φ)

For mathematically-oriented computer scientists, duality is a gold mine, because it lets one reduce a complex problem in one area to a simpler problem in another area (whether simpler means faster, as in the case of FFT, or easier to think about)..

The Law of DSLs

There’s one common duality that is fundamental in the computer world: the correspondence between data and code. In a fit of narcissism, let me sit wisely atop a tall mountain to announce Nicollet’s Law of Domain Specific Languages:

Any sufficiently complex data processing algorithm is as an interpreter for a small domain-specific language, and the data being processed is a program executed by the interpreter.

In some cases, this law only complicates things further. In many cases, however, the different angle it provides leads to many advantages, one of them being to transform a non-programming concept (such as an accounting file format) into a concept programmers are familiar with (a programming language).

A minimalist language design culture is enough to grasp several interesting concepts about executing code, which can be quite handy when processing data:

1. Compile to Bytecode

Interpreters don’t execute a string of characters. They tokenize that string, turn the tokens into an abstract syntax tree representing operations, functions and variables, then turn that syntax tree into a sequence of small, executable operations. That sequence is then fed into a virtual machine (or further compiled to machine code) to perform the actual operations.

If the input data for your algorithm is very complex, you can begin on the other side: what will the algorithm do with the data? Will it be inserting the data into a database? Constructing a data object from bits and pieces? What you are looking for is a set of atomic operations you can apply to generate the result. Implement these operations, then start working on a translation algorithm to turn the input data into such operations.

There are several common and friendly representations for such atomic bytecode:

Instruction lists are executed in order. This is your classic assembler listing, without the jumps. A typical “parse file and insert into database” algorithm would generate such an instruction list, and every instruction would be an INSERT, DELETE or UPDATE. Works best when you can read the data and generate the instructions in the right order: if you cannot get the list in the right order from the start, consider another approach.

Dependency graphs work like makefiles: you have several instruction lists floating around with relationships between them, indicating that one list has to be executed before another. A topological sort of the graph results in a single classic instruction list you can execute. A multi-file import, where some files contain data needed in other files, can be the way to go.

Nested scopes are the typical extension to instruction lists: every item in a list can be either an instruction, or another list, possibly tagged with some data. This could be a conditional (if this condition is true, execute this list), a loop (though it is best to avoid these) or a context (a “polygon” scope contains “insert vertex” operations that apply to that polygon). You can even allow variables in a let-in fashion (of which the polygon example above is just a special case) ! Note that nested scopes can be easily represented as XML.

2. Static Analysis

A side-effect of compiling to bytecode is that you get to process the entire file before you actually perform the intended operations. This makes a rollback easier if you notice that there’s an error on the last line of the file: if you make sure that no atomic operation in your target language can fail due to bad input (such as incorrect data values), then you can check your input data for correctness without doing anything to your program state.

Even better, if your compilation process is cheap (linearly traverse a file for parsing) and you have heuristics for predicting how much time and resources your individual instructions require, then you can try to accurately predict the needs of the entire process.

Static analysis also means you can optimize. If, for instance, you’re inserting data into a database and need to resolve names or keys frequently (such as “add this item to list #732″), you can easily construct a table of needed keys (that you can get in one query when the processing starts) using the dependency graph approach.You can also optimize resource allocation by using common register allocation techniques: sort your dependency graph to keep as few resources in memory as possible at any given time.

3. Caching

Try to perform most of the processing offline.

For instance, if you frequently “apply” one file to another, such as a nearly-constant “list of categories” file used to resolve the “category” key in a daily object import, you can benefit from compiling the nearly-constant file to an easily loaded, easily applied format.

You see a cached dictionary that maps keys to categories? I see a DSL that allows dictionary literals as part of the language, and a source file that contains a literal mapping keys to categories, with an interpreter that can apply constant propagation to dictionaries.

Another benefit is when applying changes to mission-critical software. Inserting lots of data into a web database can create a heavy load on the server and make the site unavailable to visitors. It might therefore be preferrable to pre-compile the imported data into requests through a process that keeps a light load on the server, then run the requests.

Besides, with proper nested scoping, you can slice an import into several transactions. This keeps the lock count low, allows spreading the transactions over time to reduce the load, and lets you resume the import process if, for some reason, it gets interrupted.

Quick Test

Published
by
Victor Nicollet
on January 4, 2010
in Functional
. 4 Comments

Here’s a very simple question:

How many times can you subtract 5 from 73, and what is left ?

Find out what your answer means by clicking here.

  • An imperative programmer answers, “you can subtract it 14 times and the remainder will be 3.”
  • A functional programmer answers, “you can subtract it as many times as you wish, and you always get 68.”

Improve

Published
by
Victor Nicollet
on January 1, 2010
in People
. 0 Comments

« Nobody’s perfect », they say. We all wish for improving ourselves in some areas. Want to lose some weight? Become a better dancer? Spend less time debugging your code? Sound smart in meetings?

There are three exceedingly simple steps to improving yourself. These are, in order:

  1. Identify what you are doing, precisely, that you would rather not do anymore.
  2. Find a way to stop doing whatever you identified in step 1.
  3. Gather enough willpower to follow the way you found in step 2 until you succeed.

Now go forth and conquer!

—

What?

Fine. I said those steps were simple, not that they were easy. They can still be helpful, though: even if you cannot seem to improve, at least you can find out which of these steps is giving you a hard time, and concentrate on that specific area.

You need to stop

There are always two ways of looking at any given improvement. You either see it as “I started doing something“, or you see it as “I stopped doing something“. Improvement is change, and change means something ends and something else begins. You start being a good dancer, you stop trampling your partner’s feet. You stop writing buggy code, you start spending less time debugging.

We bloggers love splitting people into groups: you are either a can’t-start person, or a can’t-stop person. If you want to quit smoking, but always end up lighting another one, you are acting as a can’t-stop person. If you want to exercise daily, but always find a good excuse not to run your laps, you are acting as a can’t-start person. And you had better find out which group you are in.

Because if you are a can’t start person, and you think of potential improvements in terms of starting doing things, trouble is coming your way.

Step one is all about looking at your problem from the other side. To start doing things right, you need to stop doing them wrong. As you stop making mistakes, what is left usually counts as improvement.

food_anotherslice

Ask and accept

Some humans are blessed with the ability to see, in a clear and unmistakable fashion, what went wrong about something. Enlightenment comes in a slap-your-forehead moment where the root cause of your problems is discovered.

Most of us experience trouble as a hazy and painful feeling that seems to come from everywhere at once, with no clear cause and no clear idea of how things ended up the way they did. If you cannot really understand what you are doing wrong or why you end up in trouble, consider asking someone else.Whether an expert opinion, a different angle on the situation, or the act of putting your issues in words and sentences, you will get something out of it.

This reminds me of Bob. When I was still a young intern, without much experience in the ways of men, an older developer joined my team. Despite his reasonable technical skills, Bob had a conflictual relationship with our manager, and this had a negative impact on team morale. I had been told our manager disliked it when people were late for work, even though he did not insist on the matter too much, and my experience with him confirmed that information. Since Bob was at least fifteen minutes late every morning, I decided to share that information after a quite gruesome clash, thinking it would help :

“You know, he gets angry when people are late. If you can’t come in earlier, you really should tell him about it.”

“What do you mean? I’m not late.”

“You came in at 9:25 am this morning.”

“No way, I came in at 9:00 am!“

I suspected that Bob, out of pride, would deny being late in the morning, but that he would mull over the notion and come in on time the next day. I was wrong, and he was thirty minutes late the next morning.

It is very important to carefully examine any criticism before rejecting it. When someone you know takes the time to explain that you’re doing something wrong, the least you could do is wonder if they are right, or why they thought they were right. Criticism is free advice, carefully selected to apply to whatever you do that annoys others most.

Even if Bob really was on time every morning, my advice was still worth considering because it indicated that some people thought he was late, which is quite important in a corporate environment.

Mental triggers

Once you have identified what you need to stop doing, you can move on to step two: find a way to prevent it from happening. Since you cannot keep thinking about it all the time (and even if you did, it would reduce the efficiency of whatever else you were doing), you need to find a way to remember about your decision when it actually matters.

If you experience a feeling when you perform the unwanted action, you happen to be in luck, because it is far easier to associate your decision to stop with that feeling.

While I have always been comfortable in one-on-one conversations, I used to have a lot of trouble speaking up in groups, because I tend to take a short time to think before I speak, and this means someone else in the group is going to start speaking before I can gather my thoughts. If there were pauses in the conversation, I could certainly pass off as the wise experienced guy in the corner who only speaks up when nobody knows what to do anymore, but most of the time I looked like the shy silent guy in the corner who has nothing to contribute to the discussion.

I noticed that most people who spoke up in group discussions said things that were mostly irrelevant and generally only served as an anchor to remind others of what the position and motives of that person were. A precious few people, however, seemed to always grab the attention of others and say something interesting and relevant. Among these was Jamie, a nice lady in her mid-thirties. After observing her for a while, I noticed a pattern in her way of speaking up: she would repeat the last sentence that was said.

“…and we should test if the government servers can handle the load.”

“Test if they can handle the load, yes. I just received a report about…“

By repeating the sentence, she was able to start speaking faster than anyone else, and she used that time to think about what she was about to say next. Not to mention that repeating someone’s sentence implies some level of agreement, which is always good to have in a meeting.

So I started training. Whenever I missed my turn in a conversation, I felt frustrated, which reminded me of my decision to use Jamie’s technique. So, on the next try during the same conversation, I did not think silently about what I had to say and instead repeated the last part of what the speaker said. After a while, I did not have to think consciously about it anymore.

Without a feeling to anchor your reminder to, you will have to find something else.

One trick is to artificially increase the time before you can take the unwanted action. For instance, I live in a flat in Paris and my fridge is a ten-second walk away from my desk, so it’s easy for me to stand up and fetch a quick snack without thinking. I once spent my holidays with a friend who lives in a large house in the countryside, where the fridge was two floors below my appointed room and it took a full minute of navigating slippery stairs and cold, narrow corridors before I could get to said snack, which means I never got to eat without thinking. The longer it takes you to start an activity, the greater the chances that you notice “Hey, I’m about to do this, and I decided I wouldn’t do it anymore!” and desist from doing it.

If the activity is continuous (such as “not washing the dishes“) you can try to make the consequences of that activity as obvious as possible. If the house (or source code) is a mess, then adding small amounts of messy laisser-aller are unnoticeable. If the house (or source code) is cleanly arranged and organized, any amount of mess is going to stand out and be very obvious. It is easier to keep a house or project clean, than it is to clean it up later on.

The will to go on

Finding enough willpower to enforce your decisions can be hard. A good strategy is to know your weaknesses and exploit them. For example, if you hate mediocrity and often think that “I’m worth better than that“, then thinking of unwanted activities as shameful and worthless can give you that little boost you need.

You can even shame yourself into respecting your decisions by telling other people about it. Ask other people to look over your source code daily, even if it’s not an actual code review, and you will often be too ashamed to leave undocumented methods and badly named variables around.

<div xmlns:cc=”http://creativecommons.org/ns#” about=”http://www.flickr.com/photos/avlxyz/2684089255/in/set-72157606283805523/”><a rel=”cc:attributionURL” href=”http://www.flickr.com/photos/avlxyz/”>http://www.flickr.com/photos/avlxyz/</a> / <a rel=”license” href=”http://creativecommons.org/licenses/by-sa/2.0/”>CC BY-SA 2.0</a></div>

Smart Spamming

Published
by
Victor Nicollet
on October 8, 2009
in Random Randomness
. 0 Comments

I found an interesting comment on my website today, for the article on last-minute-skinning of a page in HTML from some Javascript. It looks pretty sane:

CT — October 5, 2009 at 22:15

Interesting stuff. I don’t relish the idea of taking the vile HTML our designers produce and creating the skin files. Nice proof of concept though – I’ll have to keep an eye out for an excuse to use it ; )

This comment, while completely adequate and relevant to the article, is spam. How do I know? First, the provided website is a classic credit-rating-improvement web portal. But should I prevent people who work in the credit spam industry from posting relevant comments on my articles? Well, there are other comments on that article, too, such as:

Tom Milsom — September 8, 2009 at 11:41

Interesting stuff. I don’t relish the idea of taking the vile HTML our designers produce and creating the skin files. Nice proof of concept though – I’ll have to keep an eye out for an excuse to use it ; )

So, it looks like the spam-bot found an earlier comment on the article, copied it verbatim, and posted it with a different link. This would ensure that, if the spam domain is fresh enough not to register as such, the Akismet spam detector would let the comment go through unscathed based on its content alone. And as a human, if I did not pay attention to the author’s website while reviewing comments, I would let it go through as well because the comment would look sane. I don’t remember comments from one month ago, and I guess many people don’t.

Everyone enjoys advertising if they are looking for, or otherwise interested in, the product being advertised. I discovered Cushy CMS because it ran an ad on The Daily WTF, and I am quite happy with the discovery because I was looking for such a product. And nobody enjoys advertising for products they don’t need—I don’t give a cheese about US credit ratings. I have limited space on my screen that I’d rather not fill up with advertising about things I do not need, and my time is even more precious than that.

This spam comment blurs the line between spam comments that are irrelevant to the discussion and point to websites irrelevant to the readers, and ham comments that are relevant to the discussion and point to websites that are relevant to the readers (by virtue of usually being run by the author of the comment and thus sharing at least some elements).

Suppose that tommorrow, someone posts an original and interesting comment on one of my articles, yet links it to a credit rating website. Should I accept the comment as such, block it, or publish it without the link?

One of the main reasons why people comment on the blogs of other people is to improve their visibility on the internet. If I post a comment on a well-known blog, hundreds and thousands of people will browse over that comment, a small percentage of these will find my writing worthy enough to follow the link and end up on my blog, and an even smaller percentage will become regulars, posting comments and subscribing to my feeds. Which is good, of course, because the more comments I get on my blog, the more interesting it becomes.

This means that commenting is often quite similar to advertising one’s own blog or website. People allow commercial advertising on their blogs (ad banners and such) to get money in return, and they allow personal blog/website advertising on their blogs to get comments in return. So, I guess if an irrelevant website was linked to by a genuinely interesting comment, I would publish that comment (of course, restrictions do apply: I would not allow all websites, just like I would not allow all ad banners).

I like the blogs with good comment advertising—where I can browse the comments and find links to interesting websites.

Filling in the Holes

Published
by
Victor Nicollet
on September 14, 2009
in Strategy
. 0 Comments

As the technical lead on a software project, I get to interact on a daily basis with stakeholders that are technologically impaired. They think in high-level, end user terms like « I need a comment system » or « send the user an e-mail notification » and they expect things to happen without having to delve into the boring techno-babblish details of how it’s done. The rationale is that deciding what happens is a stakeholder job and deciding how it happens is a developer job.

Of course, no matter how hard you try to separate the two, developers are sometimes going to decide what happens, because no stakeholder can spare the time needed to walk the development team through the bloody details of every single feature. Given the productivity gains from recent advances in development tools and the social skills of the average programmer, I think it’s fair to say that an in-depth description of a feature takes about as long as the implementation of that same feature.

This is why all projects follow the same steps regardless of the methodology used:

  1. A stakeholder makes some general statement about a feature, such as user comments being available on certain items.
  2. The development team writes what they feel is the best implementation of that feature in the context of that project, filling in the missing details as they go.
  3. The stakeholder sees the results and points out what details did not match their mental model of the requested features.

This introduces several dangers: there’s the budget issue when missing details turn out to be costlier than originally envisioned, and there’s mismatch issue that makes the customer unhappy. A good project manager should strive to reduce these. How?

Gathering more requirements is a classic strategy in waterfall models. The basic reasoning is that the more details you manage to gather about the product to be implemented, the lower the chances of a surprise requirement blowing your budget away and the higher the chances of meeting the customer’s expectations. The downside is that this step takes time, which in turn uses up the budget and delays the release.

Also be careful when deciding on a budget after having gathered all the requirements. Everyone changes their mind sooner or later, and any requirement change, no matter how small, should prompt a critical analysis of the budget: adding « just one link » is indeed a tiny change, but the involved overhead (change the internal documentation, determine the impact on other feature, tell the developer, write the code, test the changes) adds up much faster than you think.

Fast iterations involving the stakeholders is the Scrum approach, shared with many agile methodologies. It deals with budget issues by developing the simplest possible implementation that matches the requirements. It also provides the customer with feedback on the estimated implementation time through poker planning before every iteration, so that requirements can be changed on the fly if sacrificing a small feature can significantly reduce costs.

Short-iteration agile projects build customer dissatisfaction into the development process: if you don’t like the existing implementation, you can ask for a change and get it done on the next iteration. It also lets the developers decide based on technical considerations (what’s easier to implement) as opposed to high-level decisions (how the stakeholder wants a feature to behave), which lets them work faster and do what they are skilled at.

The downside to these approaches is that stakeholder involvement should not be taken for granted, and even when it is, it’s not uncommon for customers to have dissenting opinions among themselves. Also, an agile process does not help if there’s a fixed deadline and a fixed set of 1.0 features, and the customer expects these to be done in time.

Having developers with common sense helps a lot—all the people working on the project should be able to tell ahead of time if a given solution is going to be unacceptable, and dismiss it if they see one. This avoids implementing useless solutions, or forwarding an useless solution to a customer for validation.

The obvious corollary is that a developer should write bug-free code without having to be told to write bug-free code. Commits that contain segmentation faults, access violations, unhandled exceptions, blank pages, broken links or performance bottlenecks should be investigated into, to determine why a mistake was made and what steps should be taken to avoid repeating it.

What other techniques have you come up with for reducing communication-related risks?

AJAX is Hard

Published
by
Victor Nicollet
on August 31, 2009
in Imperative
. 0 Comments

Seen from the outside, AJAX has become an easy technology:

$('#container').load('http://domain/path/to/page');

Even if you’re doing smarter things, like updating server-side values with asynchronous POST requests, it’s still easy:

$.post(
  'http://domain/path/to/action',
  { user_id       : $('#user').val(),
    new_user_name : $('#name').val() }
);

And, of course, it’s also easy to make mistakes in AJAX.

Not taking errors into account

In an ideal world, the AJAX request is sent to the server, completed successfully, and the response is propagated back and applied.

In the real world, the AJAX request might never reach the server because the network cable was pulled, or it could carry stale data that cannot be processed, or the user session might have expired, or something else altogether.

This “request cannot be completed successfully” issue has been solved for years in the traditional HTTP world by both servers and browsers: when you try to get to a page and that page can’t be reached, you will either get an error message from your browser or be redirected to another page by the server.

In the AJAX world, a failed request times out silently without anything happening. You have to actually implement that small “Your session has expired, click here to log in again” message box yourself, just like so many other websites did. And, of course, you need to take into account into all of your workflows that the user may be logged out of their session at any point.

Don’t forget to include cable-plugging as part of your testing protocols!

Forgetting to refresh parts

When you post some modifications to the server asynchronously, you need to refresh some parts according to the new state of the server. Which parts do you refresh?

While the answer might seen easy in every single specific case (I’m updating this list/object/grid, so I’ll just refresh it), the general answer is not so simple: your server-side modification might have an impact on other parts of your system.

Consider a typical Facebook-like interface: you have a menu with an inbox, and to the right of that inbox there’s the number of unread messages. On the inbox page, you have a list of messages with a little cross on each message that deletes it through AJAX. The naive thing to do is have that cross update the list of messages, but then deleting an unread message wouldn’t update the menu.

Inevitably, a developer working on an AJAX feature will forget to take into account that some other part of the page that needs to be updated. Or a developer will add some information to every page and forget that some pages need to update that information.

Repeating yourself

Javascript does not benefit from the same clean separation of features into classes, files, packages and namespaces. Also, IDE quality is lacking when compared to other languages. This makes it hard to refactor JS code when duplicate functionality starts to appear.

Let’s consider the asynchronous post situation. In order for that code to work, you need to have fields with identifiers ‘user’ and ‘name’ and some element to initiate the post through an event. This is not encapsulated: if another page needs similar “post user name” functionality, the code will have to be rewritten. In fact, when the code is that small, it’s actually faster to rewrite it than it is to find and call an existing function (not to mention writing that function in the first place).

No refactoring means the code repeats itself. Having two user-name-change pieces of javascript on two different pages means twice as much work to do when you eventually change how that part really works.

Allowing complex behavior to be written in two or three lines is no excuse for letting your code get out of hand: stand firm by the “once, twice, refactor” motto and do not hesitate to turn a three-liner into a ten-line reusable function with appropriate documentation.

Information Flow

Published
by
Victor Nicollet
on August 17, 2009
in Strategy
. 0 Comments

The real world is a complex place. When writing software that has to interact with the real world, there are literally thousands of concepts you have to master and tens of thousands of details you have to be aware of, or you will paint yourself into a corner where your software clashes with reality. And reality always wins.

Understanding concepts and details is a fundamental part of a project’s time budget, whether they come from the project requirements, real-world constraints, third party code or teammates. Every time information goes around in a project, it uses up valuable time, and to keep the time budget tight it becomes necessary to decide what information should be allowed to go around, and where.

Working on concurrent systems is an enlightening experience, because of the many similarities between an array of computers and a team of information workers. Computers arrays have latency issues when one thread depends on another thread to be done…

“When do you think your settings import module will be done? I’m stuck on the payment API until I can load those settings!„

…they have bandwidth issues and manipulating some data yourself is usually faster than sending the data to another part of the cluster for treatment…

“The User object? Well, it’s a bit of a weird design, but it’s rather clever. I’ll draw you a quick UML sketch on the blackboard so you can see what the five helper classes do.„

…they have to avoid data loss if a computer or network is down…

“I have no idea how this stored procedure works, you should ask Tim, he’s the one who wrote it. He’s in southern France right now but I think he’ll be back next month.„

… and they have to handle a directory of parts and a garbage collector for data…

“Wait, nobody’s written the comment moderation back-office! Who was in charge of doing it? Who wrote the comments front-end anyway?„

There are algorithms, strategies and techniques for handling and optimizing those things. Many of these can be adapted to humans, with the added benefit that, humans being smart, they can understand the point of those algorithms and compensate for minor flaws if the plan isn’t perfect.

You’re not a person

Published
by
Victor Nicollet
on August 14, 2009
in Imperative
. 0 Comments

WEEK 1

In this application, every person belongs to exactly one team.

WEEK 4

We need to manage external contractors. We could use the “person” object.

WEEK 5

Hey, we need to assign a team to every person. Let’s create an “external” team.

WEEK 127

Did you see that newspaper article about our company? They say we have an average of 30 people on every team. Do we even have 30-people teams?

Names are short. They can only convey a very limited amount of information. Even worse, that information tends to be different from its meaning in standard English: by declaring in week one that every person belongs to a team, the project designers separated the Application::Person (always in a team) from the English::Person (might be in zero, one or more teams). By week four, this separation vanished from the minds of most of the team. A developer noticed that “English::Contractor is-a English::Person” and mistakenly translated it to “Application::Contractor is-a Application::Person“.

This was the first mistake. Why didn’t he notice?

A positive property is what you can do with a thing.With the Person object, you can store a name, login, password and phone number!This is exactly we you needed! Those positive properties you need that the object doesn’t provide, you can always add them through inheritance or composition, and that’s still less work than implementing everything or having to refactor the code. A negative property is what you cannot do with an object no matter how hard you try. With the Person object, you cannot remain on your own without a team! But our brains are biased to look for positive properties first, and passively ignore negative properties until it’s too late. Positive properties are about the solution solving the problem. Negative properties are about the solution not being applicable.

The second mistake was, by far, the worst. So they finally noticed that negative property that blasted all their model away. And they went on with it, patching the issue by altering the meaning of Application::Team. It originally a project team within the company, it then represented a named group of people that could be a project team or the group of external contractors. This is refactoring: no matter how you look at it, you change the behavior of an object and let it propagate throughout the project, so you better be careful about where it propagates! In this case, they weren’t careful about propagating the change of meaning to the documentation and user interaction part of the project, who mistakenly kept the old meaning of Application::Team. This led to a naive PR team issuing a statement that included the “external” group as if it were a project team.

It’s always helpful to have an anal-retentive person in a group, preferably in a position of authority that lets them veto such changes, and who is vigilant enough to spot that “external” team early on in the design.

The real mistake was allowing a negative property to slip into the design. Negative properties hinder reuse, by definition. Sure, allowing a person to belong to zero-one-many teams is hard on every piece of code that must work on teams, because the writers have to remember to check whether the person has a team in the first place. But it has to be done. Doing it may even bring to light some issues in the original requirements (”So what happens when a person changes teams between the moment team bonuses are computed and the moment they are paid out?”) that would become annoying later on.

Best Practices

Published
by
Victor Nicollet
on August 10, 2009
in Imperative
. 1 Comment

There are hundreds of things that can go wrong even in the simplest situations. I’ve already explained why the real value of a domain expert is precisely to identify in advance everything that could go wrong with a project, so that it can be avoided.

Consider a comment form on a website. Nothing too fancy: the user fills in the “Name”, “Website (optional)” and “Comment” areas on a form, clicks the “Submit” button, and the page reloads with the comment on the page. No login required, no AJAX, no special effects. There are many things that can go wrong with this setup, and will go wrong if left in the hands of an inexperienced developer. They can be inconvenient, annoying or outright dangerous.

For example,

  • Double-posting. When the submit button is clicked, the form sends a request to the server with the comment to be added. The server responds with the new list of comments. The user clicks the “refresh” button while on that page, or navigates to another page and presses the “back” button. This cause the browser to send the request again, so the comment appears twice in the comment list. If using POST, this is slightly less dangerous : the user might get an annoying “Submit again?” window instead of double-posting.
  • SQL Injection. It is highly probable that the comments will be stored in an SQL-accessible database. If the code constructing the SQL query is not properly written, an appropriately chosen value for the comment fields can result in nasty things happening to the database.
  • Cross-Site Request Forgery. Suppose that posting the form creates a GET request like: 
    http://yourdomain/postcomment?name={name}&text={text}

    Knowing this, I can include an image tag in a forum, with a source attribute that matches the posting of a spam comment on your website. Every visitor of that forum page will send that request automatically (browsers auto-fetch images by default) and spam your comment list.

  • Script Injection. The text entered by users must be displayed back to the visitors. If that text is not escaped before being output, an malicious attacker can submit a comment containing a dangerous script like: 
    document.location = "http://www.youtube.com/watch?v=f2b1D5w82yU";
  • Encoding Issues. What happens if the page is encoded in UTF-8 but I send you ISO-8859-1 text? Conversely, what happens if the page is encoded in ISO-8859-1 and I copy-paste my comment from Microsoft Word? For that matter, what is the encoding of the database? What is the encoding of your string literals?
  • No Validation. User forgets to enter a name or a comment. No server-side check is made to determine whether the posted comment is valid and you get a mix of ugly empty comments and/or server error messages.
  • Lossy Validation. You have to prevent people from posting with no name or no comment body. This means errors will be displayed on the page and, if the detection of such errors happens on the server after the initial post, it’s easy to forget displaying back the text the user entered in the first place. “Sorry, you forgot to enter a name so I’ve thrown your ten-line comment away” [#]
  • Does not work in Internet Explorer. There are many possible causes for it, such as respecting W3C specifications.
  • Legal Issues. If a malicious commenter uses your page as a soapbox for illegal activities, some countries will hold you responsible. For instance, in France, you can be condemned if anonymous posters engage in holocaust denial on your website.

That’s nine, just thinking about the obvious problems that would happen if following the simplest approach to this, and I have seen many of them happen in three situations: novice programmers (such as interns), freelancers and low-wage programmers. The worst offender is by far the code written in naive PHP, which has the peculiarity of “the simplest thing” being almost always “the incorrect thing” as well.

Still, if you can’t let an intern write a simple user comments page, what are you going to let interns do?

All of the above issues are easy to correct once you know about them. Always send data as POST, check the referrer, convert everything to UTF-8, validate your data, use prepared statements instead of inline SQL, respond with a 303 redirect to a GET page, include the posted data and any errors in the session and display them back in the form if present, take all your dynamic generation text through an an HTML escaping function, add “type=submit” to buttons, and add a quick moderation tool to hide unwanted messages quickly.

Knowing about the issues and acting to prevent them is the hard part, which is why every project should have at least one experienced developer who knows about the errors. Or be using a framework that prevents such errors from happening in the first place (then again, if the documentation for Zend_Form has an “user refreshes page, double-posts by mistake” error, who can we trust?)

Although it has been taken over by marketing folks, there are still good thinks to be said about “best practices”. The basic idea is to have a set of practices available for the less experienced developers to follow. Such practices are usually very simple to understand and follow (never display data in a POST controller, never change the model significantly in a GET controller), reasonably simple to verify automatically (assert that no output happened as part of a POST controller response) and have the immediate effect of preventing a classic mistake (no re-post on a page refresh).

I’m a big proponent of enforcing good code through practices first, and then code-based contraptions if developers insist on ignoring them. The problem with going for the contraptions first is you have to explain how to use the contraptions anyway, and people will be tempted to move around the contraptions and still write bad code.

If your code is reviewed by a compiler or an automatic code analysis tool, you can learn how to game the system. This results in code that does not trigger the alarms, while still being bad. Compare with having your code reviewed by a live person, who is experienced and anal-retentive about respecting practices and makes it horribly clear that if you don’t follow them, you will be forced to follow them, on your free time before you can commit your code. Such reviews leave no room for wiggling, and as long as the judgment of the reviewer is fair, will actually motivate the team to respect the standards.

[#] Viadeo actually did even worse things to me (”Sorry, I forgot to tell you that you were only allowed 255 characters in this box, so I’ve deleted everything for you so you can try again. Oh, and don’t try the back button of your browser, I have also deleted your input on the previous page.“) so I suspect it has been written by Java rookies with close oversight by non-technical management.