It’s is Better™ to have many short files than a few long ones. One reason is incremental compiling with ocamlbuild : that the smaller your files are, the smaller the percentage of code to be compiled when you make a small change. Another reason is that files provide a natural delineation of code that makes it slightly easier to reason about.

The very process of splitting a large file into smaller files is also an excellent way to clean up the code. Every split is an opportunity to move some code to a more generic location — why have a CMember_importParser module when all of its functionality could fit into an OzCsv plugin module ? Even when no such generic solution exists, cutting through the jungle that a 2000-line module contains helps clean up dependencies, identify shared functionality and imagine better ways to design code.

Still, when cutting up code this way, the problem of encapsulation remains. If code that relates to pictures (an upload module, a transform module, a download module, an access rights module) is split across several files, it is desirable to let each file access functions and values from other values that would not otherwise be shown to modules not related to picture processing. For instance, a get_download_link function should be available throughout all picture-related modules, but the rest of the application should use the get_download_link_for_user function that checks whether the user is allowed to download the file.

In order to achieve several nested levels of encapsulation required to work with modules this way, I have come up with a convention :

• A module name (and thus, a file name) is composed of segments written in camelCase and separated by underscores. For instance, CEntity_view_grid is a module name containing segments CEntity, view and grid.
• Modules with only one segment are public. Any other module may include, open or otherwise reference them with no limitations beyond what the module signature says. So, CEntity may access MGroup freely.
• Modules with N > 1 segments are private. They may only be accessed by modules which share the first N-1 segments. So, CEntity_view is available to modules CEntity and CEntity_edit but not CPicture.
• A module with N segments may export any module with N+1 segments it can access, possibly under a more restrictive signature. For instance, CEntity_view is available to all other modules as CEntity.View.

To make these rules easier to respect, private module dependencies are made explicit by adding a list of module aliases at the top of each file. The top of my cEntity_view.ml file starts with :

module Sidebar     = CEntity_sidebar
module Unavailable = CEntity_unavailable
module Edit        = CEntity_edit
module Info        = CEntity_view_info
module Directory   = CEntity_view_directory
module Grid        = CEntity_view_grid
module Wall        = CEntity_view_wall

It is forbidden to use a private module without going through such an alias, and it is forbidden to define such an alias anywhere except at the top of the file. This makes it extremely easy to determine whether private access rules are respected.

The rule of thumb for splitting files (in my particular coding style) is :

• Code for separate layers (model, view, controller…) go into separate public modules.
• For complex code (such as complex rules in model or controller code), consider splitting files larger than 200 lines.
• For simple code (such as HTML template or JSON serialization definitions), there is no splitting limit except for factoring out common behavior.

Lakshen Perera provides a list of basic patterns for everyday programming, illustrated in Javascript and Ruby. I thought it would be interesting to provide an OCaml illustration as well, and perhaps a handful of additional patterns as well.

Verify object’s availability before calling its methods or properties

In many languages, there is a possibility for objects to be missing — whether this is represented as NULL, null, nil or None. Regardless of the language, it is important to keep in mind at all times whether a given value is optional or not. If it is not optional, then you may assert so using a type declaration if your language supports it :

method name  : string (* not optional *)
method email : string option (* optional *)

In other languages, a runtime assert will do, for instance in PHP :

assert (isset($this->name))

OCaml will not require you to explicitly declare all values as optional or mandatory. Instead, it will deduce that information from the way you use each value. For instance, since json_of_string expects a non-optional string argument, you can simply write :

let parsed_content = json_of_string json (* json is not optional *)

If json is an optional string, this will create a compilation error unless you explicitly define what should happen when the string is missing. The most common approach is to decide that the result should be missing too :

let parsed_content = match json with
  | None   -> None
  | Some s -> Some (json_of_string s) (* by definition, s is not optional *)

If using the Batteries library (which, by the way, you should), you can express this more easily :

let parsed_content = BatOption.map json_of_string json

Set a default value with assignments

This advice is almost identical to the previous one : if you need to construct a non-optional value, but only have access to an optional value, you will have to provide a default value. Using standard OCaml :

let role = match person.role with
  | None      -> `Guest
  | Some role -> role

Using Batteries, there is an almost equivalent version :

let role = BatOption.default `Guest person.role

This version is almost equivalent, because it will evaluate the default value even if it is not required. Let’s assume that the default value is itself provided by a complex computation, such as a database access :

let role = match person.role with
  | None      -> readfrom database (* only executed if person.role is missing *)
  | Some role -> role 

let role = BatOption.default
  (readfrom database) (* Always executed, even if unnecessary *)
  role

The Batteries library provides an alternate function for that :

let role = BatOption.map_default
  readfrom database (* only executed if person.role is missing *)
  role

Checking whether a variable equals to any of the given values

If the values have a legitimate reason to be strings, integers or other types with unlimited numbers of values, then « is in list » predicates are the preferred choice :

if List.mem current_day ["Monday";"Wednesday";"Friday"] then
  (* do something *) 

Days are a quite bad example, because these are better represented as a variant (which then type-checks whether you have written Mornday instead of Monday). If you have already defined the type of weekdays, then :

if List.mem (current_day : weekday) [`Monday;`Wednesday;`Friday] then
  (* do something *)

If this is a one-shot check, or if you would rather not define a weekday type yet, you should instead go for an exhaustive pattern-matching :

 match current_day with
  | `Monday | `Wednesday | `Friday ->
    (* do something *)
  | `Tuesday | `Thursday | `Saturday | `Sunday -> ()

Extract complex or repeated logic into functions

This is a fairly fundamental concept — but it consists in two distinct parts. There is a separation and naming part (you pull out a piece of code and give it a name, which helps understand what it does and how it relates to the rest of the program) and there is an extraction and reuse part (the piece of code is pulled out into a more globally accessible location and parametrized, so that it may be used in other places).

With the above example, simple separation-and-naming would be :

let is_discount_day = match current_day with
 | `Monday | `Wednesday | `Friday -> current_date > 20
 | `Tuesday | `Thursday | `Saturday | `Sunday -> false
in

if is_discount_day then 
  (* do something *)

The variable is defined in the same scope it is use in, and it assumes that current_day and current_date values have been defined previously in that scope. Extraction-and-reuse would go further :

type weekday =
  [ `Monday | `Tuesday | `Wednesday
  | `Thursday | `Friday | `Saturday | `Sunday ]

let is_discount_day (day:weekday) date =
  List.mem day [`Monday;`Wednesday;`Friday] && date > 20

...

  if is_discount_day current_day current_date then 
    (* do something *)

Now, is-discount-day is a global function available from everywhere in the code, and it uses the provided parameters to determine whether this is indeed a discount day.

Memoize the results of repeated function calls

OCaml has several ways to perform memoization. One of them is lazy evaluation :

val discount_day = lazy (is_discount_day current_day current_date)
method discount_day = Lazy.force discount_day

The lazy expression will only be evaluated the first time the Lazy.force function is called on it.

Note that if the current day or current date can change, then the memoization actually breaks things !

Memoization is also helpful when dealing with a function that requires arguments, in which case a different result will be provided for each argument set. A common solution is to use a hash table to store these :

let fibonacci =
  let memo = Hashtbl.create 100 in
  let rec fib n =
    try Hashtbl.find memo n with Not_found ->
      let result = fib (n-1) + fib (n-2) in
      Hashtbl.add memo n result ; result
  in fib

This works fine for short-lived functions — don’t do this for global functions that might stick around for a long time, because the memoization hash table will grow and its contents will never be garbage-collected. If you really have to, use a weak hash table, such as Batteries’ BatInnerWeaktbl, so that the garbage collector may reclaim the memoized values when it runs out of memory.

Also don’ t overdo memoization — it only works when arguments are reliably passed more than once and the time to compute the value is significantly larger than the time to retrieve and store it and it is worth the memory usage and the function has no side-effects.

Use the seven list manipulation primitives

Almost any processing on collections of items can be expressed in terms of seven fundamental patterns. Recognizing those patterns can help improve the clarity of both the code and the underlying algorithm.

1. Map transforms a list into another, item by item, in linear time. Use a map operation when all you need is a one-to-one transformation. The line below extracts three recipes from the database using their identifier.

let recipes = List.map from_database [ "omelet" ; "cheeseburger" ; "risotto" ]

2. Reduce transforms a list of values into a single value by repeatedly applying a function that combines together two values into one. The typical example is a fold, which uses a function to combine each list element, in turn, with an accumulator. It can be used to extract the sum of values in a list, for example :

let total = List.fold_left (+) 0 [ 5 ; 6 ; 3 ; 8 ; 9 ; 0 ; 7 ; 6 ]

This transform allows a preliminary map step which transform the values inside the list into values that can be combined. For instance, to find the age of the oldest person in a list of people :

let oldest = List.fold_left (fun acc person -> max age person.age) 0 people

3. Extract works like a map, but the transformation function returns zero, one or more results for each call. All the results are included in the final list. The most elementary implementation is literally to have a map (that transforms a list into a list of lists) followed by a concatenation (that transforms a list of lists into a list). For instance, to get all the ingredients involved in a list of recipes :

let ingredients =
  List.concat (List.map (fun recipe -> recipe.ingredients) recipes)

4. Filter is a subset of Extract where the transform may not return more than one result — but it may still return none, so its result is simply an optional type. For instance, to extract the list of all recipes that have a wine recommandation along with their recommended wine :

let wines = BatList.filter_map
  (fun recipe -> BatOption.map (fun wine -> recipe,wine) recipe.wine) recipes

The OCaml language also provides a standard List.filter function which keeps values for which a property is true. For instance, to get the list of recipes that have a wine recommendation :

let have_wines = List.filter (fun recipe -> recipe.wine <> None) recipes

This approach is weaker — the wines in the resulting list are still treated as optional by the type system, so you will need bogus pattern matching for a case that never happens (no wine) to extract the actual wine values. A filter_map lets you encode the filter property in the type of the result, which makes using the filtered list easier.

5. Sort unsurprisingly sorts the list. The canonical sort — using the canonical order relationship — is a theoretical curiosity, and in practice most sorts use a projection function p such that A < B iff p(A) < p(B). This is best illustrated in SQL, in the form of the ORDER BY <projection> statement. Two useful helper functions :

let project compare p a b = compare (p a) (p b)
let descending compare a b = compare b a

For instance, to sort the list of recipes based on how long each of them takes :

let by_duration =
  List.sort (project compare (fun recipe -> recipe.duration)) recipes

6. Group works like Sort, but further regroups « equal » items together by returning a list of lists. It works using a comparison function that is usually based on a projection function. For instance, to get three lists containing one-star, two-star and three-star recipes :

let by_stars =
  BatList.group (project compare (fun recipe -> recipe.stars)) recipes

There is a special case when the projection function returns booleans, which is known as a partition. For instance, to extract recipes that are desserts and recipes that are not :

let desserts, non_desserts =
  BatList.partition (fun recipe -> recipe.is_dessert) recipes

7. Search extracts one element from the list (if possible) based on a certain condition or property. Elementary searches are « first element » and « last element. » More complex searches : finding a value by key in a list of key-value pairs using List.assoc or using a predicate using List.find. The heavy-duty search tool is BatList.find_map, used below to find a recipe that is recommended by at least one person, and the recommending person :

let recipe, recommender = BatList.find_map
  (fun recipe -> BatOption.map (fun person -> recipe, person) recipe.recommended)
  recipes

These seven patterns can be used in conjunction to perform almost any algorithm on collections, sequences or lists. For a more complex example, assume we need the ten ingredients that appear the most often in desserts. We would filter recipes by desserts, extract their ingredients, group them by name, sort the sub-lists by list length and take the name of the first element of the first ten sub-lists :

open BatPervasives (* for operator |> *)
open BatList

let ten_best_ingredients recipes = recipes
  |> filter (fun r -> r.dessert)
  |> map (fun r -> r.ingredients) |> concat
  |> group (project compare (fun i -> i.name))
  |> sort (descending (project compare length))
  |> take 10
  |> map hd
  |> map (fun i -> i.name)

Article © brewbooks — Flickr

If your job is anywhere as hectic as mine, you will have to fix and deploy urgent patches, even when your application code is in a half-written, half-debugged state because of the feature of the month.

This is what branches are for. You keep two versions of the code, one of which is called the trunk and is always ready for deployment, and another which holds the changes that you are working on.

When your feature is done, you merge the two versions together. You want to keep the merge operation painless. To do so, you have several kinds of branches available.

The repository branch is built into your SourceSafe/subversion/git/whatever. It creates two independent copies, and you need to migrate changes from the trunk to every branch out there as soon as possible, or the merge will make you wish for a sweet and merciful death.

By the way, changeset-oriented tools (like git or mercurial) make this easer, while revision-oriented tools (like subversion) make it harder.

The feature branch is done using programming logic. The code you deploy to production supports the new feature, but it is turned off for everyone except yourself. This technique is great for adding features, but inefficient when changing existing ones.

A side effect of the feature branch is that you can stress-test new code by rolling it out to increasing numbers of users progressively.

The comment branch is an odd gambit. It involves ripping out an entire module and replacing it with another that has a different interface. This will involve large amounts of re-wiring all over the code base, and these will take hours or days before they can be compiled, let alone tested.

Use a comment structure such as this one:

/*[*/ old code /*|* new code *]*/

It is trivial to build a text-replacement macro that turns the above into the code below and back:

/*[* old code *|*/ new code /*]*/

Use the macro to switch between development mode (when you write new code and desperately try to get it to compile) and fix mode (when you edit the old code and deploy it). For consistency, always commit the old version to the repository.

Why use comment branches instead of repository branches ? Maybe your source control tool sucks at branches. I use Subversion. Yes, I know. Legacy, pain and unlikely hopes of a brighter future.

When a trunk change occurs in a part that has been erased or reworked in the branch, that change will cause a conflict that will require manual intervention. Even with git or mercurial. For a large number of small changes sprinkled over a large codebase that is routinely involving many small updates, repository branches turn into a merge minefield.

Does your branch involve a small number of well-defined files ?

Then you should use repository branches, because conflicts will only happen in those files, and will usually be easy to fix.

Does your branch involve many changes in many files everywhere in the project ?

Then use comment branches.

Last and possibly least, there is the TODO-branch. This involves non-breaking, purely cosmetic changes. 25% of my project uses this syntax for historical reasons:

Table.get id |-> function
   | None       -> return 0
   | Some value -> return value.count

Then, a convention change happened, and this is used instead:

let! value_opt = breathe (Table.get id) in
match value_opt with  
   | None       -> return 0
   | Some value -> return value.count

Then, another convention change happened, and this should be used instead

let! value = breathe_req_or (return 0) (Table.get id) in
return value.count

And then, there’s the current version:

let! value = breathe_req_or (return 0) $ Table.get id in
return value.count

Whenever I change coding conventions, I do not spend the time to reformat the tens of thousands of lines of code in my application. That would have been wasteful. Instead, every time a piece of code is refactored, it is refactored to the most recent style.

The same happens when using an old and a new version of a given API. My code uses two libraries for handling HTML forms, uses both Javascript and Coffeescript, and a variety of similar two-hammers-one-nail situations.

These are, for all practical purposes, branches. They are work that is being performed for long durations. The benefit of TODO-branches is that code in the middle of such changes is still compatible with the trunk. It all happens in the head of the developer, who remembers what changes should be done the next time a piece of code is rewritten.

Article Image © Dominic Alves — Flickr

Something interesting happened in the Pacific Ocean after World War II: during the war, American soldiers had set up air strips on Pacific islands, complete with little control towers that had radio stations. And the natives observed that those soldiers would speak into a weird piece of metal, and a giant iron bird would land from the sky and deliver food and supplies. Then, the war ended, the soldiers went home, and the natives tried to bring the giant iron birds back by building shacks that looked like legitimate control towers, and spoke strange words into rocks that looked like legitimate microphones, and it worked!

No, it didn’t. Cargo planes did not come back because it was not the shape of the microphone and antenna that brought in the planes, it was their invisible electromagnetic properties. These are the cargo cults: imitating the outside appearance of things that work and expecting the imitation to work as well. The same happens with object-oriented programming — the use of classes, or inheritance, or design patterns is not the reason why good code is good: they are merely tools which happened to be applied by knowledgeable and skilled programmers in order to avoid problems that they knew would happen if they did not preempt their appearance. To look at their code, find out that they used classes, and deduce that classes were responsible for their success is about as rational as expecting to steal the strength of your defeated foe when you eat his flesh. Object-oriented programming can only solve design problems you know you need to solve.

Imagine a piece of old, spaghetti C code written in procedural style, with functions calling each other and accessing global variables all over the place. I suspect many young graduates these days never had to deal with these — and I wouldn’t have either, were it not for my unrequited love for mental anguish and programmer pain. Back to the point: in your mind, draw an arrow from A to B if function B calls function A or accesses global variable A. This is a dependency graph: if you were to change the run-time behavior of A, then the behavior of B would almost certainly change as well. By following all outgoing arrows from the point you altered, you can find every element that could possibly be affected by it.

No programming style or design methodology on earth is going to change this ; whether object-oriented or aspect-oriented or functional, functions call other functions, and changing one part will have an impact on many other parts. That is something we accept, because propagating changes is part of the programming job, and we have even elevated refactoring — the propagation of the absence of changes — to the level of a good thing since sliced bread.

And since we have to track down all the dependencies of a given entity sooner or later, our national sport is to make it as easy as possible. One way is to let the compiler or build process help — compilers detect type mismatches that are usually a hint of non-propagated major behavior changes, while automated unit tests and regression tests handle more subtle changes, and both of these are fairly independent of your programming style. Another way is to willingly reduce the number of dependencies through architectural constraints — Model-View-Controller prevents a model from being dependent on a controller, because altering a controller should not change the behavior of a model. These are fairly common ways to reduce static dependencies.

Then, there are the dynamic dependencies — functions which alter the behavior of other functions at run-time. Remember those global variables from the C program? If function A writes to the global variable and function B reads from the global variable, then calling A will probably change the behavior of B. Runtime dependencies are fairly obvious to map: every time a function writes to a global variable, draw an arrow from the function to the global variable and mark the function as having a side-effect. Then recursively do the same for every function that calls it, and so on until you run out of functions. Your dependency graph just became a lot hairier than it was before.

Such dependencies are the reason why working on the report-printing feature in your invoicing software somehow managed to break the database storage : even though both modules are independent and the static dependency tree for report-printing does not flow into the static dependency tree for database storage, there are a handful of global variables that allow your changes to follow dynamic dependencies and leave a flag unset when it should have been set and ultimately blow up the nuclear silo.

So, when you have global mutable state, then changes can jump from module to module through those pieces of global state. Procedural programming mostly dealt with this by cleanly wrapping up any global state in a module, and having modules interact with each other by means of interfaces that managed to guarantee some invariants on the global state. But still, some problems remained. I am reminded here of the code to Dungeon Crawl, an open-source game written in procedural style, where the module responsible for dealing damage to the player will make frequent calls to the module responsible for outputting messages about the damage being dealt (“The hobgoblin hits you!”). As written, you cannot use the damage-dealing code without displaying messages, so writing an AI module that tries to predict the outcome of an attack is harder than it seems, because it cannot use the existing damage-dealing code to run predictions.

They actually have sludge elves.

Functional programming solved the global mutable state issue by eliminating global mutable state altogether. In the aforementioned example, the damage-dealing module would return the new damaged player along with a list of messages that were generated, and these messages could then be discarded silently in the AI subroutine, or forwarded to the screen in the main routine. I am a big fan of functional programming, but let’s face it: not only is it harder on the brain than good old set-variable-values-everywhere programming, but it does not suit well to situations which have an implicit reliance on global mutable state — such as web servers connected to a persistent database. I look forward to a pure functional friendly database, but it’s not there yet.

Both of these solutions rely on encapsulation: whatever happens to be the implementation of a procedural or pure functional module is hidden, and only available through those functions. This creates a dependency bottleneck, so any changes performed within the implementation can only leak out through the interface in tightly controlled ways — at compile-time, those are called contracts, at run-time they are called invariants.

Object-oriented programming works by deprecating global mutable state. It’s not completely gone, and some programmers yearning for the good old days still try to enjoy some global goodness by using the Gang of Four Singleton pattern, but it’s mostly deprecated. The sickle and hammer of object-oriented programming are dependency injection and late binding, and it uses them to break down the structure of static dependency graph bourgeoisie.

Yes, object-oriented programming is all about building the dependency graph at run-time. Then again, a program that generates machine code at run-time does the same. Object-oriented programming merely provides a set of tools that are easier on the human brain than the extremes of generating machine code at run-time. As an aside, keep in mind that it is not the only toolbox available to you: any language with closures can do the same, and although closures are both more elegant and slightly more complex to handle in the general case, they have found their way into event-based programming because an event handler is just, well, a closure. This is why jQuery passes functions around everywhere instead of requiring the user to implement an IAjaxResultVisitor interface: it’s shorter, but still fairly easy to understand in that context. If you push things far enough, most objects are usually nothing more than dictionaries of closures.

Back to the point: writing a proper object-oriented program with those tools is about the same as creating a statue out of stone using sculpting tools, and inheritance will not turn someone into an object expert any more than holding a chisel turns them into a mutant ninja turtle disambiguation page. How does it happen?

The original problem with procedural programming was that there was no easy way to unplug the damage-dealing module from the message-printing module. Dependency injection means that the damage-dealing module (which is now an object) does not know about the message-printing module (which is now also an object), instead, whoever needs to use damage-dealing code must provide it with a message-printing object that will be used to print whatever messages come up. That message-printing object might actually print those messages to the screen, or it might silently discard them, or it might keep them around and only display them to the screen if a specific event happens, or it might be an unit testing mock, or any other amount of different behaviors. In terms of dependency graphs, there is no static connection between damage-dealing and message-printing: the program decides at run-time what message-printing object should be bound to what damage-dealing object, and can create new objects on the fly should the situation require it.

Aside from that, there are no significant architectural benefits to using objects or classes — although touted as an object-oriented achievement, encapsulation is commonly available in functional and procedural programming as well, and using object.function() instead of function(object) is a matter of taste, not architecture.

As a final word, I am fairly doubtful of the ability of schools to teach their students about good object-oriented programming. They can certainly teach them how to use the tools, but the pains that object-oriented programming is meant to solve only become obvious when you have to work with a large, multi-developer project over a long duration and with changing requirements — anything less, and your pains will be subtle feelings of awkwardness instead. Definitely not something you can learn from. If you have never written an unmaintainable piece of mud, you cannot know how object-oriented programming can keep you from writing one.

You may now dish out punishments in the comment box below. Have fun.

See you around!

Savings accounts work on the basis of simple interest : what you get paid is a linear function of both time and money. Lend for half a year? 3% ÷ 2 = 1.5% Lend for two years? 3% ×2 = 6%

An important thing to bear in mind is that interest is paid at fixed intervals, for instance at the beginning of January. You don’t have to spend those 30 € : you can them on the savings account and earn simple interest on them after a year (3% of 30 € is 0.90 €).

Using this strategy, lending for two years is done at a 6.09% rate instead of 6%, because you get interest on interest. This is known as compound interest : what you get paid is an exponential function of time. Lend for two years ? (+3%)² = +6.09% Lend for three years ? (+3%)³ = +9,27%

The mathematical justification is that, with a 3% interest, your total amount of money is multiplied by 1.03 every year:

1,000 + 30 = 1,000 + 3% of 1,000 = 1,000 + 0.03 × 1,000 = 1.03 × 1,000

So, after two years, the amount is multiplied by 1.03 two times, and so on.

1,060.90 = 1.03 × 1,030 = 1.03 × 1.03 × 1,000

In short, percentages have a multiplicative effect.

And now, pop quiz : I’ve gained +5% weight over the winter holidays. What percentage of my weight do I have to lose to be back to normal ?

If you answered -5%, you missed the point. Multiplicative effect means the total change of weight would be +5% × -5% = 1.05 × 0.95 = 0.9975 = -0.25%. I would be losing too much weight !

The correct answer was 1 ÷ 1.05 = -4.76%.

Similarly, if the number of graduates of a given school increases by +10% on year one and +25% on year two, the total increase is +37.5% and not +35%.

Duality

This is where mathematicians (and computer scientists) use an interesting little concept called duality. Percentages are numbers that are easy to understand, but hard to combine. We can transform them into something that is a little bit harder to understand, but easier to combine.

The traditional way to transform multiplication into addition is to exponentiate, due to an interesting property of the exponential function:

exp(a) ×exp(b) = exp(a + b)

So, I wish to find a percentage operator (§) such that:

• we conserve some values, 0§ = 0% and 100§ = 100%
• applying A§, then B§, is equivalent to applying (A+B)§

Then this uniquely defines an operator which is called exponential percentage:

A§ = B%  ↔  A = 100 × log(1 + B ÷ 100) ÷ log(2)

Some common values:

So, if I gained +5§ weight over the holidays, I can lose -5§ weight and be back to where I started, and if a number increases by 10§, then by 25§, it increases by 35§ overall.

And of course, a yearly interest rate of 4.2§ = 3% compounded over ten years is 42§ = 34%.

No Free Lunch

Normal percentage rules make compounding hard, but it’s reasonably easy to estimate a percentage based on a fraction. Exponential percentage rules make compounding easy, but evaluating a percentage based on real figures is harder.

In practice, compounding happens less often than evaluating, so humans use normal percentage rules. And computers are good at compounding through multiplication, so they don’t need exponentiation.

Duality does have some other uses, though. For instance, there’s the duality between two representations of complex numbers:

a + ib = r exp iθ

The cartesian (a,b) notation makes it easier to add numbers, but multiplication is harder:

a + ib + c + id = (a+c) + i(b+d)

The polar (r,θ) notation makes it easier to multiply numbers, but addition is harder:

r exp iθ × s exp iφ = (r × s) exp i(θ+φ)

For mathematically-oriented computer scientists, duality is a gold mine, because it lets one reduce a complex problem in one area to a simpler problem in another area (whether simpler means faster, as in the case of FFT, or easier to think about)..

The Law of DSLs

There’s one common duality that is fundamental in the computer world: the correspondence between data and code. In a fit of narcissism, let me sit wisely atop a tall mountain to announce Nicollet’s Law of Domain Specific Languages:

Any sufficiently complex data processing algorithm is as an interpreter for a small domain-specific language, and the data being processed is a program executed by the interpreter.

In some cases, this law only complicates things further. In many cases, however, the different angle it provides leads to many advantages, one of them being to transform a non-programming concept (such as an accounting file format) into a concept programmers are familiar with (a programming language).

A minimalist language design culture is enough to grasp several interesting concepts about executing code, which can be quite handy when processing data:

1. Compile to Bytecode

Interpreters don’t execute a string of characters. They tokenize that string, turn the tokens into an abstract syntax tree representing operations, functions and variables, then turn that syntax tree into a sequence of small, executable operations. That sequence is then fed into a virtual machine (or further compiled to machine code) to perform the actual operations.

If the input data for your algorithm is very complex, you can begin on the other side: what will the algorithm do with the data? Will it be inserting the data into a database? Constructing a data object from bits and pieces? What you are looking for is a set of atomic operations you can apply to generate the result. Implement these operations, then start working on a translation algorithm to turn the input data into such operations.

There are several common and friendly representations for such atomic bytecode:

Instruction lists are executed in order. This is your classic assembler listing, without the jumps. A typical “parse file and insert into database” algorithm would generate such an instruction list, and every instruction would be an INSERT, DELETE or UPDATE. Works best when you can read the data and generate the instructions in the right order: if you cannot get the list in the right order from the start, consider another approach.

Dependency graphs work like makefiles: you have several instruction lists floating around with relationships between them, indicating that one list has to be executed before another. A topological sort of the graph results in a single classic instruction list you can execute. A multi-file import, where some files contain data needed in other files, can be the way to go.

Nested scopes are the typical extension to instruction lists: every item in a list can be either an instruction, or another list, possibly tagged with some data. This could be a conditional (if this condition is true, execute this list), a loop (though it is best to avoid these) or a context (a “polygon” scope contains “insert vertex” operations that apply to that polygon). You can even allow variables in a let-in fashion (of which the polygon example above is just a special case) ! Note that nested scopes can be easily represented as XML.

2. Static Analysis

A side-effect of compiling to bytecode is that you get to process the entire file before you actually perform the intended operations. This makes a rollback easier if you notice that there’s an error on the last line of the file: if you make sure that no atomic operation in your target language can fail due to bad input (such as incorrect data values), then you can check your input data for correctness without doing anything to your program state.

Even better, if your compilation process is cheap (linearly traverse a file for parsing) and you have heuristics for predicting how much time and resources your individual instructions require, then you can try to accurately predict the needs of the entire process.

Static analysis also means you can optimize. If, for instance, you’re inserting data into a database and need to resolve names or keys frequently (such as “add this item to list #732″), you can easily construct a table of needed keys (that you can get in one query when the processing starts) using the dependency graph approach.You can also optimize resource allocation by using common register allocation techniques: sort your dependency graph to keep as few resources in memory as possible at any given time.

3. Caching

Try to perform most of the processing offline.

For instance, if you frequently “apply” one file to another, such as a nearly-constant “list of categories” file used to resolve the “category” key in a daily object import, you can benefit from compiling the nearly-constant file to an easily loaded, easily applied format.

You see a cached dictionary that maps keys to categories? I see a DSL that allows dictionary literals as part of the language, and a source file that contains a literal mapping keys to categories, with an interpreter that can apply constant propagation to dictionaries.

Another benefit is when applying changes to mission-critical software. Inserting lots of data into a web database can create a heavy load on the server and make the site unavailable to visitors. It might therefore be preferrable to pre-compile the imported data into requests through a process that keeps a light load on the server, then run the requests.

Besides, with proper nested scoping, you can slice an import into several transactions. This keeps the lock count low, allows spreading the transactions over time to reduce the load, and lets you resume the import process if, for some reason, it gets interrupted.

Understanding concepts and details is a fundamental part of a project’s time budget, whether they come from the project requirements, real-world constraints, third party code or teammates. Every time information goes around in a project, it uses up valuable time, and to keep the time budget tight it becomes necessary to decide what information should be allowed to go around, and where.

Working on concurrent systems is an enlightening experience, because of the many similarities between an array of computers and a team of information workers. Computers arrays have latency issues when one thread depends on another thread to be done…

“When do you think your settings import module will be done? I’m stuck on the payment API until I can load those settings!„

…they have bandwidth issues and manipulating some data yourself is usually faster than sending the data to another part of the cluster for treatment…

“The User object? Well, it’s a bit of a weird design, but it’s rather clever. I’ll draw you a quick UML sketch on the blackboard so you can see what the five helper classes do.„

…they have to avoid data loss if a computer or network is down…

“I have no idea how this stored procedure works, you should ask Tim, he’s the one who wrote it. He’s in southern France right now but I think he’ll be back next month.„

… and they have to handle a directory of parts and a garbage collector for data…

“Wait, nobody’s written the comment moderation back-office! Who was in charge of doing it? Who wrote the comments front-end anyway?„

There are algorithms, strategies and techniques for handling and optimizing those things. Many of these can be adapted to humans, with the added benefit that, humans being smart, they can understand the point of those algorithms and compensate for minor flaws if the plan isn’t perfect.

None of this exists in OCaml.

It can be rather easily reconstructed using existing language semantics, however:

let scoped user resource clean =
  try
    let result = user resource in
    clean resource ;
    result
  with exn ->
    clean resource ;
    raise exn 

let with_input name f =
  scoped f (open_in_bin name) close_in 

let with_output name f =
  scoped f (open_out_bin name) close_out

Quite frequently in the course of software development, certain parts of a program are deemed global at some point in time: there never ought to be a need for more than one instance, nor the need for polymorphic handling of a similar yet distinct part. And sometimes, but not always, this initial impression is proven wrong, and the need for more than one instance arises. The consequences of such an error are usually quite extreme: a one-of global entity tends to be referenced by name, requiring a refactoring effort to convert it to a parameter everywhere this is needed. And so, developers are given the choice between going with the initial impression (with the risk of having to convert it) or ignore the initial impression (and spend time on something they might never need).

I propose here a missing link, a third alternative which implements a one-instance form with minimal effort, yet can be easily refactored into a several-instance if the need arises.

Of course, this applies only in situations where a doubt exists. If it is obvious right away that a given entity will be a several-instance entity, then it should be written as such from the beginning. Only in situations where the current state of the design (especially in Agile methodologies) only requires a single instance should this missing link be considered.

But first, a few background information about this.

Procedural and Objective

A while ago, I wrote a short article that discussed how procedural programming approached the issue of global state. The basic conclusions were that by grouping global state into modules, the number of possible interactions decreased, and those that occurred were made obvious by the structure of the code. As such, a procedural program is a set of stateful entities that interact with each other along a dependency graph.

So is an object-oriented program: stateful objects reference each other and communicate through message-passing. At runtime, there is therefore no major difference in this respect between an object-oriented program and a non-objective procedural program. The main difference is elsewhere. Objects, unlike modules, are rarely defined in the code: they are usually created dynamically at runtime. So, while the two dependency graphs may look the same, the object-oriented runtime dependency graph is a purely dynamic construct that can be altered at will, whereas the procedural runtime dependency graph is a static construct that is tightly mapped to the structure of the code.

The extensibility of object-oriented programming comes from the dynamic aspects of its dependency graph: if objects can be created, replaced and removed at will, even by the program itself, then the program will have a flexibility and adaptability that exceeds the static module skeleton of a traditional procedural program. On the other hand, the static module skeleton is easier to design and implement than a fully flexible object architecture where elements can be hotplugged—even with language support for object-oriented programming.

When you have to build a bridge, you don’t build a fully automated moving bridge just for the fun of it: if the simplest thing that could possibly work is a plain old suspended bridge with no moving parts, then this is what you do.

Remember that we’re considering here the case of a one-instance object being converted to a multiple-instance object even though there is no apparent need for it yet. Why is that more difficult than using the one instance directly?

• Polymorphism requires abstraction, and abstraction usually comes from having several instances—then you can identify what’s general and what’s specific. Trying to abstract a concept from a single instance will miss some general aspects that could have been useful, and keep some specific aspects that will limit extensibility.

• It’s easier to program something which makes sense. If having multiple instances does not make sense yet, then the programmer cannot rely on his instinct or his intuition to determine whether what he’s writing works or not, and the single-instance assumption might fight its way stealthily into the program anyway. Testing a multi-instance system when you only have one instance is quite difficult, too.

• When using multiple instances, one has to consider what scope they should exist in, and how they are propagated down. Of course, there’s the code required to have the multiple instances trickle down to where they are used. But the real difficulty is deciding what the correct scope is—and since there isn’t a reason yet to consider multiple non-global instances, it’s quite probable that such a decision will turn out to be wrong in the end anyway.

• Handling a flexible design is harder, mentally, than handling a fixed design. In a static module skeleton, the dependencies are clear and, when they are not, can be extracted from the code very easily. In a dynamic dependency graph, one has to think about how that graph will be constructed at runtime before they know whether a certain piece of code can affect a certain concept (especially if that concept is still a one-instance global).

As such, delaying the setup of a flexible and dynamic dependency graph as long as possible reduces the amount and difficulty of development efforts. The downside, of course, is the refactoring effort required if the initial assumption is proven wrong. A solution to this conundrum would keep the conceptual simplicity of a single-instance static skeleton, while having a code layout that can be easily changed to a multi-instance dynamic graph as soon as the reasons for such a change are discovered.

Singletons?

While often used to implement single-instance cases, the Singleton design pattern does not solve this issue: it is a pattern for representing the single instance, and the problem here is related to how the rest of the program refers to the instance. If all your code uses a certain instance by name, it does not matter whether that instance is a global variable or the instance of a singleton class: that code cannot use a second instance until it uses that instance by reference, and that reference is a parameter of some kind.

So, the choice of a singleton (or any other approach) to represent the instance is completely independent of any solution that could be found to this problem.

Bottom-Up Construction

An approach which can be used in many cases and leads to avoidance of single instances is bottom-up development. Since the system is being built bottom-up, most of the time when a module would need to access a single instance, that single instance has not yet been written or even designed yet. In that situation, the correct approach is to create an interface representing what the module expects, and have a parameter somewhere to receive an instance implementing that interface. Then, when the single instance is finally designed, it can be made to implement that interface (or adapt to it) and then passed as an argument to the relevant elements.

Bottom-up construction eliminates many of the problems related to handling single instances. That is, by refusing to assume anything about the object being used beyond its interface (including, of course, how many instances exist that implement that interface) development automatically deviates from the single-instance approach and can handle multiple instances automatically.

This does not solve everything, however. Bottom-up construction is notoriously ill adapted to fast development cycles, because the actual functionality of the program appears only when the top-level objects have been developed, which is at the end of the cycle. By contrast, with a top-down approach, the top-level object can be developed first and filled with mocks, and the mocks replaced with the actual functionality on the fly. Besides, bottom-up construction is also subject to violating YAGNI, because objects are created before their users exist (and may end up not existing at all).

The Missing Link

The key of the problem here is that directly converting a single-instance approach to a multiple-instance approach requires a lot of argument-passing: since the instances are manipulated through references, and those references are parameters, a lot of parameters are required to pass the instance from its point of creation (quite probably somewhere near the entry point) down to every part of the program that uses it. A possible solution would be do to as much of that work as possible (replacing as many of the global accesses as possible with references) without having to heavily increase the number of parameters.

Aside from global symbols, one other kind of value is available in functions without being passed as a parameter: member variables.

The Screen Design Pattern

The problem: methods of a class needs to access a global variable directly. Making that variable non-global is unnecessary right now, but might be necessary in the future.

The solution: have the class keep a screen member reference to the global variable. That reference is initialized in the constructor and used everywhere else. Member functions other than the constructor are never allowed to reference global variables.

What would be an example of refactoring using this pattern? Consider the original C# code:

public class GlobalUser
{
  public GlobalUser() {}
  public void MethodA() { Global.Instance.Use(); }
  public void MethodB() { Global.Instance.Use(); }
}

Methods in this code use a global instance. Let’s replace that with a member reference to the global instance:

public class GlobalUser
{
  private Global screen;
  public GlobalUser() { this.screen = Global.Instance; }
  public void MethodA() { this.screen.Use(); }
  public void MethodB() { this.screen.Use(); }
}

Using the pattern, the refactored code can be written straight away (instead of writing the original code and then refactoring it) without requiring mental effort, which means that using the pattern bears almost no cost (it only requires an additional member variable and an additional assignment in the constructor). And if the screened variable suddenly has to become non-global, only the constructor will be affected, saving precious time that would have been lost refactoring the methods as well.

Note that creating a property which returns the global variable (instead of a member variable assigned in the constructor) is equivalent in terms of writing the initial refactored code (the this.screen = Global.Instance is replaced with { get { return Global.Instance; } }), but does not have the same benefits when refactoring to remove the global instance because a connection will have to be made between the value returned by the property and the parameter received in the constructor, which requires more code. I therefore suggest using a member variable assigned in the constructor instead of a property.

Also note that the above can be seen as the preliminary step of a dependency injection and follows the same general idea as preparing your application for Spring.NET integration (for example). The difference is that the pattern does so at the class scale, and emphasizes development speed (using the pattern should be no slower than using the non-pattern approach) rather than external interoperability and flexibility.

Memory issues

Of course, storing a screen reference in every object might not be welcome. Some very small objects, such as vectors or polygons, cannot afford to store an additional reference because of the time and space overhead this would imply.

My suggestion in these situations is as follows: eliminate the global dependency altogether. Resist the temptation to have a certain small object frobnicate itself using a global instance, and instead have a larger object frobnicate the small objects. This makes the optimization more transparent (the small objects are now just bits of data without any kind of polymorphic behavior that are externally manipulated) and allows using the Screen design pattern on the larger object.

Of course, soon after the tax appeared, people reduced the number of windows and doors, sometimes going as far as wall every doors but one, and brick-up all the windows.

A similar thing happened in France in the 1990s: until then, insurance for work accidents was provided equally to all companies, and all companies paid the same flat rate based on the national accident rate. In an effort to make large companies pay for improved safety measures (because they had the funds for it), the insurance rate became deregulated for companies above a certain size, so that a large unsafe company paid more than if it was safe. Small companies still paid a national rate.

Unlike what was expected, most large companies spent no money on safety and instead outsourced dangerous jobs to small companies, thereby improving their own rates. And since small companies often had worse safety than large companies, the overall workplace safety slightly decreased as a consequence of this deregulation.

The same happens in the computer world. When your log software advertises a thousand visitors per day, it simply hides the fact that HTTP has no notion of visitor:

• A given person may use several computers and IP addresses to visit the same site.
• A given person may use a proxy cache to read the website without having to connect to it.
• The same IP address may be shared by several people (by ISP shuffling, on public computers and on LANs).

Similarly, when your download client displays a download speed, it’s quite probably an average over the last few seconds, meaning that if you are suddenly disconnected from the network, the download speed will remain above zero for a short while.

Of course, every user of the HTTP protocol knows, deep down, that HTTP has no notion of visitor and therefore a visitor is just a set of requests from the same IP over a given duration, just like every user of a download client can understand that download speed is an approximation and isn’t really above zero when there is no network connection.

Yet, it’s quite easy to forget.

An example upon which I’ve stumbled recently was a paying web site. Paying users were allowed access to some functionalities of the website, so there was a test performed on every of these pages to see if the user account was associated to valid payment information before displaying the page. Nothing too fancy, a mere if(user.Subscribed) test that didn’t deserve factoring out. Then, one day, marketing suggested adding a temporary test account that could be used for a limited duration to allow using those functionalities.

The obvious problem is that the previous assumption “only paying users can read this page” that was spread all over the system became incorrect. Cheating by changing the definition of user.Subscribed was impossible, because subscribed users have valid payment information but test accounts don’t. So, a new test had to be added to every page: if(user.Subscribed || user.TestAccount). Over a few months, the test became increasingly large as new properties were added for access.

Luckily enough, a sane developer came around and refactored the thing as an if(Functionality.IsAllowed(user)) function which allowed changing the access restrictions for these pages in a single place, and developers from that team learned not to make explicit assumptions within the code, not even on extremely simple tests.

The Window Tax Antipattern

The Window Tax Antipattern can appear when:

• A given module provides a piece of information, Provided.
• A rule (that is neither a consequence of the definition of that module nor an absolute truth) describes how another piece of information, Deduced, can be deduced from Provided.
• Other modules use Deduced by manually deducing it from Provided.

In such a situation, if the rule changes, the deduction becomes incorrect and every module which uses the deduced information will have to be changed as well. As an additional issue, since the deduction is not explicit, finding every module that makes that deduction is often difficult. The solution is to identify the rule, and wrap it in a function that constructs Deduced from Provided (thereby allowing easier refactoring should the rule change).

In our above examples:

• The provided information is the number of doors and windows, and the rule states that wealth can be deduced from that number.
• The provided information is the accident rate, and the rule states that the safety of workers can be deduced from that rate.
• The provided information is whether the user has a paying account, and the rule states that the page can be viewed only by users with paying accounts.

In some cases, the antipattern does not apply. For instance, if the rule is a consequence of the definition of the original module, then the rule cannot change unless the underlying module also changes (which is an acceptable reason for refactoring). So, if the rule is “when you push an element onto a stack, the stack is not empty until you pop it”, then assuming that a stack onto which you just pushed an element is not empty is not a problem (until, of course, either you or another part of the program pops that element).

However, every single business rule (those that can be changed by management, marketing, clients or even laws) is potentially subject to this antipattern unless cleanly wrapped.

In an environment that is heavy on business rules, the Adapter pattern becomes of particular interest. Since every object usually cares about information that is not directly contained in another object business rules crystallize as adapters around objects, providing the information that the current object requires (for instance, “can this user view this page?”) based on the information that the other object and its environment provides (for instance, “is this a test or paying account?”). View with suspicion every direct interaction between two business objects, and never hesitate to insert adapters or explicit rules whenever the used information is not a direct, obvious and inevitable consequence the provided information.