It’s is Better™ to have many short files than a few long ones. One reason is incremental compiling with ocamlbuild : that the smaller your files are, the smaller the percentage of code to be compiled when you make a small change. Another reason is that files provide a natural delineation of code that makes it slightly easier to reason about.

The very process of splitting a large file into smaller files is also an excellent way to clean up the code. Every split is an opportunity to move some code to a more generic location — why have a CMember_importParser module when all of its functionality could fit into an OzCsv plugin module ? Even when no such generic solution exists, cutting through the jungle that a 2000-line module contains helps clean up dependencies, identify shared functionality and imagine better ways to design code.

Still, when cutting up code this way, the problem of encapsulation remains. If code that relates to pictures (an upload module, a transform module, a download module, an access rights module) is split across several files, it is desirable to let each file access functions and values from other values that would not otherwise be shown to modules not related to picture processing. For instance, a get_download_link function should be available throughout all picture-related modules, but the rest of the application should use the get_download_link_for_user function that checks whether the user is allowed to download the file.

In order to achieve several nested levels of encapsulation required to work with modules this way, I have come up with a convention :

• A module name (and thus, a file name) is composed of segments written in camelCase and separated by underscores. For instance, CEntity_view_grid is a module name containing segments CEntity, view and grid.
• Modules with only one segment are public. Any other module may include, open or otherwise reference them with no limitations beyond what the module signature says. So, CEntity may access MGroup freely.
• Modules with N > 1 segments are private. They may only be accessed by modules which share the first N-1 segments. So, CEntity_view is available to modules CEntity and CEntity_edit but not CPicture.
• A module with N segments may export any module with N+1 segments it can access, possibly under a more restrictive signature. For instance, CEntity_view is available to all other modules as CEntity.View.

To make these rules easier to respect, private module dependencies are made explicit by adding a list of module aliases at the top of each file. The top of my cEntity_view.ml file starts with :

module Sidebar     = CEntity_sidebar
module Unavailable = CEntity_unavailable
module Edit        = CEntity_edit
module Info        = CEntity_view_info
module Directory   = CEntity_view_directory
module Grid        = CEntity_view_grid
module Wall        = CEntity_view_wall

It is forbidden to use a private module without going through such an alias, and it is forbidden to define such an alias anywhere except at the top of the file. This makes it extremely easy to determine whether private access rules are respected.

The rule of thumb for splitting files (in my particular coding style) is :

• Code for separate layers (model, view, controller…) go into separate public modules.
• For complex code (such as complex rules in model or controller code), consider splitting files larger than 200 lines.
• For simple code (such as HTML template or JSON serialization definitions), there is no splitting limit except for factoring out common behavior.

You might not understand what I am rambling about in the two sentences above. What is going on?

Until recently, humans were fairly similar to each other in terms of capabilities. If an individual decided to annoy, harass or harm others, their impact would be limited to what they could do on their own before being stopped, or they would have to convince enough people to help them. When you were a large organized group of people, you only had to care about other large organized groups of people.

Even with the advent of modern technology up to the 1980s, when two people in a stealth bomber could mash tens of thousands to a pulp by pressing a button, this was still the consequence of an industrial infrastructure allowed by a large organized group that willingly granted those two people such power.

Guns are a special case. An individual with a gun can do more damage than average, faster and without retaliation. What happened next is hardly surprising: some people decided to fight fire with fire and buy their own guns as a deterrent, while others decided that gun ownership — or rather, the increase in destructive power provided by guns — should be heavily regulated.

And then, computing and the internet happened. Owning an internet-connected general-purpose device is quite affordable these days.

And owning an internet-connected computer increases the capabilities of individuals by several orders of magnitude, as far as processing data and communicating with others is concerned. While computers are not as lethal as guns (tech support calls excluded), the increase in lethality provided by guns is much smaller than the increase in processing power and communication reach provided by computers. This is literally the first time that we, as a species, have to deal with some individuals having enough power to harm, disrupt or topple large organized groups.

A handful of people can harness the power of computing to send billions of spam messages, bring down entire web sites or networks or nuclear plants, make elite KGB spies green with envy, collect personal information about thousands of people, tell millions of people about a new law that mainstream media are silent about, or illegally distribute content without compensating its creators.

Is it any surprise that corporations, organizations and countries are fighting to regulate such incredible power?

You may not run software on iWhatever devices without consent from Apple. In France, you are legally responsible for when third parties commit crimes using your IP address. Various laws around the world including SOPA, PIPA and ACTA aim to provide counter-measures to illegal distribution of content. Germany and the UK are getting anal-retentive about what cookies you are allowed to send to your users.

And yet, there are immense benefits still to be reaped from a free, open, uncensored internet, and I am certain they far outweigh most of the costs involved. To willingly throw away those benefits in order to maintain existing business models and political habits strikes me as a very bad idea indeed, and one we should fight against.

Another alarming development is that the general public, lawmakers included, are woefully incompetent when it comes to computers and the internet. They do not understand why it works, and so they do not understand why actions they take might prevent it from working or what negative consequences. Laws are drafted and voted on without asking any experts for input, even when experts are quite outspoken against them.

And we, the experts, are the cause. We are dealing with people who thrive on communicating with the public — we are fairly good when it comes to communication, but only with each other. Is it any wonder that no one listens to us? We are so familiar with the details, nooks and crannies of our high-tech world that we fail to explain, in simple terms, why non-experts should care about these issues.

Why did it take something as critical as SOPA and PIPA to get us moving? Shouldn’t we be the ones leading the conversation on computers and the internet, instead of mumbling in our collective niche beards when clueless members of our parliaments speak of «series of tubes»? Could it be that we are so used to wielding awesome individual powers of communication, that we have forgotten how to team up to make our voices heard?

I am a PHP developer

• I am not a Zend Framework or Symfony or CakePHP developer
• I think PHP is complicated enough

I like building small things

• I like building small things with simple purposes
• I like to make things that solve problems
• I like building small things that work together to solve larger problems

I want less code, not more

• I want to write less code, not more
• I want to manage less code, not more
• I want to support less code, not more
• I need to justify every piece of code I add to a project

I like simple, readable code

• I want to write code that is easily understood
• I want code that is easily verifiable

Without surprise, a large swath of the community did not take it well, for similar reasons to my earlier piece against Zend Framework — deviation from the commonly accepted norm.

I have come a long way since I wrote that article, and I must have been walking in circles, because I actually ended up where I originally begun : why do we call these things frameworks ?

Zend, Symfony, CakePHP — as well as Node.js, Rails, Django, Ocsigen … — actually contribute three different things to projects that use them.

Libraries

A library provides functionality used for solving general problems in a flexible, standalone manner. Zend_Mail is a classic example of the library aspect of Zend Framework: you can plug it into your application and start sending e-mail. The interface you would use is uncluttered by details that are not directly related to sending e-mail.

The core qualities of a library are its power (how many different aspects of a problem does it let me solve — attachments, rich text, bouncing, MIME handling…) and the clarity of its interface. What problems can you solve, and how fast can you solve them?

Conventions

When you hear «conventions» you immediately think of opening brace positions and variable naming rules. It’s about more than that.

The Model-View-Controller separation is an example of convention: it has been decided that under no circumstances should HTML rendering occur in Model code, no HTTP or session handling should happen in View code, and no SQL queries happen in Controller code.

Good conventions are designed to let the developers assume interesting properties about the code without having to actually read it. A convention like «no global variables» means I never have to care about global state in my code, ever. A convention like «view code must respect the law of Demeter» means all the data used by the view is right where it is being initialized.

They are also designed to make reuse and interoperability easier by reducing the number of ways in which a possible interface can be implemented. A convention could say the values are passed by assigning them to members post-construction and not as constructor arguments, so you have one less point of contention between the object that is initialized and the object that does the initialization.

Last but not least, conventions are usually based on experience of things that could go wrong if certain behavior is allowed. A typical example is the requirement to escape all strings as they are being output — eliminating any ambiguities as to whether the string has already been escaped elsewhere and should be output as-is: it has not.

Zend comes with a variety of useful conventions enforced both through the interface of its tools — this is how you use a view, this is how you define a view helper that should be available from within any view, this is how you bind a piece of code to an URL, and so on. I happen to disagree with many of those conventions myself — because I believe they solve the wrong problems — but they are certainly better than a project with no conventions.

For the reference, my PHP conventions are described in the user manual for Ohm.

Framework

A framework is actually going a step further than mere conventions. They are super-conventions designed to be respected by plugin authors. The point is that if plugin A and plugin B respect the set of conventions provided by the framework, then they can be used together in the same application.

Consider a practical example : a plugin that implements a CAPTCHA field in a form, and a plugin that displays and submits a form through AJAX. On a bad day, it goes like this :

1. When an error occurs, the server-side AJAX-form plugin sends out a small piece of JSON containing the fields that have errors, along with the error messages. A small client-side script applies these.
2. However, the CAPTCHA plugin expected the image to be reloaded when an error occurs.  It may either keep the same image and target word — defeating the purpose of a CAPTCHA — or change the target word without knowing that the image could not be changed.
3. You then need to post on StackOverflow hoping for a solution, search online for a patch to either plugin that could make it work as expected, or try to read the code to either in order to create the patch yourself.

Had the framework provided a clean notion of « this field must be refreshed on every attempt » as part of their form interface, both plugins would have used it — the CAPTCHA plugin would have marked its field as such, and the AJAX plugin would have implemented a special case for such fields.

As such, the purpose of a framework is to provide a clean, unambigous and extensive vocabulary that all the plugins should be able to speak, and that is designed to cover as much real-world situations as possible.

Zend Framework and Symfony in particular do an absolutely great job of this. When you can have a pager component push its data to the page through a progressive enhancement component, and log its performance to FirePHP when an user authentication component  determines that the viewing user is a developer, and all of it works by plugging square pegs into square holes, you know there has been a lot of great work going on below the hood.

Back to the point

Using a framework is all fun and games until you need to disagree with it. You need to plug out what does not work, and plug your own implementation in its place. The more complex the vocabulary, and the harder it will be to write new code — frameworks make it easy to connect existing components, at the cost of having to deal with more concepts when actually implementing new things.

What it boils down to, in the end, is whether you expect to be reusing a lot of third party components, or to write a lot of your own code. In the latter case, MicroPHP — and lean environments that do not have a heavy framework side to them — is actually an improvement over trying to fit a six-inch wooden square peg into a mini-USB port.

The exception to this is, of course, being so familiar with a particular framework that you immediately know what changes you need to do without fighting against third party code.

Sometimes, on the less important decisions, you might get out of it by default — A disagrees with B, but doesn’t care enough about the topic to actually do something about it, so B proceeds anyway.

Sometimes, decisions will be dictated by competence. If A disagrees with a proposed solution because of objective technical or legal implications, well, that’s it.

Quite often, though, there is no such solution. This is where one must be aware of the dangerous tendency of consensus-based teams to devolve into compromise-based discussions.

Consensus-based discussions feel like this :

A: I’m going to frobnicate the thingamajig.
B: If you frobnicate the foobar instead, you would get a 10% increase in floogum output.
A: You’re right. Let’s frobnicate the foobar, then.

That is, initially differing opinions evolve through an argumentation process, so that everyone agrees that the final outcome is the best one.

Compromise-based discussions feel like this :

A: I’m going to frobnicate the thingamajig.
B: I would rather that we frobnicate the bazquux instead. Might get better results.
A: Actually, I have a hunch that the thingamajig is the better option.
B: Tell you what, let’s frobnicate 50% of the thingamajig and 50% of the bazquux. This way we have everything covered.
A: All right.

Here, initially differing opinions are settled through a negotiation process, so that everyone’s input is respected, but no one agrees the final outcome is really the best.

Compromise is a bad idea for several reasons.

• Quite often, going 100% with any reasonable solution is better than investing only 50% into two different solutions. In more general terms, lack of commitment to a single strategy or objective is a dangerous thing to do, and often less effective than commiting to any strategy or objective one might come up with.
• Compromise is not agreement, it’s negotiation : you give something and get something else in return until everyone agrees that it’s an even trade. People who are good negotiators tend to dictate their terms in such circumstances, and others might feel helpless and useless after a few unbalanced trades. It also leads to « I agreed with your idea yesterday so agree with mine today » or « You’ve already changed my idea enough, stop asking me to change it again, » both of which are content-free sentences that aim to make a decision based on interpersonal history instead of objective analysis.
• The inability to come to a logical conclusion sometimes happens because there is not enough data available to decide. The first priority in such circumstances should be to actually search for the data (possibly accepting one of the proposed solutions as temporary until the data is collected), not settling for an arbitrary compromise.
• Building consensus is hard when people have trouble putting their insight into words, no matter how convincing or true that insight might be. I’m objectively right, but I cannot seem to explain to you the reason for it. This can mean going for a compromise today instead of a consensus tomorrow — hasty decisions are seldom good. Always make sure everyone has had enough time to think the decision through, even if it means adjourning it until later.

The C++ solution is quite straightforward : have a destructor that closes the file handle, and create the handle as an auto variable:

std::ofstream out = std::ofstream("out.txt");
out << whatever();
// file handle is always closed by the language

On the other hand, OCaml does not have destructors, but can simulate the RAII spirit using a closure: you provide a function that will be called with the file handle as its argument, and the file handle will be destroyed when the function returns or raises an exception.

BatFile.with_file_out "out.txt" begin fun out ->
  BatIO.nwrite out (whatever ())
  (* the file handle is always closed by the language *)
end

This is a special case of a more general principle.

Two-way binding

The standard let keyword performs a one-way binding: bind value to variable then evaluate expression. Two-way binding adds a post-processing step : when you’re done with the expression, do something else. Such a behavior has important consequences for writing concise and readable code.

In my OCaml code, two-way binding is performed with keyword let! that is preprocessed as follows :

let! pattern = value in expression
(* Is translated to *)
value (fun pattern -> expression)

For instance, the above file manipulation script would be written as:

let! out = BatFile.with_file_out "out.txt" in
BatIo.nwrite out (whatever ())

This syntax expresses the actual intent of the code better than the anonymous callback syntax did: bind the file handle to this variable, but don’t forget the post-processing steps.

Here are a few more examples of situations that may be improved by this syntax :

Events and reactive programming

Reactive programs can be constructed either using the typical “register this function to be called whenever this value changes or this event happens” semantics, or  by using binding semantics instead:

let () =
  let! user = User.on_change (#last_login) in
  if user # notify_login then
    Mail.send (user # email)
      ("Someone has logged in to your account at " ^ datetime (user # last_login))

The underlying signature of User.on_change (which registers a listener callback and returns unit) remains the same.

Retry semantics

CouchDB implements transactions with retry semantics: you read a document, compute some changes and try saving them back, and if the document was changed by someone else in the mean time, you will have to try again. It makes sense for the code inside the transaction to be 1° idempotent and 2° wrapped away in a function that 3° takes the latest version of the document as an argument :

let set_title article_id new_title =
  let! article = Database.transaction article_id in
  Database.write article_id { article with title = new_title }

In such a design, the write function would throw a specific exception if a collision occurs, and the transaction function would intercept that exception and try again until the transaction succeeded or a maximum number of retries happened.

Monads

Value binding in monads benefits from having a syntax that actually looks like binding.With the option monad, one can turn this :

match Files.get file_id with None -> None | Some file ->
  match file # owner with None -> None | Some user_id ->
    match Users.get user_id with None -> None | Some user ->
      Some (user # name)

Into a more straightforward version :

let  open BatOption.Monad in
let! file    = bind $ Files.get file_id in
let! user_id = bind $ file # owner in
let! user    = bind $ Users.get user_id in
return (user # name)

Also, one can deal with Lwt threads almost as well as the Lwt-specific syntax extension:

open Lwt
open Lwt_io

let process_lines channel process =
  let loop () =
    let! line_opt = bind $ read_line_opt channel in
    match line_opt with
      | None -> return ()
      | Some line -> loop () <&> process line
  in
  loop ()

Being Silly

let fold init list f = List.fold_left (fun acc x -> f (acc,x)) init list
let map list f       = List.map f list

let probabilities odds =
  let sum =
    let! accumulator, odd = fold 0. odds in
    accumulator +. float_of_int odd
  in
  let! odd = map odds in
  float_of_int odd /. sum

The Syntax Extension

In case you don’t know how to create it, this is the preprocessor file for this syntax extension :

open Camlp4.PreCast
open Syntax

EXTEND Gram
 GLOBAL: expr;

 expr: LEVEL "top"
 [
   [ "let"; "!"; p = patt ; "=" ; e = expr ; "in" ; e' = expr ->
     <:expr< (($e$) (fun $p$ -> $e'$)) >> ]
 ] ;

END;

By the way, I find that this extension has a significant advantage over the Lwt extension – it is readily compatible with syntax highlighting in most editors.

Here is a hypothetical situation :

The kingdom is in trouble, and the King must enact a new law. Of course, such a law will make many people happy for years, but some people will be annoyed by it for a few days. He has two possible choices :

Law A will make 20% of the people happy at the cost of annoying 1%.

Law B will make 90% of the people happy at the cost of annoying 10%.

Which law should the monarch enact ?

This is not the kind of problem that engineer types like me enjoy — no matter how much we try to abstract away the details and build a sound foundation for that decision, some ethics will inevitably seep in.  Is it right to annoy an additional 9% of the people so that 70% more people become happier? Where do we draw the line in term of percentage, and in terms of annoyance — a few days might be fine, but what about a few months or years?

As you might expect, this is not an entirely hypothetical exercise. It is, in fact, the very core of the opt-in versus opt-out debate. Above, law A is opt-in : only 21% of the population knows about its effects, but at least there will be very few complaints ; law B is opt-out : it ensures that 100% of the population knows about it, but it will annoy the 20% who will have to manually opt out.

My start-up hosts discussion forums for associations. Every one of our customers has faced the same decision : should they send an e-mail to their association members telling them « please come and sign up on our forums » or should they import their member directory and let naysayers unsubscribe ?

With the opt-in approach — please come and sign up — the initial e-mail is followed by a small number of sign ups, usually from the more active or dedicated members, and will be ignored by everyone else. If the number of initial adopters does not reach a critical mass, the forums will simply die off and be forgotten by everyone.

With the opt-out approach — import all the members — everyone will be notified every time a new message is posted, as would happen on a mailing-list. A forum message from a friend is a lot more interesting than a « please come and sign up » e-mail and drives more members to connect and participate. The critical mass is reached far easier (and faster!) and the forum becomes an essential part of the community. However, those who did not wish to participate will receive e-mail that is literally unsolicited, and they will complain about it — while the number of active users increases significantly, the number of complaints and unsubscription requests increases even faster. The delicious irony of it all is that the number of complaints is driven up because the communication tool helps those annoyed members find each other and speak up in unison.

Before continuing, let me fend off two possible problems.

First, the opt-out approach is not intended to be a sneaky trick — we always strongly advise customers to send a preliminary e-mail to all their members in advance, telling them about the plan to move to a new discussion system. Not only does it keep things civil and honest, but people who absolutely hate receiving messages from their community can ask to opt out before it is too late. Online communities have trouble blooming when 10% of the messages are complaints about the very existence of the community, so it is in everyone’s interests to keep naysayers out.

And if anything else fails, we can wipe out members from our system on demand.

The second problem will be familiar to readers of Seth Godin — permission. Through the eyes of a permission marketer, to import all  the members without their prior explicit opt-in consent is absolute heresy.

I find this view a bit too extreme. Well, it does make sense when trying to sell things that 99.99% of the people will not care about, such as viagra or cheap hotels in Bangkok. But members actually care about what is going on in their association, and — based on our experience — only a minority of members ever asks to be completely removed from the forums. Most members only unsubscribe from individual discussions that they do not care about, and choose to remain available on the forums as a whole.

It feels sad that so many people would miss out on a great experience just so a handful of curmudgeons can spare the effort of clicking an unsubscribe link.

Back to the point.

As far as I can tell, the approach that helps most members get involved in an online community is :

1. In advance, send them an e-mail telling them that the association is about to move to another online community system — in our situation, it could be described as a mix between a forum (for those who wish to be very active) and a mailing-list (for the occasional participants) — and that each of them will receive those communications on an opt-out basis.
2. You will receive messages saying « this is a bad idea » or « I don’t want to receive those communications » and you should take steps to make sure that no person who opted out at this stage is ever imported into the forums. Unless they ask for it later, anyway.
3. After enough time has passed — at least 48 hours for large associations — import everyone into the forums, and write a welcome message there explicitly asking them to say hello when they reach it for the first time.
4. If any other people request to be completely unsubscribed, you may simply remove them from your forums and they will stop receiving messages. If you need to absolutely make sure that all their data has been wiped from our database, drop us a line and we’ll take care of it for you.

We have had several customers build a thriving online community for their association with this approach, and have even seen a few « get me out of here » naysayers change their minds and come back online, once they understood everyone else was using it.

Where do you stand on the matter ?

Article image © Carlo Piana — Flickr

Lakshen Perera provides a list of basic patterns for everyday programming, illustrated in Javascript and Ruby. I thought it would be interesting to provide an OCaml illustration as well, and perhaps a handful of additional patterns as well.

Verify object’s availability before calling its methods or properties

In many languages, there is a possibility for objects to be missing — whether this is represented as NULL, null, nil or None. Regardless of the language, it is important to keep in mind at all times whether a given value is optional or not. If it is not optional, then you may assert so using a type declaration if your language supports it :

method name  : string (* not optional *)
method email : string option (* optional *)

In other languages, a runtime assert will do, for instance in PHP :

assert (isset($this->name))

OCaml will not require you to explicitly declare all values as optional or mandatory. Instead, it will deduce that information from the way you use each value. For instance, since json_of_string expects a non-optional string argument, you can simply write :

let parsed_content = json_of_string json (* json is not optional *)

If json is an optional string, this will create a compilation error unless you explicitly define what should happen when the string is missing. The most common approach is to decide that the result should be missing too :

let parsed_content = match json with
  | None   -> None
  | Some s -> Some (json_of_string s) (* by definition, s is not optional *)

If using the Batteries library (which, by the way, you should), you can express this more easily :

let parsed_content = BatOption.map json_of_string json

Set a default value with assignments

This advice is almost identical to the previous one : if you need to construct a non-optional value, but only have access to an optional value, you will have to provide a default value. Using standard OCaml :

let role = match person.role with
  | None      -> `Guest
  | Some role -> role

Using Batteries, there is an almost equivalent version :

let role = BatOption.default `Guest person.role

This version is almost equivalent, because it will evaluate the default value even if it is not required. Let’s assume that the default value is itself provided by a complex computation, such as a database access :

let role = match person.role with
  | None      -> readfrom database (* only executed if person.role is missing *)
  | Some role -> role 

let role = BatOption.default
  (readfrom database) (* Always executed, even if unnecessary *)
  role

The Batteries library provides an alternate function for that :

let role = BatOption.map_default
  readfrom database (* only executed if person.role is missing *)
  role

Checking whether a variable equals to any of the given values

If the values have a legitimate reason to be strings, integers or other types with unlimited numbers of values, then « is in list » predicates are the preferred choice :

if List.mem current_day ["Monday";"Wednesday";"Friday"] then
  (* do something *) 

Days are a quite bad example, because these are better represented as a variant (which then type-checks whether you have written Mornday instead of Monday). If you have already defined the type of weekdays, then :

if List.mem (current_day : weekday) [`Monday;`Wednesday;`Friday] then
  (* do something *)

If this is a one-shot check, or if you would rather not define a weekday type yet, you should instead go for an exhaustive pattern-matching :

 match current_day with
  | `Monday | `Wednesday | `Friday ->
    (* do something *)
  | `Tuesday | `Thursday | `Saturday | `Sunday -> ()

Extract complex or repeated logic into functions

This is a fairly fundamental concept — but it consists in two distinct parts. There is a separation and naming part (you pull out a piece of code and give it a name, which helps understand what it does and how it relates to the rest of the program) and there is an extraction and reuse part (the piece of code is pulled out into a more globally accessible location and parametrized, so that it may be used in other places).

With the above example, simple separation-and-naming would be :

let is_discount_day = match current_day with
 | `Monday | `Wednesday | `Friday -> current_date > 20
 | `Tuesday | `Thursday | `Saturday | `Sunday -> false
in

if is_discount_day then 
  (* do something *)

The variable is defined in the same scope it is use in, and it assumes that current_day and current_date values have been defined previously in that scope. Extraction-and-reuse would go further :

type weekday =
  [ `Monday | `Tuesday | `Wednesday
  | `Thursday | `Friday | `Saturday | `Sunday ]

let is_discount_day (day:weekday) date =
  List.mem day [`Monday;`Wednesday;`Friday] && date > 20

...

  if is_discount_day current_day current_date then 
    (* do something *)

Now, is-discount-day is a global function available from everywhere in the code, and it uses the provided parameters to determine whether this is indeed a discount day.

Memoize the results of repeated function calls

OCaml has several ways to perform memoization. One of them is lazy evaluation :

val discount_day = lazy (is_discount_day current_day current_date)
method discount_day = Lazy.force discount_day

The lazy expression will only be evaluated the first time the Lazy.force function is called on it.

Note that if the current day or current date can change, then the memoization actually breaks things !

Memoization is also helpful when dealing with a function that requires arguments, in which case a different result will be provided for each argument set. A common solution is to use a hash table to store these :

let fibonacci =
  let memo = Hashtbl.create 100 in
  let rec fib n =
    try Hashtbl.find memo n with Not_found ->
      let result = fib (n-1) + fib (n-2) in
      Hashtbl.add memo n result ; result
  in fib

This works fine for short-lived functions — don’t do this for global functions that might stick around for a long time, because the memoization hash table will grow and its contents will never be garbage-collected. If you really have to, use a weak hash table, such as Batteries’ BatInnerWeaktbl, so that the garbage collector may reclaim the memoized values when it runs out of memory.

Also don’ t overdo memoization — it only works when arguments are reliably passed more than once and the time to compute the value is significantly larger than the time to retrieve and store it and it is worth the memory usage and the function has no side-effects.

Use the seven list manipulation primitives

Almost any processing on collections of items can be expressed in terms of seven fundamental patterns. Recognizing those patterns can help improve the clarity of both the code and the underlying algorithm.

1. Map transforms a list into another, item by item, in linear time. Use a map operation when all you need is a one-to-one transformation. The line below extracts three recipes from the database using their identifier.

let recipes = List.map from_database [ "omelet" ; "cheeseburger" ; "risotto" ]

2. Reduce transforms a list of values into a single value by repeatedly applying a function that combines together two values into one. The typical example is a fold, which uses a function to combine each list element, in turn, with an accumulator. It can be used to extract the sum of values in a list, for example :

let total = List.fold_left (+) 0 [ 5 ; 6 ; 3 ; 8 ; 9 ; 0 ; 7 ; 6 ]

This transform allows a preliminary map step which transform the values inside the list into values that can be combined. For instance, to find the age of the oldest person in a list of people :

let oldest = List.fold_left (fun acc person -> max age person.age) 0 people

3. Extract works like a map, but the transformation function returns zero, one or more results for each call. All the results are included in the final list. The most elementary implementation is literally to have a map (that transforms a list into a list of lists) followed by a concatenation (that transforms a list of lists into a list). For instance, to get all the ingredients involved in a list of recipes :

let ingredients =
  List.concat (List.map (fun recipe -> recipe.ingredients) recipes)

4. Filter is a subset of Extract where the transform may not return more than one result — but it may still return none, so its result is simply an optional type. For instance, to extract the list of all recipes that have a wine recommandation along with their recommended wine :

let wines = BatList.filter_map
  (fun recipe -> BatOption.map (fun wine -> recipe,wine) recipe.wine) recipes

The OCaml language also provides a standard List.filter function which keeps values for which a property is true. For instance, to get the list of recipes that have a wine recommendation :

let have_wines = List.filter (fun recipe -> recipe.wine <> None) recipes

This approach is weaker — the wines in the resulting list are still treated as optional by the type system, so you will need bogus pattern matching for a case that never happens (no wine) to extract the actual wine values. A filter_map lets you encode the filter property in the type of the result, which makes using the filtered list easier.

5. Sort unsurprisingly sorts the list. The canonical sort — using the canonical order relationship — is a theoretical curiosity, and in practice most sorts use a projection function p such that A < B iff p(A) < p(B). This is best illustrated in SQL, in the form of the ORDER BY <projection> statement. Two useful helper functions :

let project compare p a b = compare (p a) (p b)
let descending compare a b = compare b a

For instance, to sort the list of recipes based on how long each of them takes :

let by_duration =
  List.sort (project compare (fun recipe -> recipe.duration)) recipes

6. Group works like Sort, but further regroups « equal » items together by returning a list of lists. It works using a comparison function that is usually based on a projection function. For instance, to get three lists containing one-star, two-star and three-star recipes :

let by_stars =
  BatList.group (project compare (fun recipe -> recipe.stars)) recipes

There is a special case when the projection function returns booleans, which is known as a partition. For instance, to extract recipes that are desserts and recipes that are not :

let desserts, non_desserts =
  BatList.partition (fun recipe -> recipe.is_dessert) recipes

7. Search extracts one element from the list (if possible) based on a certain condition or property. Elementary searches are « first element » and « last element. » More complex searches : finding a value by key in a list of key-value pairs using List.assoc or using a predicate using List.find. The heavy-duty search tool is BatList.find_map, used below to find a recipe that is recommended by at least one person, and the recommending person :

let recipe, recommender = BatList.find_map
  (fun recipe -> BatOption.map (fun person -> recipe, person) recipe.recommended)
  recipes

These seven patterns can be used in conjunction to perform almost any algorithm on collections, sequences or lists. For a more complex example, assume we need the ten ingredients that appear the most often in desserts. We would filter recipes by desserts, extract their ingredients, group them by name, sort the sub-lists by list length and take the name of the first element of the first ten sub-lists :

open BatPervasives (* for operator |> *)
open BatList

let ten_best_ingredients recipes = recipes
  |> filter (fun r -> r.dessert)
  |> map (fun r -> r.ingredients) |> concat
  |> group (project compare (fun i -> i.name))
  |> sort (descending (project compare length))
  |> take 10
  |> map hd
  |> map (fun i -> i.name)

Article © brewbooks — Flickr

If your job is anywhere as hectic as mine, you will have to fix and deploy urgent patches, even when your application code is in a half-written, half-debugged state because of the feature of the month.

This is what branches are for. You keep two versions of the code, one of which is called the trunk and is always ready for deployment, and another which holds the changes that you are working on.

When your feature is done, you merge the two versions together. You want to keep the merge operation painless. To do so, you have several kinds of branches available.

The repository branch is built into your SourceSafe/subversion/git/whatever. It creates two independent copies, and you need to migrate changes from the trunk to every branch out there as soon as possible, or the merge will make you wish for a sweet and merciful death.

By the way, changeset-oriented tools (like git or mercurial) make this easer, while revision-oriented tools (like subversion) make it harder.

The feature branch is done using programming logic. The code you deploy to production supports the new feature, but it is turned off for everyone except yourself. This technique is great for adding features, but inefficient when changing existing ones.

A side effect of the feature branch is that you can stress-test new code by rolling it out to increasing numbers of users progressively.

The comment branch is an odd gambit. It involves ripping out an entire module and replacing it with another that has a different interface. This will involve large amounts of re-wiring all over the code base, and these will take hours or days before they can be compiled, let alone tested.

Use a comment structure such as this one:

/*[*/ old code /*|* new code *]*/

It is trivial to build a text-replacement macro that turns the above into the code below and back:

/*[* old code *|*/ new code /*]*/

Use the macro to switch between development mode (when you write new code and desperately try to get it to compile) and fix mode (when you edit the old code and deploy it). For consistency, always commit the old version to the repository.

Why use comment branches instead of repository branches ? Maybe your source control tool sucks at branches. I use Subversion. Yes, I know. Legacy, pain and unlikely hopes of a brighter future.

When a trunk change occurs in a part that has been erased or reworked in the branch, that change will cause a conflict that will require manual intervention. Even with git or mercurial. For a large number of small changes sprinkled over a large codebase that is routinely involving many small updates, repository branches turn into a merge minefield.

Does your branch involve a small number of well-defined files ?

Then you should use repository branches, because conflicts will only happen in those files, and will usually be easy to fix.

Does your branch involve many changes in many files everywhere in the project ?

Then use comment branches.

Last and possibly least, there is the TODO-branch. This involves non-breaking, purely cosmetic changes. 25% of my project uses this syntax for historical reasons:

Table.get id |-> function
   | None       -> return 0
   | Some value -> return value.count

Then, a convention change happened, and this is used instead:

let! value_opt = breathe (Table.get id) in
match value_opt with  
   | None       -> return 0
   | Some value -> return value.count

Then, another convention change happened, and this should be used instead

let! value = breathe_req_or (return 0) (Table.get id) in
return value.count

And then, there’s the current version:

let! value = breathe_req_or (return 0) $ Table.get id in
return value.count

Whenever I change coding conventions, I do not spend the time to reformat the tens of thousands of lines of code in my application. That would have been wasteful. Instead, every time a piece of code is refactored, it is refactored to the most recent style.

The same happens when using an old and a new version of a given API. My code uses two libraries for handling HTML forms, uses both Javascript and Coffeescript, and a variety of similar two-hammers-one-nail situations.

These are, for all practical purposes, branches. They are work that is being performed for long durations. The benefit of TODO-branches is that code in the middle of such changes is still compatible with the trunk. It all happens in the head of the developer, who remembers what changes should be done the next time a piece of code is rewritten.

Article Image © Dominic Alves — Flickr

We negotiate several times a day, even if we do not recognize those occurences as such — « accept or walk away » is a fairly instinctive, if somewhat inefficient, negotiation strategy, and it goes hand in hand with the modern « take it or leave it » strategy employed by most mass consumer shops.

What is negotiation anyway, and why does it happen?

It happens because when humans cooperate, they get a cake that is bigger than the sum of its parts, and they need to decide how to split it. For instance, if I grow potatoes and you breed cattle, we could decide to put together my delicious Désirée tubers and your tasty Charolais steaks, which is a big improvement over otherwise potato- or beef-only diets. And then, we would argue over how much of the final mix each of us would get:

ME: We both contributed equally, so let’s get 50% each.
YOU: Breeding cattle is harder than watching potatoes grow, so I want 65%.
ME: Suit yourself, I know a guy who only asks for 55%.
YOU: Fine, I’ll go as low as 60% because I like your blog.
ME: I just told you I know a guy who…
YOU: …and that would save you the effort of finding that guy before lunch.
ME: Right. 60% it is, then.

You were a good negotiator, so you managed to squeeze 10% out of this. What happened? Let’s break this down.

Irrational Arguments

« We both contributed equally » might be true. Maybe I spent as much time growing my potatoes than you spent working in your ranch. But that is not my argument. What I have written between the lines is that it is fair for equal efforts to receive equal rewards.

And fairness is not a rational argument. It is a subtle way to circumvent rational thinking and aim for our unconscious ethical programming and, by definition, make decisions that are not in our best interest. In the eyes of a good negotiator, this is a loophole to be exploited.

There are many other such loopholes in the human mind. My favorites are:

Reciprocity is an atavistic impulse to give back whenever we receive something, even if that something is mostly unrelated to our objectives. In the example above, going from 65% to 60% is cleverly framed as a gift instead of a simple acceptance of my terms, which makes me unwilling to further haggle over whether it should be 57.5%.

Imprinting is our tendency, when we have no idea how much something is worth, to go with the first guess, clue or hint that we find about it. By suggesting a 50-50 trade, I imprinted you on the idea that potatoes are about as valuable than meat, and this serves as a basis for your « reasonable » 35-65 counter-proposal instead an extreme trade like, say, 3-97.

Guess what the market rate for potatoes and Charolais steak is? 3-97. I would like to correct my previous sentence:

You were a good bad negotiator, so you I managed to squeeze 10% 37% out of this.

And then, there’s a slew of emotional manipulation that does not appear at all in my example above. You can feign disappointment to try and get more out of a deal — “I really hoped you, of all people, could get this done for me.” You can pretend to be offended, shocked or angry in order to cause a large shift in the terms being negotiated — when you haggle in a bazaar, quote a price that is too low and you will be told that it is insulting, and kicked out of the store.

There are many other examples. Go read this blog article and read Predictably Irrational by Dan Ariely.

Rational arguments

You might expect this section to be as rich as the previous one, but it is not. There are only two different rational arguments you can say in a negotiation: “I will  not accept this outcome because [credible reason]” and “You will accept this outcome because [credible reason]“. Let’s examine our conversation with this convention:

ME: Suggests 50-50
YOU: No, because [breeding cattle is harder than watching potatoes grow], suggests 35-65
ME: No, because [I know a guy who only asks for 55%]
YOU: Suggests 40-60
ME: No, because [I know a guy who only asks for 55%]
YOU: Yes, because [that would save you the effort of finding that guy before lunch]
ME: Accepts 40-60

This is the reason why the most fundamental principle in negotiations is be willing to walk away. If you cannot or will not walk away, and the other guy knows this, then you can no longer say « No, because » and this cripples your ability to negotiate properly.

Typical reasons why I will not accept an outcome:

• The costs are greater than what I get out of it. Maybe I’ll ask for more money, or non-financial benefits (exclusive rights, free advertising…)
• It’s too risky for me. Maybe I’ll ask for insurance, or add terms to our contract that provide me with acceptable protection.
• I can get a better offer elsewhere. You can either match that offer, give me something I cannot get elsewhere, or give up.

Typical reasons why you should accept an outcome:

• There are additional benefits you did not take into consideration, such as not spending the time to find a better offer elsewhere, or my goodwill in terms of future trades, or my reputation as your client/provider.
• If you do not accept now but change your mind later, I will not be able the same offer to you again in the future.

These reasons do not have to be true, they only have to believable. I don’t know anyone who would accept a 45-55 split on potatoes and Charolais steak, but you don’t know that, so I can still bluff my way through our negotiation, bringing you down from 65% (which is still a 32% win for me) to 60%. If you knew that no sane person would accept a 45-55 split, you could certainly call my bluff and even bring me back up to 10-90 or so.

How you present those arguments is a matter of style. Here are four different ways of presenting the same argument (you will not get more than $30k for an APL job) :

1. No one else in town hires APL developers like yourself, so you either accept this $30k job or get out.
2. I am certainly interested in your APL skills, but I would be losing money if I paid you more than $30k.
3. We are looking to fill our APL development position, but I am not at liberty to offer you more than $30k.
4. We have interviewed several candidates with your skills, and most of them are below your wage requirements.

Some aggressive types would prefer approach 1, while manipulator types would go with 2 (which is a lie), passive types would hide behind a third party authority using 3, and sneaky types would try to get an even lower salary quote by withholding the $30k figure in approach 4.

Alternative Benefits

This is a subtle but essential part of many successful negotiations. It’s not only about the money. In the above example, I traded 5% for the ability to get my food in time for lunch — a concept that had not been mentioned at all before. You decided that instead of matching my requirements as I expressed them, you would offer me something that was unexpected and which, to you, cost less than 5% but still, to me, was worth at least 5%.

Going down alternate routes is an excellent way to get a stuck negotiation moving.

Article Image © William Neuheisel — Flickr

1. Start with an arbitrary solution.
2. Mutate the solution : if the mutated solution is better than the current one in terms of fitness, keep it and repeat this step.
3. If you discard too many mutated solutions in step 2, remember the current solution and mutate it.
4. When time runs out, return the best solution found so far.

Step 2 is a fairly standard ascent towards a local maximum : try a random similar solution and keep it if it improves fitness. Step 3 detects that the algorithm is stuck in a local maximum, so it shakes the solution out of there and starts again from that new position.

What is interesting about this solution is that it is exceedingly easy to implement, easy to adapt to a problem (as long as mutation and fitness are possible), and is reasonably efficient for most small problems.

I used it today to solve an original problem on RunOrg. We have a history of recently connected users, and we wanted to display a simple metric along the lines of « 10 users connected in the last hour » or « 11 users connected this week » as a short summary on the administrators’ dashboard. The problem, of course, is to pick a reasonable duration based on the current data ! If one user connected seven days ago, and ten users connected a few minutes ago, then both « 10 users connected in the last hour » and « 11 users connected this week » are correct descriptions of the situation, but the former is intuitively better than the latter. Intuitively, we should use the shortest duration with the largest amount of users.

And intuition is a bad thing, because it is hard to turn into an algorithm.

In the end, I settled for a handful of periods of time : the last 5 minutes, the last 1 to 6 hours, today, the last 2 or 3 days, and this week. Each of these periods of time contains a certain number of user connections (which is fairly easy to determine based on the connection history) and receives a score computed by multiplying the number of user connections with a magic constant that is tied to that period. So, if the magic constant for « the last hour » is 0.76 and that for « this week » is 0.03, then 10 × 0.76 = 7.6 and 11 × 0.03 = 0.33 meaning that the « 10 users connected in the last hour » version will be preferred.

Then, I came up with several examples of connection histories, and manually picked the intuitive duration that should be chosen to display them. This turned into the fitness function : any set of magic numbers would have to agree with as many of those examples as possible, the more the better. Mutation, on the other hand, consisted in changing one of the magic constants at random.

Within 2000 iterations and a few seconds, the annealing algorithm came up with a solution that satisfied 94% of my constraints — and I am not even certain whether 100% could be satisfied, since I might have left a few contradictory intuitions in there. The final output is:

== Fitness: 94.44%
5min : 1
1h : 0.761091036722064
2h : 0.42295812451487896
3h : 0.26899204507851066
4h : 0.20923702855563558
5h : 0.14858146007043194
6h : 0.14846461447836096
today : 0.1477566411417444
2days : 0.07381054881381345
3days : 0.04202811963812667
week : 0.033268828300871064

And so, these are the magic numbers that our algorithm uses. If it makes wrong decisions, we can always run the annealing again with new constraints, and feed the magic numbers back in.

Article image © Detlef Schobert — Flickr