Building Homes and Building Software

"It cannot be created by decision. It cannot be designed. It
cannot be predicted in a plan. It is the living testament of hundreds
and thousands of people, making their own lives and all their inner
forces manifest. And, finally, the whole emerges."

--Christopher Alexander, The Timeless Way of Building.

My wife and I recently had a house built, yielding me the
opportunity to contemplate the patterns present in both home building
and software development. In general, we are pleased with the
finished house. However, along the way, various problems arose, some
of which have still not been resolved. How did we get some aspects of
the house right? What caused problems with other aspects? Could a
different process have prevented them? And do the failures and
successes teach us something about the construction of software?

Architect Christopher Alexander discovered the power of pattern languages, which attempt to describe essential forms that repeat themselves, with endless variety, in a large number of settings. An example of a pattern from the world of architecture is "Entrance Transition," which states that the psychological movement from the public space of the street to the private one of a home should be reflected in the construction of the entryway to a house. The idea of patterns is finding application in the creation of software, especially in the area of Object-Oriented
Design. (For more information on software patterns, see the Patterns Home Page or the "official unofficial" Pattern FAQ page. Each of these contains many more references.)

I will present some patterns that I've found are common to
both constructing a house and building a piece of software. Each
pattern I describe illustrates that it is not enough to merely follow
the right "methodology" when constructing a complex artifact like a
house or a piece of software. Personal involvement on the part of the
participants and real communication between them are necessary as
well.

The defining elements of a pattern are a name for the pattern, a
description of the inner structure of the pattern, the conflicting
forces it attempts to resolve, and the context(s) in which it is
useful. Since the context is, in most of the patterns below, the
same, I will say more only when there is something salient to note
about a particular pattern's context. I use the term "worker" below
to designate nonmanagerial project participants, since I don't know
of any other common term that would denote equally well a programmer,
a carpenter, a laborer, a technical writer, etc.

The general context for the following patterns are:


A construction (in the broadest sense of the term) project that
will last at least a few months.
A specialization of work. In software, a project may involve
programmers, analysts, DBAs, system administrators, testers, technical
writers, project managers, and more. Our house construction involved
an architect, a general contractor, carpenters, plumbers,
electricians, sheet rockers, painters, masons, floor installers,
tilers, a backhoe operator, cabinet installers, wood stove installers,
and others whom I've forgotten.
A consumer of the product who does not possess technical
expertise in the construction domain.

The Patterns...

Name: Organization Follows Location


When the cabinet installers arrived at our site to start work, we
had a pipe sticking out of our kitchen floor in the middle of where a
cabinet was supposed to go. This would not have occurred if the crew
placing the pipes had been in communication with the crew installing
the cabinets. A friend of mine, who is a carpenter, succinctly
expressed a common attitude about this type of communication: "Hey,
it's not my job to talk to anybody. I just nail up stuff where they
tell me to."

Workers often feel responsible for doing only what they are told.
This may entail nonsense: carpenters hanging closet doors swinging
inward, so that they render the closet unusable (yes, this happened to
us!), or programers building a "daily" load process that can't
possibly run in less than fifty hours on the available hardware.
Formal meetings may have some value in alleviating this, but too
often, workers are reluctant to speak up at these meetings, for fear
of looking stupid or antagonizing a boss. Since nothing of substance
is said, workers feel that the meetings are a waste of time, keeping
them from getting their work done. But workers don't hesitate to talk
shop during lunch and coffee breaks, and will chat with each other
while working, if they can see each other.


To state the problem more generally, we can say that systems
integration is hampered by the weak communication links between teams.
Just as the QA department tends to step in at the very end of
development to start testing, and communicates with developers only by
a stream of bug reports, carpenters and electricians tend not to talk
to each other much. The end result is subsystems that do not mesh
into a larger system.


The pattern that resolves this problem is to keep people working in
related areas of a project physically close to each other.
(James Coplien of Bell Labs named and described this pattern.)
In the software world, this means putting
workers in nearby offices that share common areas such as lunchrooms,
conference rooms, bathrooms (yes, useful thoughts are shared even in a
bathroom!), and copy/fax/mail rooms. The people who are going to test
and document a product should sit near the people who are building it.
On a house-construction site the chief barrier to proximity is time:
it is not unusual for workers building the same area of a house
not to meet during the course of the project. This pattern would
place workers in the house at overlapping times. Ideally, they would
even come to a site several times before they start their job.


This pattern should, if possible, be used in conjunction with
Alexander's pattern "Common Areas at the Heart." For instance, a
centrally located lunch area will increase the chance it is used, as
well as leading people to walk through it even when they are not going
to eat.


This pattern would not apply in cases where someone's job is
completely defined in advance, and the work is messy, dangerous,
requires extreme concentration, etc. For instance, the person who
puts the finish on hardwood floors should work alone. The fumes are
unpleasant, the floor cannot be walked on for several days after being
finished, and it's hard to imagine how the floor would be finished
differently due to a conversation the finisher has with a plumber or
electrician. A software context where the pattern doesn't apply is
when a programmer has to write a subsystem with an interface that is
simple and completely defined, but with inner workings that are
extremely complex. For the most part, this person is going to want
quiet time alone, in order to deeply penetrate those inner workings.
This is the perfect time to tell the programmer to work from home for
a week, or send him with a laptop to a quiet bed and breakfast by the
beach.


Name: Explanations Even a Layman Can Understand


Most specialists understand that a general description of what they
will do may be required. A framer may inform the customer, "Today,
we're going to frame the roof." But the framer may not go the next
step to tell the customer, "We're using trusses for the roof, because
they are prefabricated and will save framing time. Since we don't
need to leave a large access hole to the attic, they should be fine."
This detailed explanation gives the customer the chance to say, "Hey,
wait a second, I need to get a couch up there." Similarly, a
programmer can tell a user, "We chose not to store this data in our
RDBMS, because it rarely changes." Again, the customer has an
opportunity to say, "Rarely changes? We update that data all the
time."

When we built our house, the "technical" decision to use baseboard
heat effectively doubled our cost for central air conditioning. (The
ductwork for the air conditioning, rather than being included in the
cost of the heating system, was now an extra expense.) But this
decision and the trade-offs involved in it were never discussed with
us. Technical decisions should not be made in a vacuum. All such
decisions involve trade-offs (or there wouldn't be a decision), and
their effects will usually spread beyond any specialty's domain.

Many such decisions arise during the course of a project. What
wood, what programming language, how to frame, which OOP framework,
what level of electrical service, what CPU? The specialists must
communicate each important choice to the end user in a way that he can
understand it, and must give him a clear view of the trade-offs
involved in the choice.


Name: The Job Ain't Done Until a Demo Has Been Run


At our house, two of the lights behaved in a "less than optimal"
fashion when we turned them on. One immediately short-circuited. The
other was an outdoor light controlled by a motion detector. After
dark, the motion-activated light should have stayed on at least
long enough to unload a car and walk the path to the front door.
Jesse Owens could not have made it to the door in the time this light
stayed on, which was about two or three seconds.

Workers are generally confident of their own work. They feel they
know what they are doing, and they are probably correct. However, all
humans make mistakes, and often we make them because of something we
overlooked. It follows that we often can't see our own mistakes,
because it is an occlusion that led to them in the first place.
Having put in ten thousand lights, an electrician may be so confident
in his ability that he no longer bothers to turn them on when done!
Or having written fifty linked lists, a programmer can neglect
checking to see if an insertion at the beginning works properly.

These forces are resolved by instituting a rule that a worker
cannot call a job done until he can demonstrate to an outside party (a
manager, architect, owner, business sponsor, coworker, etc.) that the
parts he constructed work properly. It is not enough for him to test
the job himself; it must be demonstrated to someone else.

... And Their Meaning


The particular way Alexander formulates patterns is gaining
widespread acceptance in the software development community. But the
quote at the top of this page illustrates Alexander's deeper
philosophy, which motivated the search for pattern languages. This
philosophy is given much less attention in our industry than the
formal structure of patterns. (James Coplien, mentioned above,
Doug Lea, of SUNY Oswego, and other members of the Hillside Group
are notable exceptions in this regard.)

To quote Alexander, "A [pattern] language is a living language only
when each person in society, or in the town, [or on a software
project,] has his own version of this language. For it is then... a
deeper thing, a felt thing, a thing lived through...." The top-down
imposition of order, as is often recommended for software projects, is
bound to produce a lifeless piece of software. Top-down control of a
construction project produces a lifeless house. It is only through
the intimate involvement of those who must realize a design and those
who must live with it, and through both of their abilities to
continuously alter that design as the context of its construction
evolves, that a piece of software or a building can come to life.

Let's consider some examples of "living" versus "dead" software.
Much of the most exciting, context-altering software has come from the
"disorderly" environments of free software and open systems.
"Chaotic, fragmented" UNIX was the chief platform for developing
remote procedure calls, tool-based development with pipes and filters,
TCP/IP, C, C++, Perl, networked GUIs, USENET, SMTP mail, the Web,
shared libraries, IPCs, regular-expression-based processing of files,
kernel-based OSes, and more. The Web exploded into corporations with a
frenzy of creative efforts, often giving users their most satisfying
corporate software experience ever. HTML, Linux, Perl and Apache meld
together beautifully, yielding what many webmasters feel is the most
powerful, flexible web platform available. Yet, they are not the
products of any master plan to produce this platform, nor even of a
master plan to produce the current version of the individual tools.
Each is the result of an organic growth process where the presence of
the other tools, and the input of many thousands of developers and
users, subtly changed them. Users of these tools are excited about
using them, and excited about their rapid evolution. Many corporate
users of free software are claiming they've switched from commercial
packages because freeware has better support and quicker bug fixes, a
strength exactly where commercial vendors said there would be
weakness.

Contrast this feeling with that produced by the accounts receivable
package of a typical large corporation. Users dread using these
systems, and programmers attempt to dodge working on them. The
paradigm of a dead-end technology job (at least before the Y2K
brouhaha) was a Cobol maintenance programmer patching a
thirty-year-old accounts receivable program. This is a system built
on top-down control, with a form to fill out to begin work on a file,
a form to describe the work you've done, a form detailing how you've
spent each hour of the day, a form to have a test run, and a form to
explain what went wrong. None of the forms are ever read, as everyone
else is busy generating his own. For every five minutes of creative
work, a programmer may be forced to spend an hour in bureaucratic
drudgery.

IT departments are terrified by the loss of control abandoning
top-down construction seems to entail. But the "control" IT
departments have today is just an illusion. You can enforce coding
standards, but unless these standards come from everywhere, from the
response of the developers to the nature of the project itself, the
standards will be empty of any meaningful measure of quality. It is
not hard to write garbage that doesn't solve a user's problem, while
maintaining strict adherence to the letter of a standard. Metrics can
be rigorously employed, but if the programmers haven't bought into
them, you'll just get more lines of code, or more decision points, or
whatever else you've decided to measure "productivity" by. You can
demand unit testing, integration testing, user sign-off, and as many
other tests as you want, but unless those doing the work are in good
communication with each other you will still wind up with buggy
software. This "control" works in one direction: It controls
development from being done too well, or too quickly, from being too
responsive to users. But it can never control people to produce
quality work rapidly, because for this they must be operating on their
own impulse, with their own intelligence fully engaged.

Only when people are "living" the design will the design come
alive. The extent to which we succeeded with our house is due to our
being there almost every day, interacting with the builders, feeling
what it would be like to live there. Almost without exception, our
failures were in areas where we abdicated responsibility, and let the
experts, who only had to be in the house for a short time, make
decisions for us. If you reflect on the successful software projects
you have been involved in, and the failures, I think you may find this
same truth. Having experienced the timeless way of building for
ourselves, we can then infuse our software with new life, and everyone
-- the programmers, the managers, the testers, the writers, and,
especially, the end user -- will be the better for it.

Related web sites


Christopher Alexander: An Introduction for Object-Oriented Designers
Patterns Home Page

Patterns Discussion FAQ
James Coplien's homepage
Doug Lea's homepage
Hillside Group


Dr. Dobb's Journal 1998 Excellence in Programming Awards
(Dr. Dobb's Journal, March, 1998)

About Dr. Dobbs Journals Online Op/Eds

Periodically, we publish op/eds covering all aspects of software
-- from design issues to industry insights -- on our web site. If you
have questions or comments, or are interested in contributing a piece,
send e-mail to eekim@ddj.com.

Dr. Dobb's Web Site Home Page --