I’ve updated the software running this blog. As a result, there were a few changes:

• I broke all of the permalinks to posts older than this one. Sorry about that - the search box above will help to find old posts if you’re looking for something in particular.
• Comments made to previous entries have not been not migrated over. In many cases, I updated the entry based on feedback from commenters. Thanks to everyone who’s commented on previous posts!
• I closed comments on all of the old entries.
• A few old entries that weren’t very good or contained outdated information have been deleted.

The old feed URLs should be automatically redirected to the new ones. If for some reason your subscription to this blog isn’t working, please update the feed URL in your feed reader with this one:

http://benpryor.com/blog/feed/

Thanks for reading!

When designing a class for subclassing, it’s important to avoid calling any method that the subclass could or must override from the superclass constructor. This includes any non-final public or protected methods (either abstract or concrete). The only methods that are appropriate to call from the superclass constructor are final methods or private methods in the superclass.

Unfortunately, this mistake is all too easy to make - I’ve made it myself lots of times. Usually, this is done because the superclass has some invariant that must be enforced, and part of enforcing the invariant involves calling at least one method that the subclass could or must override.

For example, consider the following contrived class:

class SuperClass {
    private final FooBar fooBar;
    private String message;

    public SuperClass(FooBar x) {
        fooBar = x;
        message = computeMessage(x);
    }

    protected String computeMessage(FooBar x) {
        return x.someMethod() + "..."; // details of the message are not important for this example
    }
}

This class exhibits the problem this entry describes. Class ‘SuperClass’ is designed for subclassing, but it calls an abstract method from its constructor. The intention of this class is to enforce the invariant that the ‘message’ field will never be null after construction, and computing the correct value for the ‘message’ field involves the subclass. Even though this is a contrived example, many needs for calling subclass methods from a superclass constructor have this form in essence.

The reason this is bad is because in Java, superclass constructors run before subclass constructors (that is, a subclass must invoke super() before other statements in its constructors). By invoking a subclass method, code in the subclass will be invoked before the subclass gets a chance to initialize itself. The subclass will see default values for all its fields, and will not have access to any data passed in the subclass constructor.

For example, consider the following subclass of the class given above:

class SubClass extends SuperClass {
    private MessageHelper helper;

    public SubClass(FooBar x, MessageHelper helper) {
        super(x);
        this.helper = helper;
    }

    protected String computeMessage(FooBar x) {
        return helper.getMessage(x); // NullPointerException!
    }
}

In this example, the subclass takes a helper object in its constructor. The helper object is used in the overridden abstract method. However, because the superclass constructor calls the abstract method, the subclass does not have a chance to assign the helper object to a field (its constructor has not run yet). This requires the subclass to either guard for this condition (but what should it do when the condition happens?) or contain a NPE bug as in the example above.

Fortunately, this problem is easy to fix with the following design pattern. First, observe that there are two issues at play here: (1) enforcing an invariant and (2) designing for subclassing. The best way to satisfy both issues is to make the superclass abstract and require (through documentation) that all subclasses enforce the invariant in their constructor.

For example, here is a fixed version of ‘SuperClass’ and ‘SubClass’ from above that fix the problem:

abstract class SuperClass {
    private final FooBar fooBar;
    private String message;

    protected A(FooBar x) {
        fooBar = x;
    }

    protected final void initialize() {
        message = computeMessage(fooBar);
    }

    protected abstract String computeMessage(FooBar x);
}

class SubClass extends SuperClass {
    private MessageHelper helper;

    public SubClass(FooBar x, MessageHelper helper) {
        super(x);
        this.helper = helper;
        initialize(); // subclass MUST call initialize
    }

    protected String computeMessage(FooBar x) {
        return helper.getMessage(x); // no NullPointerException
    }
}

The superclass is made abstract to avoid the possibility of creating an instance in which the invariant is not satisfied. This is OK, since a fundamental part of this design is to allow for subclassing. In addition, all subclasses must call the protected initialize() method from their subclass constructor. If they fail to do so, the superclass-subclass contract is broken and invariants can’t be guaranteed.

The fix may be considered a little cumbersome. It requires the creation of at least one other class (since before the fix, the superclass could be used directly). The fix also requires (but can’t enforce through the compiler) that subclasses all follow certain rules. However, these tradeoffs are much better that having the situation in which subclass code is invoked before the subclass can initialize itself. By the way, I am pretty sure this issue is also discussed in Effective Java (I’d be sure, but I don’t have a copy handy at the moment).

Comment summary:

One commenter asked: Does this relate to Bug 6596304? http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6596304

I replied:

Yep, that issue has the same root cause. It’s a particularly insidious example of how things can go wrong when this guideline is broken.

If you compile and disassemble the code provided in that bug, it’s pretty easy to see what’s going on. The Child class’s constructor contains bytecode to set all of Child’s fields to their default values (e.g. aconst_null, putfield). This code runs after the code that sets the Child’s field value in the overridden init() method. It undoes the set to the field that happened when the superclass constructor called the overridden method.

The moral of the story is that calling any method that can be overridden from a superclass constructor is a dodgy practice and should be avoided.

Thanks for the link to the bug - that makes a great example for explaining this issue.

Eric Wilson said: C# has the same rules for constructor execution, so it can also be subject to the same issue.

Twice within the last week, I’ve found an easily preventable bug in my code. The problem, in both cases, was that I put a TODO comment in the code, fully intending to come back later and fix up whatever issue I put the TODO in for. If you’ve developed software for any amount of time, you know the rest of the story. I never went back in, completely forgot about the TODOs, and blissfully shipped the code. Over a year later, my forgotten TODOs came back to bite me in the form of bugs. Luckily for me in both cases the bugs were very minor - but still quite annoying.

TODOs in code should be considered harmful. They’re bugs waiting to happen, and they’re so easy to forget about. There’s a sort of pathology I see a lot in code in which the author hits a particularly sticky problem (or perhaps just something they can’t be bothered to do) and they just slap in a big TODO comment. For me, no more TODOs. They’re not visible enough, not noticeable enough, and they’re a lame excuse for not doing the right thing in the first place.

There are a couple of better alternatives to TODO comments. The first is to instead file a new bug against yourself to fix whatever issue you would have written the TODO for. This has much better visibility than TODOs. It also allows you to estimate, prioritize, and schedule the needed work. Depending on your tools, doing this will range from cumbersome to elegant. A very nice IDE/issue tracking integration feature would be to allow for creating new issues from within the IDE, in the context of a file that you’re editing.

Another alternative is to write some quick, naive code that is observably incorrect. Again, this has much better visibility than TODOs, since anyone using the software should be able to tell at a glance that things aren’t right. Throwing an exception would fit in this category, as long as the exception is observable to a user/tester of the software. However, in most cases you want to write code that actually works, but has some behavior that reminds everyone that additional work still needs to be done before the feature is complete. This allows testing / other work to progress, but doesn’t allow the issue to fall off the radar.

If you do testing or QA, here’s an easy way to find bugs. Take a look through your codebase for TODO comments. For each comment you find, determine if the comment is 6 months or older, if the author of the comment is no longer with the team, etc. For those comments that have fallen by the wayside, there’s an excellent chance you’ll uncover some bugs around them. Personally, as a developer, I am going to be much more disciplined about not using TODO comments in my own work.

Further reading:
Point: http://c2.com/cgi/wiki?TodoCommentsConsideredHarmful
Counterpoint: http://c2.com/cgi/wiki?TodoCommentsConsideredUseful

In the fictional world of Star Trek, the Prime Directive is a principle at the heart of Starfleet and the Federation. This principle says that developing species shouldn’t be interacted with until their development reaches a point at which their species can interact with the Federation as equals. It is a principle that overrides all others, and violating it is the highest offense.

I hereby propose a Prime Directive for software development. This simple principle should be followed for all of the development you do:

Never engage in speculative development.

Speculative development is when you go off and spend significant time writing code that does not satisfy a known need, or that does not add value to the business you are writing code for. Unfortunately, this practice is all too common in the software industry. I’ve been guilty of it myself many times, and I’ve seen it happen on every single project I’ve worked on.

If you are writing code in the employ of a company, speculative development is not ethical. The reason is that someone is paying you to add value to their business, and speculative development does not add value. In fact, speculative development often subtracts value, since it creates more code, the potential for more bugs, and can create higher maintenance costs for a product.

There are several common manifestations of speculative development. Perhaps the most well-known is premature optimization - a well-known phrase made popular by Donald Knuth. When you spend time writing complex code to satisfy unknown or imaginary performance requirements, you’re engaging in speculative development. For most business and shrinkwrap software being written today, performance is not the primary concern. Even when performance is a concern, no software developer can intuitively know where the performance bottlenecks in their code lie. To avoid this kind of speculative development, follow this maxim: first make the code correct, and then, only if necessary, make it faster.

Another common example of speculative development is framework development. We’ve all seen this: you’re on a project, and someone on the team goes off and writes a framework to solve some problem orthogonal to the domain you’re writing code in. Most of us will admit that we’ve been that person on a number of occasions. As I’ve observed the framework writing phenomenon many times, I’ve come to realize a few things about it. First, it is often well-intentioned. We end up writing frameworks because we hate code duplication. Here’s the problem, though: we often end up spending more time writing the framework than we would have spent maintaining a small amount of duplicated code. Another reason framework writing happens so frequently is that it is interesting work. Most developers enjoy creating abstractions and coming up with new models for problems, especially when other work to do is boring and mundane. To avoid framework writing, consider whether a situation really warrants creating a framework. Often, you must accept a small amount of code duplication or impurity in your design - that’s just the nature of imperfect development environments. I promise your customers won’t know, and they will benefit more if you put that effort into other areas.

A third kind of speculative development that I see a lot is engaging in endless what-if scenarios when writing code. Don’t misunderstand - I’m not arguing against design or a thorough considering of edge cases. The difference is that you have to know when to stop. At some point, you have to decide what the boundaries of normal use for your application are. You’ll never be able to write code robust enough to cover all of the edge cases outside of those boundaries, and for most businesses, it’s not efficient to try to cover them. Filling your code with obscure edge case handling makes for very fragile and hard to maintain code. To avoid playing the what-if game, ask yourself how likely these edge scenarios really are. How will the business be impacted in the unlikely event that one of these scenarios occurs? Establish firm boundaries, and decide to let go of scenarios outside of them. Don’t make your software worse for 95% of your users in order to provide a hypothetically better experience for 5%.

The key to all of this is to think consciously about everything you do as you write code. With every piece of work, ask yourself if you’re adding value to your codebase, your business, or your problem domain. Ask yourself why you’re spending time on this particular activity and what the alternatives to it are. Analyze the alternatives, and investigate whether any of them add more value. If you get in the mindset of constantly evaluating your work and looking for ways to add value, the amount of speculative development you do will vastly decrease. Pretty soon, you too will agree that this is the Prime Directive of software development:

Never engage in speculative development.

When writing Java code, it’s useful to differentiate between two types of targets: a “normal” environment and a “managed” environment [1]. The difference between the two is simple. In a normal environment, you (the person writing the code) call the main() method [2]. In a managed environment, you do not. Managed environments are sometimes called container environments because they usually follow a containment or hosting model. In this model, the host container is the code that contains the main() method, and independent units of third-party code (hereafter plugins[3]) are managed by the container.

One of the most familiar examples of managed environments to most Java developers is the application server model used to host server-side Java applications. Such an application server can range from a full-blown JEE container implementation to a smaller-scale operation that hosts servlets with possibly a few EE features. Another example of a managed environment that most Java developers will have interacted with is the Eclipse IDE. Eclipse is entirely built around a container model (for the last few versions, that model has been OSGi). Users of Eclipse need not know or care about this, but anyone who writes plugins for Eclipse must understand the model and its implications.

It’s helpful to think about the differences between managed and normal environments. In theory, the environment should be mostly transparent to the code executing in that environment. In practice, however, there are some issues that need to be considered. This is especially true if you are writing library-level code: some libraries are much more “container-friendly” than others. If you are writing application-level code this may or may not be an important issue for you. Occasionally, you might have a single code base that you want to use in both types of environments.

The biggest difference between the two types of environments is the classloader layout. Normal Java environments are usually pretty simple in this regard [4]. Without going into unnecessary detail, this layout contains a bootstrap classloader that loads the system classes (like java.*) and an application classloader that loads everything else, including the classes that you write and any libraries those classes depend on. In contrast, a managed environment usually implies a more complicated classloader hierarchy. There will typically be a separate classloader for each plugin. There will often also be one or more classloaders in the hierarchy to support the container itself, and one or more classloaders that are shared among all of the plugins [5].

Why is the classloader layout important to think about? Two big reasons: visibility and static variables. Visibility is a simple property that has to do with the relationship between classloaders. Class A has visibility of class B if A’s classloader is able to load class B. Visibility becomes important in a variety of contexts. For example, many APIs will instantiate objects for you. In order to do this, they might need to have visibility of the class to be instantiated [6]. Static variables are associated with a class instead of an instance of that class, and are often used to obtain global variable semantics in Java. However, static variables aren’t really that similar to global variables - they do have a scope, and that scope is the class they are associated with. Such variables are only visible to classes that have visibility to the static variable class. If two different classloaders in the application each load a class that defines a static variable, there will be two instances of that variable. This situation may or may not be anticipated by the author of the class.

In the “normal” environment, it’s hard to make things go wrong classloader-wise. Any libraries your application code has dependencies on are pretty much guaranteed to have visibility of your code, since there’s essentially just one flat classloading space. Any static variables behave mostly like global variables for the same reason. In fact, you can pretty much ignore many classloader issues in a normal environment.

In a managed environment, there is much more potential for things to go wrong. As mentioned above, each plugin will have its own classloader. Most managed environments have complicated rules or configuration options that govern the relationships between the plugin classloaders, any shared classloaders, the container classloaders, and the bootstrap classloader. Since there is not a flat classloading space, visibility issues come into play. For instance, a library may not be able to instantiate an object because it does not have visibility to a classloader. A class may be loaded twice, and then both versions of the class may be attempted to be used in the same context (leading to very confusing ClassCastExceptions!). In short, all kinds of problems can happen, and debugging them can make for hours and hours of fun.

A particularly interesting problem that often occurs in managed environments is an insidious type of memory leak. Managed environments often provide for some amount of dynamic loading and unloading of plugins at runtime. This is most often implemented by discarding the plugin classloader when the plugin is unloaded. The plugin classloader contains references to all of the plugin classes, which in turn contain references to any static variables defined by the plugin. If all goes well, the garbage collector can reclaim the plugin classloader and everything referenced by it. However, it only takes one outside reference to a single object from the plugin to keep the classloader from being garbage collected. This is because every object has a reference to the classloader that loaded it. It turns out to be very non-trivial to cleanly unload a plugin at runtime - just perform a web search for “classloader memory leaks” to read all sorts of war stories. The Java language and libraries are just not designed for that kind of thing. Luckily, most managed environments do not have core functionality that depends on dynamic unloading [7].

Another problem occurs with the issue of configuration. Let me skip straight to an example: using Java system properties for configuration simply does not work in a managed environment [8]. System properties have global scope - when a system property is set, the value is read by every other part of the application. In particular, system properties span classloader visibility. Suppose a managed environment contains plugin A and plugin B, both of which depend on a library which is placed in a shared classloader space. Assume further that said library relies on system properties for configuration. This is bad because plugin A can now affect plugin B’s configuration.

So when writing Java code, whether a library or not, consider the target environment of your code. Managed environments are becoming more and more common. Managed environment techniques like custom classloaders that would have been considered advanced 5 years ago are now much more commonplace, which means you’re much more likely to run into them. Managed environments, long the standby of the server-side Java space, are now increasingly popular on the desktop. The Java community is starting to take a lot of interest in standardizing managed environments [9]. If you’re writing application-level code, understand which type of environment you’re targeting. If you’re writing a library, take the time to write container-friendly code.

Footnotes:

[1]
The term “managed” here means something different than another common use of the word: in the .NET world, the term “managed” refers to code that uses automatic memory management as opposed to manual allocation and freeing. That’s not the meaning I’m using in this article.

[2]
That is, the public static void main(String[] args) method. In other words, you control the entry point of the Java application.

[3]
Plugins may not be the best term, but “containees” or “independent third-party units of code” was just too unwieldy to use throughout the rest of the article. If you’re a server-side person, just substitute “web application” everywhere you see “plugin”.

[4]
But not always. As non-trivial applications grow, they almost always eventually start playing magic classloader tricks to support various features. At that point, they start to look a lot more like managed environments from the perspective of some of the code.

[5]
For example, see this document for a description of the classloader hierarchy in one managed environment.

[6]
Of course, a well-written API that does this will not require such visibility. Instead, it will either require that a classloader that does have visibility be passed, or it will make use of hacks such as the thread context classloader.

[7]
For instance, most JEE containers support hot redeploying of applications. This functionality is very useful during development but rarely enabled when the application is put into production.

[8]
Despite the fact that it does not work, many libraries do it anyway. You’ve been warned! :)

[9]
See OSGi, JSR 277, and JSR 291.

When designing an API, one of the considerations you usually have to address is concurrency. In other words, for every class in the API, what is the class’s threading policy? At the very minimum, an API should document how it behaves with regard to concurrent access, and even better is an API designed with concurrent access in mind.

Writing a good API that is concurrency-friendly is hard. More than anything, it requires lots of reasoning about how all of the moving pieces will work together under concurrent access. In this article I’m going to discuss a few simple guidelines you can follow when designing APIs and reasoning about concurrency. The specifics of what I’ll discuss apply to Java, although the general concepts probably apply equally well to .NET and other similar environments.

The goal is, in general, to choose the design that is the easist to reason about and still has acceptable concurrent performance and API usability. Here are four guidelines that can be used as a starting point when thinking about what the threading policy of a class should be:

1) For non-collection-oriented classes, default to immutable
2) For collection-oriented classes, default to mutable and thread safe
3) Prefer collections of immutable elements
4) For collections of mutable elements, make copies and treat the contained elements as immutable internally

These guidelines will help you to design classes that make it easier for the consumer of your API to reason about concurrency policy. This reasoning is made possible by clear documentation of each class’s design for concurrent access. Failure to consider the threading policy for a class leads to under-documented APIs, unexpected behavior at runtime, and obscure bugs that are hard to reproduce. Documenting the concurrency of an API is essential in order to guide the API’s consumers to correctly using the API.

Note that documenting an API as “not thread safe” is a valid design choice. It may not be the best choice (depending on the API). However, it is better to document that a class isn’t thread safe than to make the API consumer guess or rely on undocumented behavior that could change.

In the discussion below, I make a distinction between collection-oriented classes and non-collection-oriented classes. Collection-oriented classes contain multiple similar child objects (elements). They are usually easy to recognize since they often contain methods to add elements, remove elements, find elements, and perform similar operations across the contained elements. Collection-oriented classes shouldn’t be confused with composite classes that are made by composing together dissimilar classes.

Now I’ll go over each guideline. Remember that ultimately, we are looking at a class and trying to determine what a reasonable threading policy for that class might be.

For non-collection-oriented classes, default to immutable

Immutable classes are one of the best design choices you can make when designing an API. For classes that aren’t collection-oriented, default to using an immutable design.

Only design mutable classes if it seems that API consumers would be greatly inconvenienced by the immutable classes. Usually though, immutable non-collection-oriented classes aren’t a big hassle for users.

Immutable classes have the inherent property of being safe for simultaneous access from multiple threads. You don’t have to do any internal or external locking. This is the first big advantage of immutable classes - thread safety for free and no performance hit under concurrent access.

Another big advantage of immutable classes is that when you use them, you can more easily build a mental model of the system you’re designing. Since immutable classes are very easy to reason about, it takes less mental RAM to think about how instances will behave at runtime - they can only ever be in a single state (post construction).

Immutable classes are also a great way to set API consumer expectations. One of the most frustrating things about an API is when some class, say a service of some kind, takes a mutable object as initialization data. What happens after you hand the mutable configuration data off to the service? Are you still allowed to modify the configuration data? Does the service care? Will it break? Using an immutable class design for the configuration data instead solves these problems.

Designing immutable classes is extremely easy in Java. Briefly, here are the requirements you should satisfy when designing a class in order to call it immutable:

1) All fields in the class should be declared final
2) All fields should be either immutable objects or mutable objects that are not mutated outside of a constructor
3) The this reference does not escape a constructor
4) No mutable objects passed to a constructor are retained by the instance or any of its components
5) No mutable objects escape the instance

Satisfying these requirements is not the only way to design a thread safe immutable object, but other techniques are much harder to explain and require deep knowledge of Java’s memory model.

For collection-oriented classes, default to mutable and thread safe

When designing a collection-oriented class, the default choice should be to write a mutable container that can be safely accessed by multiple threads concurrently. Most collection-oriented classes should be mutable since that’s what the API consumer will expect. When using a collection-oriented class, the most common reason is because you want to modify the collection (add or remove elements) or modify the contained elements themselves. An immutable collection-oriented object makes consumers jump through hoops to do this.

Often, collection-oriented classes in an API will be used only from a single thread for many scenarios. It can be tempting not perform the necessary locking to make these classes thread safe, since that locking will be unnecessary for the majority of usage scenarios. In this case, one valid design decision would be to skip the locking and document the collection as not being thread safe. However, doing this penalizes the users of the class who are in concurrent-access scenarios, since the burden of implementing thread safety is now on them.

I recommend going ahead and doing the internal locking to make these types of classes thread safe. On modern Java runtimes, the cost of uncontended synchronization is extremely low (JCIP talks about this in detail). Even when the common case is single-thread use, it’s better to design a class to be thread safe if there are potential concurrent scenarios. If a profile reveals that the locking is a hotspot when using the class, then you have a good reason to avoid it. Otherwise, assume that locks are essentially free when uncontended.

Although it’s not the first choice I’d make, for some APIs immutable collection-oriented classes may not be a bad decision. You get all of the advantages mentioned above for immutable classes. If API consumers will not often need to mutate the collection or the collection’s elements, doing this may make sense. For example, some collection-oriented objects are often just passed around to other parts of the API, and rarely changed. If you decide to design an immutable collection-oriented class, be sure to document this very explicitly. Also, avoid method names that imply that the receiver is being changed as they can be confusing to a casual user of the API. An immutable collection-oriented class should not have a “public void add(Foo f)” method since the signature of that method implies that it alters the receiver.

Prefer collections of immutable elements

Whenever possible, collection-oriented classes should contain immutable elements. This helps reduce confusion about the API, since it is clear that the state of the elements can’t be changed while the collection contains them. I’ve found that designs that use small, immutable objects as building blocks and contain them in mutable containers tend to be very robust and easy to understand.

The biggest reason this is important is that most non-trivial collection-oriented classes have one or more invariants that they must enforce. For instance, a collection-oriented class may store elements that each have an identifier of some sort, and the collection may guarantee that contained elements will have unique identifiers. This is just an example - often the constraints can be much more complicated. If the contained elements are externally mutable, it will be very hard or even impossible for the collection to enforce those invariants.

Mutable collections of immutable objects are also very fast to make copies of. Since the contained elements are immutable, copying the collection only involves making a shallow copy. Supporting copies of collections is important for many APIs, so anything that makes this easier and faster is a win.

For collections of mutable elements, make copies and treat the contained elements as immutable internally

Unfortunately, it’s not always possible to design using only collections of immutable elements. Often, for one reason or another, the collection must contain mutable elements. One case of this is a collection-oriented class where the elements themselves are collection-oriented. No matter the specifics, there is a design pattern you can follow in this case.

The collection should make copies of the mutable elements as they are added or retrieved, ensuring that no external clients have a reference to the actual contained instances. In other words, every time a mutable element is added to the collection, the collection makes a copy and adds the copy instead. Every time a mutable element would be obtained, a copy is obtained instead. Internally, the collection should treat the contained elements as though they were immutable and should not call any method on the elements that could change them. By doing this, the collection will contain “effectively immutable” elements. The collection is then free to enforce constraints on the contained elements and know that no external client can break the constraints.

I can attest that this kind of design works well, but it does require some careful API documentation. Without proper documentation, clients may expect that they can obtain a contained element, mutate it, and have those changes automatically show up in the collection. Instead, this kind of design facilitates a more transactional usage. Clients obtain an instance, perform some changes to it, and then must add that instance back in to the collection. This usually isn’t a huge burden on the clients as long as expectations are set correctly.

That covers the 4 guidelines. Note that these guidelines are really meant to only cover simple cases - however, the simple cases make up the bulk of most APIs. There are certainly complex classes that don’t fall easily into one of the categories above, and they will need to be designed with more thought.

In today’s global software industry, character encoding issues are frequently encountered on almost all software projects. One factor that causes trouble when discussing and solving character encoding issues is terminology. In order for developers to share problems and successes with each other, a generally agreed-upon set of terms is needed. Another problem is that different character encoding standards work differently. It can often be useful to abstract away the details and talk in general about how character encoding standards work. By doing this, it can be easier to understand the mechanics of a specific character encoding by fitting it into a more general framework.

You might be familiar with the OSI seven layer model for describing network protocols. Many computer science and software engineering education programs teach this model (e.g., in an introductory networking class). This model is useful because it abstracts away the specific details of particular network protocols and provides a generalized stack of layers that all protocols can be thought of as having.

A similar abstract layered model can be used to define and discuss character encoding standards. In this article, I’ll define and explain a layered model for character encoding. By learning this model, you’ll be able to more easily diagnose and solve character encoding issues, as well as gain the ability to easily understand new encodings by fitting them into an existing mental model. The model I describe here is defined by the Unicode standard in the Unicode Technical Report #17, where it is known simply as the “Character Encoding Model”. Even though this model is defined by the Unicode standard, it is a general purpose conceptual model and can be used for character encoding standards other than Unicode.

Before discussing the individual layers of the model, it’s useful to remember what character encoding is, in a very basic way. At the risk of overstating the obvious, the point of character encoding is to go from a sequence of characters to a sequence of bits. The bits could be used in memory, persisted on a disk, or transferred across a network. At some point in the future the process is reversed, and the bits are decoded back into characters. By breaking this mechanism down into abstract layers, it is easier to understand all of the different transforms involved.

The first and most basic layer is an Abstract Character Repertoire. This defines a collection of characters that are described and given names. For instance, phrases like “the XYZ alphabet” and “the script used by XYZ” could be said to define character repertoires. An important aspect of character repertoires is that in no way do they define representations of characters. A repertoire simply defines member characters by naming them.

One point to note - a character repertoire assumes a definition of what a “character” is. For the purposes of this article, I’m going to hand-wave over the entire concept of character identity and what a character truly is. That is a topic that could easily make up an article all by itself. For now, whatever definition of “character” you have in your head is good enough to understand this layered model.

The second layer is a Coded Character Set. This is the first level at which actual encoding takes place: each character in the character repertoire is given a unique integer number to represent it, called a code point. Therefore, the coded character set level defines the first representation of each character, which is simply an integer. In other words, a coded character set defines a mapping from characters in a character repertoire to code points.

A coded character set, by its nature, defines a code space. The code space is the domain of the code points, and defines the minimum and maximum code point values. For large repertoires, it can be helpful to break the code space up into smaller sub-sections and give those sub-sections names.

Representing a sequence of characters as a sequence of numbers gets us a little closer to our ultimate goal of a sequence of bits, but it’s not a huge difference from characters. When compared to a sequence of bits, a number sequence is still a pretty abstract concept. The code points in a coded character set may have very different magnitudes (e.g., 10 vs. 10000), and a coded character set says nothing about how to represent these abstract integers as bits.

The third layer is a Character Encoding Form. A character encoding form transforms a sequence of code points in a sequence of equal-sized integers called code units. This is the first level at which bits are introduced into the encoding - code units are called equal size because each the code unit size is expressed as a number of bits. The size of code units may vary from character encoding to character encoding, but for a particular character encoding form the size is fixed. So when you hear the term n-bit character encoding, it refers to a character encoding form in which the code units are n bits long.

It’s easy to confuse the concepts of code points and code units for a few reasons. For one, they are both integers. For another reason, many character encodings use an identity mapping as a character encoding form, in which each code point value is equal to the code unit value. In such character encodings, the character encoding form is said to be one-to-one (i.e., one code point maps to one code unit). To remember the difference between the two concepts, keep a few things in mind. A code point is an abstract integer (e.g., 17), or just a point on some number line. A code unit is a fixed-size integer (e.g., 17 expressed as an 8-bit value or 0×11). Even though many encodings map one code point to one code unit, such a one-to-one mapping is not the case for all encoding standards.

The fourth layer is a Character Encoding Scheme, which maps individual code unit values to specific sequences of bits. For encoding standards in which the code units are of length 8 bits or less, the character encoding scheme layer typically does nothing. For encoding standards in which the code units are longer than 8 bits, the encoding scheme must map the code unit values into a sequence of bytes. This is where endianness issues arise - in this case a character encoding scheme specifies the ordering of the sequence of bytes for a code unit value.

The UTR #17 model also defines an optional fifth layer called a Transfer Encoding Syntax. This layer is different than the previous four layers. A transfer encoding syntax is almost always separate and orthogonal to the other four layers, and is often not specified as part of a character encoding standard but is used in addition to a defined standard. The most common use of a transfer encoding syntax is to apply some sort of post-processing to the sequence of bytes produced by the other four layers. For example, the sequence of bytes may be compressed to save space (e.g., according to an algorithm such as LZW). Or, the sequence of bytes may be further encoded by an algorithm so that it can be more easily transmitted over certain media (e.g., an algorithm like Base64).

It’s most useful to think of a transfer encoding syntax as a completely optional and separate fifth layer that can be added on to a stack of the other four layers.

To summarize the layers, an abstract character repertoire defines a set of named characters. A coded character set encodes a sequence of those characters as a sequence of abstract integer code points. A character encoding form represents the character sequence as a sequence of fixed-length integer code units. Finally, a character encoding scheme then represents the character sequence as a sequence of bytes.

Now that the levels have been defined, it is possible to give a slightly more precise definition of a character encoding standard. A character encoding standard specifies a stack of these four layers that when combined ultimately maps from a sequence of abstract characters in a repertoire to a sequence of bytes.

To further explain these layers, here’s a few examples using several character encoding standards that many software professionals will be familiar with.

First, consider a character encoding known as windows-1252. This encoding defines a character repertoire of 256 characters that are in the Latin alphabet and used in languages such as English (primarily), French, German, etc. This encoding defines a coded character set that maps each of the 256 characters in the repertoire to an integer value between 0 and 255. Further, since each code point value is between 0 and 255, a very straightforward character encoding form is used in which each code point value maps to an 8 bit code unit having the same value, which is obtained by 0-padding out each integer code point value to 8 bits. The character encoding scheme layer does nothing since the code units are only 8 bits in length.

As you can see, in a very simple character encoding standard such as windows-1252, some of the layers blur together or appear to be unused. This is a reflection of the fact that more complicated character encodings exist in which those layers are more distinct.

As a second example, consider a standard in which all of the layers are easily seen - the UTF-16 standard as defined by Unicode. The character repertoire defined by Unicode is huge. Unlike all other character encoding standards, Unicode is an attempt to include all useful characters in its repertoire - spanning languages, cultures, and even history. Unicode defines a coded character set in which each Unicode character is given a code point in the range from 0 to 0×10FFFF. Unicode code point values are often written in the form “U+hexadecimal code point value” (e.g., U+0041). UTF-16 defines a character encoding form that uses 16 bit code units. Each code point maps to either one or two code units. Finally, the encoding scheme specifies how the sequences of code units should be serialized as sequences of bytes. UTF-16 is actually a family of encoding standards in which the individual standards in the family differ only in the encoding scheme (in other words, they differ in byte ordering). For example, UTF-16BE uses an encoding scheme in which code units are serialized in big-endian form.

It is also useful to observe the transformation of a character as the representation moves through the layers. Consider the character “A” as encoded by windows-1252. “A” is included in windows-1252’s repertoire, and is given the code point 65 (coded character set layer). The character encoding form layer maps code point 65 to the code unit 0×41. The character encoding scheme layer does nothing since the code unit 0×41 serializes a single byte (1000001 in binary).

Now consider the character U+10140 (greek acrophonic attic one quarter) as encoded by UTF-16BE. This character is an ancient Greek number character that has only historical significance. I picked it at random since I wanted a character that would map to more than one UTF-16 code unit. This character is included in Unicode’s character repertoire and given the code point U+10140 (coded character set layer). UTF-16 maps the code point U+10140 to the two code units 0xD800 0xDD40 (character encoding form layer). The UTF-16BE encoding maps the two code units to the byte sequence 0xD8 0×00 0xDD 0×40 (character encoding scheme layer).

By learning this layered model for character encodings, you will gain both an understanding of how character encodings work and a mental model you can apply when things go wrong. It will also be easier to discuss character encoding issues with other software professionals since you can share a common set of terms. Finally, when learning new character encodings you can easily fit them into an existing framework, comparing and contrasting them with encodings you are familiar with.

Every so often (more often than I’d admit) I come up with a software development book idea. This idea is almost always for a book that has no equivalent currently on the market. For a short time I dream about pitching my idea to O’Reilly, Manning, APress, or Pragmatic. However, I usually realize pretty quickly there is a reason my book idea isn’t already written.

Here I present to you my latest book idea, complete with a chapter listing.

Whiteboarding for Software Developers

1. Know Your Equipment: A Guide to Whiteboards, Markers, and Erasers
2. The Good, the Bad, and the Ugly: Drawing for Software Developers
3. Whiteboarding UML: A Beginner’s Guide
4. Persistence: How To Save Your Diagrams Across Erases
5. Color: How To Add Clarity and Flair to Your Drawings
6. Whiteboard Etiquette: Using Shared Whiteboards in a Collaborative Environment (thanks to Scott Ambler)
7. Presenting: How To Effectively Incorporate A Whiteboard When Speaking To A Group
8. Under Pressure: How To Not Flop During Interview Whiteboarding
9. Cleaning and Care: Maintaining Your Whiteboard
10. Alternatives: When Not To Use A Whiteboard

You may think I’m joking about this book idea. However, if I came across this book in a bookstore it would be an immediate buy.

I don’t know what it is. Somehow sitting in front of the computer interferes with my thought process. This happens to me all the time. I’ll be sitting at my desk, thinking through some programming problem. I’m not getting anywhere. Then I do something that makes me get up from the desk, like walk over to a colleague’s office or walk down the hall to the restroom. Invariably the solution I was looking for pops into my head shortly after leaving my computer. This happens subconsciously - I’m not thinking about the problem at all when the solution suggests itself.

Today it happened again. I spent the last few hours at work trying to figure something out. Finally around 5:00 I gave up in disgust (I hate leaving on that note!) and headed home. About 2 minutes after leaving my desk (I was halfway through the parking lot) the solution I was searching for hits me.

I don’t know what it is, but somehow when you’re sitting in front of a computer for extended periods of time, your brain gets in a rut. It seems like all that’s needed to break out of the rut is to get away from the computer and stop thinking about the problem, even for a very short period of time (1-2 minutes). I’ve talked with other software professionals who’ve noticed basically the same thing.

So next time you’re blocked on a problem, try to stop thinking about the problem for a little while. Take a short walk or something. Don’t stay in front of the computer - checking email or blogs doesn’t get you out of the rut. You have to physically step away from the computer and completely take your mind off of the problem.

Stop whatever you’re doing for a moment, and take a look at the books on your bookshelf that you have to help you develop software. If you don’t have a bookshelf or aren’t a book kind of person, just play along anyway :).

There’s a book that should be on your bookshelf, and I think there’s a good chance it’s not. The purpose of this entry is to convince you to add this book to your bookshelf. It’s a book that you need to have close at hand, ready to pull out at a moment’s notice. It will be a help no matter what domain you work in, and is useful across programming languages.

The book is a thesaurus, and if you don’t currently use one when writing code I’m going to try to convince you to start.

It’s a well accepted and somewhat obvious fact that you’re not writing code solely for the benefit of the computer that will run it. In fact, the most important audience for your code is the programmers who will maintain it in the future (included your future self). The code should be optimized for reading. Write once, read many.

Coming up with good names for things in code is incredibly important. Good, of course, is subjective. Your domain, programming language, and team culture all play a part in deciding what “good” means for you. Spending a little extra time up front to come up with good names pays dividends down the road.

Here are two easy steps you can take, starting now, to improve the naming in the codebases you work on:

1) Take a little time to think of good names.
The more visible the thing you’re naming, the more important it is to have a good name for it. I’m thinking mostly about public, shared classes, as they are the most important things to create good names for. However, method names, variables, etc, are all very important too.

In order to come up with a good name, you need to have a clear understanding of the role of the class (or method name, etc. - from here on out I’ll just use class but feel free to substitute whatever construct you like) you’re naming. If the class doesn’t have a well defined role don’t waste time trying to name it. First refactor your code and split out responsibilities, and then come up with good names.

Be creative. This is where that thesaurus comes in. The English language has many ways of expressing a concept. Use them. Use them all, or make use of nuances. Whatever works well for your situation. There’s no reason that program code needs to have such a limited vocabulary as is often used.

2) Refactor mercilessly to keep names up to date.
Things change. Classes change behavior. Aspects are added or removed. Don’t be so in love with your names that you can’t change them when they’re no longer appropriate. Many modern IDEs have excellent refactoring support, allowing you to safely perform renames and automatically take care of updating references.

Whatever you do, don’t let names get out of date. There’s nothing worse than trying to make sense of unfamiliar code when the names don’t align with the functionality. Keep in mind that even though you may have a great understanding of the codebase, the newcomer to your team, the future maintenance programmer, or yourself down the road won’t necessarily have that same understanding.

Here are a few other things to keep in mind:

Good names should be unique, or close to it. This is especially important when you have a large codebase and people will often open types by doing a search (ctrl-T in Eclipse) by name. Here’s where that thesaurus comes in handy again. It should be relatively easy to come up with a name that’s unique enough to not get confused with other names that are important in your system. Here’s an Eclipse-specific tip: you can tell Eclipse to filter out certain packages when searching for classes by name. For instance, if you are not a Swing/AWT programmer and don’t want to see java.awt.List come up every time you search for “List”, you can filter out the java.awt package. Go to Window -> Preferences -> Java -> Appearance -> Type Filters.

As you name things in your codebase, you’re creating a taxonomy and a common nomenclature for your team. You will have to refer to these things by name to your team members, and they will come up in discussions, design meetings, support issues, etc. Are you choosing names that make sense in this context? Sit back from the keyboard for a moment, and try to talk about a class or several classes in complete English sentences. Assuming this works, many of those words you used are great candidates for names in your system. Choose names that are close to your domain and express what you’re trying to accomplish. A great name is better than a weak name with a large comment block that attempts to explain the role of the class.

A great book to check out along these lines is Domain Driven Design by Eric Evans. The book talks in great detail about focusing the development process around the concept of a strong domain. Choosing good names is a part of that.