Perhaps you’ve heard the term Pack200 before but haven’t had a chance to become familiar with it. Or maybe you already know that Pack200 is related to deployment of Java applications, but aren’t sure how it could be used with your application. Compared to other new features of Java 5, Pack200 hasn’t gotten as much attention. In this entry I’ll answer two questions: first, What is Pack200?, and second, Why would I want to use it?

Pack200 was released as part of the Java 5 platform, and is essentially a technology for achieving much better compression ratios of deployable Java code. Java code has traditionally been packaged and deployed as JAR (Java Archive) files, which are nothing more than standard zip files with the extension .jar. Pack200 can result in radically higher compression ratios of Java bytecode when compared to traditional JAR packaging.

The name Pack200 is derived from two sources, and to understand why the name was chosen you have to know a little bit of the history of the technology. William Pugh (best known as the developer of FindBugs) released a paper detailing a number of advanced techniques for compressing Java class files. These techniques were used by Sun to decrease the size of the JRE and the JDK downloads, starting sometime around the Java 1.4.1 release. William Pugh’s ideas and the format Sun used were referred to as Pack. Around the same time period, Java Web Start / JNLP technology was becoming relatively popular and Java applets were also starting to become more popular once again. A JSR (Java Specification Request) was created called JSR 200. The JSR was created to specify a “dense download” format for Java bytecode, based on the technology presented by William Pugh in his paper and already in use internally at Sun. This technology was eventually called Pack200 (from the JSR number) and became public and supported with Java 5.

The target use case for JSR 200 was to enable more optimal web deployment of Java applications, specifically in the case of Java Web Start and applet applications. The motivation there was to reduce the download / update time for the client and the bandwidth usage for the server. Java 5 includes hooks in the JNLP bits and the browser plugins so that you can use Pack200 out of the box with Web Start applications and applets (see Sun’s deployment guide). Even though Pack200 was designed for the Web Start type scenario, it is very applicable to any type of client-side Java - I’ll explain how later on in this entry.

The Pack200 technology is intended to be a deployment vehicle, and doesn’t come into play at runtime. A Java virtual machine still works only with class files and JAR archives on the classpath - Pack200 packages must be unpacked before the bytecode inside them can be loaded into a virtual machine. The right way to think about Pack200 is that it’s a faster, more efficient way to get bytecode onto client machines.

The basic premise of Pack200 is simple. Since the JAR format is a standard, generic file format, it is not designed to treat Java class files any differently from other file types. However, Java class files have certain properties that can be cleverly used by a more specific packaging technology. By exploiting specific aspects of the Java class file format, an extremely dense representation of a collection of class files can be generated. Traditional JAR files normally achieve compression ratios of 1:2 or 1:3. It’s not uncommon for Pack200 files to achieve compression ratios of 1:10 or greater.

For an example, I took a collection of about 1000 class files that make up a library in one of the projects I’m working on right now. I created a traditional JAR archive and a Pack200 package containing these class files. The files take up 3,987,908 bytes on disk. The JAR archive takes up 1,834,269 bytes on disk, or about 46% of the size of the original class files. The Pack200 package takes up 266,803 bytes on disk - 15% of the size of the JAR file, or an amazing 6.7% of the original size.

So how does Pack200 achieve such high compression ratios? For the full answer, read William Pugh’s paper linked above. In short, Pack200 combines a number of different low level techniques, each of which exploits a different aspect of the Java class file format or Java bytecode. Among other things, Pack200:

– Merges the constant pools of each individual class file to form one constant pool that’s shared for a collection of class files. This is a big win since many of the same constants (common class names for instance) appear in many of the individual constant pools. Basically, this technique eliminates the redundancy in the individual constant pools of class files in a package.

– Uses delta encoding whenever possible. For trivial example, to store two Strings that share a common section of characters, only one complete String needs to be stored, and the second String is stored as a delta off the first. This kind of encoding can be applied in a lot of different areas - even storage of similar sequences of numbers optimized with this technique.

– Implements variable length encoding. Variable length encoding allows values of the same data type to be stored using different amounts of space. For example, a small numeric value of a data type can be stored using fewer bytes than a larger value of that same data type.

– Enables more optimal secondary compression. Pack200 packaging is normally followed immediately by a gzip stage, which reduces the size of the package further.

– Incorporates a number of optional techniques for further reducing the size of class files. For instance, debug attributes can be stripped out of the class files.

Pack200 packing takes a JAR file as an input. A .pack file is then produced with the packed bytecode. The pack process is normally finished by gzipping the .pack file (the secondary compression mentioned above), ending with a .pack.gz file. The unpack process simply reverses the above: first ungzipping the .pack.gz file, and then producing a JAR file from the .pack file. The pack / unpack process is fairly resource intensive (especially compared to simple JARing). The pack process is done once on the server, and the unpack process is done once on each client machine at the time of install or deployment.

Pack200 has two interfaces: a command line interface and a programmatic interface. The Java 5 JDK and JRE ship with two command line tools: pack200 and unpack200. Interestingly enough, these command line tools are native C++ executables - they have no dependency on a Java runtime. There is also a class than can be used to programmatically pack and unpack - java.util.jar.Pack200.

Even though Pack200 works against JAR files, it’s not correct to think of it as a generic compressor for JAR files. For instance, a JAR file containing mostly resources will not achieve that high of a compression ratio - but a JAR file containing mostly class files will. Pack200 is a compressor for Java bytecode - it just uses JARs as input for convenience and integration with existing packaging mechanisms. The Pack200 format is lossy - run a JAR all the way through (pack and unpack) and the final JAR will be different from the original. Of course, the two JARs will be equivalent from the point of view of a Java virtual machine. This has some implications for JAR signing, so be sure to read through Sun’s deployment guide linked above if you’re going to combine Pack200 and JAR signing.

Don’t make the mistake of thinking that Pack200 is useful only for Web Start and applet scenarios. Any time there is a need to move Java bytecode from a server onto many client machines, Pack200 can be used to make life easier for both the client and server. A heavyweight Java client application could greatly benefit from Pack200. All it requires is Java 5 on the client machine, and an install routine that unpacks the Pack200 packages as part of the installation. This install routine could itself be written in Java and would make use of the Pack200 support in Java 5 (either programmatically or through use of the command line tools). An install scenario like this could reduce the download size of a client-side Java application by half or more, greatly reducing the bandwidth used by the server and the wait time on the client for the download.

A real-world example of where this technology is being successfully used outside of the Web Start and applet case can be found in the popular Eclipse IDE. Starting with version 3.2, the Eclipse update manager (used to get new Eclipse features and update existing features) will make use of Pack200 technology when Java 5 is available on the client. This will greatly reduce the wait time for updates to through the update manager, as well as reducing the bandwidth burdens on the Eclipse mirrors.

If you’re writing client-side Java applications in 2006 and haven’t yet looked at Pack200, take some time to evaluate the technology and see how it might fit into your overall deployment strategy.

(This is part of a series about API Design)

I think most software developers have had the experience of using some API and seeing completely unexpected results. Maybe it was the exception that was swallowed inside the API with nary a trace, or perhaps it was the benign-looking method call that resulted in an unnoticed-until-too-late side effect.

The principle of least surprise says that using a well-designed API should be as unsurprising as possible. What does it mean for an API to be unsurprising? An API that follows the principle of least surprise conforms to the consumer’s expectations.

Consumer Expectations

Predicting (and then meeting) a consumer’s expectations is an extraordinarily tricky business. In contexts other than API design, this has been the subject of many books and articles. Ultimately this is a hard problem to solve completely since different users have different expectations. Worse, many of the expectations may be contradictory or mutually exclusive.

So we can start by stating that even a well-designed API will not be able to meet every expectation of every consumer. Having said that, there are some general techniques that go a long way towards creating an intuitive API. By understanding how an API consumer’s expectations are formed, we can gain some insight into what steps an API designer can take to follow the principle of least surprise.

Generally speaking, API consumers base expectations of the API on three things: existing platform idioms, API identifier naming, and API documentation.

Follow Platform Idioms

An API should follow the established idioms of the platform that it is written for. In the same way that you can write Fortran in any language, it is very possible to create a C++ API in Java (for example).

For some APIs, the API is wrapping a service that is written in the same platform or programming language as the API itself. In this case, the API designer is more likely to follow platform idioms, since they will be very familiar with the platform. But it’s often the case that an API is written as a bridge from one platform to another. In these cases, often the API designer is not as proficient in the API platform.

In the cases where an API designer is inexperienced in the target platform for the API, it’s essential to try to get a feeling for the idioms of the platform. Failing to follow them will create an API that seems surprising before it’s even used. The best way to get a feeling for a platform’s idioms is to read some well-written code for that platform (think open source). Many platforms also have great books available that can sum up years of hard-earned experience in idioms. On the Java platform, the book Effective Java is essentially a compendium of idioms for the Java language.

For example, I would expect a Java API to contain method names beginning with lowercase letters, and I would expect a C# API to do the opposite. Another example is that APIs should use the collection classes that are available on the platform instead of rolling their own collections (I’ve seen a few APIs in Java with class XYZ and collection clas XYZList everywhere). I could go on - the examples here are pretty much limitless - but the basic point is simple: don’t go against the grain of the platform.

Use Single-Minded API Methods

A great way to throw API consumers for a loop is to include methods that seem to do one thing, but in fact do that as well as something entirely different at the same time (a side effect). Avoiding side effects is a good design principle in general, but becomes even more important in the context of an API.

The goal as an API designer is to guide the consumer into forming a mental model of an API. The mental model allows the consumer to approximate the effects of the API given some usage of it - in effect the consumer will use the mental model to simulate the API before actually invoking it. As the API consumer discovers method side effects, it’s a little bit like punching holes in that mental model.

As an API evolves, it can be tempting to create side effects as new functionality is added. It’s often easier to reuse an existing method than it is to create new public API that must be documented and supported. But whenever possible, avoid this and keep API methods simple and single-minded.

Error Handling

Improper error handling is a common way for APIs to be surprising. Again, this is an area in which an API designer should defer to the conventions of the target platform of the API.

On the Java platform, this means following well-established best practices for exception handling. Going into detail on all of those could be the subject of a blog entry of its own. In short, don’t swallow exceptions unless they are expected and able to be handled inside the API, don’t log exceptions before throwing them up, and prefer unchecked exceptions when possible.

It’s especially important for APIs that wrap other services to return complete error information from the wrapped service. For example, an API that wraps a remote web service will need in some way to handle errors that come back from the web service. However they are handled in the API, the complete error must also be made available to the API consumer. There’s nothing as frustrating as trying to debug an error condition when one of the layers below you is hiding error state.

Some APIs encode usage information by creating unexpected errors. For example, an API might have a method that can only be called after another method has been called first. The API might throw an exception if this precondition is not met. Certainly it’s better for an API to fail fast than to hide erroneous usage from the consumer. If the consumer calls the wrong method at the wrong time, throwing an exception is better than silent failure. But it’s much better to design an API in such a way that there are no “wrong methods”. I’ll talk more about this when I cover the principle of least resistance. But if you find yourself writing lots of “sanity check” error code, trying to get the consumer to do the right thing by throwing errors at them, it’s probably time to rethink the API usage.

Clear Naming

Clear and consistent naming of identifiers across an API is hard. But API consumers will use class and method names to form expectations of how an API works and what the effects of certain usages will be.

API designers don’t intentionally give identifiers misleading names. But we’ve all come across APIs with methods or classes that are very badly named. This usually happens because of API evolution - as some new functionality is added, existing methods must be changed, and often perform different functions or perform the same functions in different ways. It’s important to take care when enhancing an existing API to keep the names consistent. It’s probably better to deprecate existing API clients and add a new method or class instead of having parts of the API with warped names.

Simple naming techniques can make a big difference. Make sure to differentiate between an identifier for something, and the thing itself. For example, a method called getFile() should return a file handle - not a file name or an input stream (those could be called getFileName() or getOpenedFileInputStream()). Names should always be descriptive - calling a method getHitCount() instead of getCount() makes it much less confusing when a second method called getXYZCount() is added later.

Documentation

Sometimes an API seems to do everything above right, but still ends up being confusing and counterintuitive. It’s possible this is unavoidable in some cases. Perhaps the API is providing access to esoteric or highly technical services, for which there is an unavoidable learning curve. Perhaps the intended audience of the API will not be able to quickly form a mental model of the code.

In these cases, great documentation can make an API less surprising. The documentation should point out trouble spots in the API - areas where an intuitive understanding is likely to lead to bad usage. If the documentation is too long or too formal, many consumers probably won’t read it and will jump right in to tinkering with the API. So keep the documentation short, to the point, and use it to explain only those things that aren’t going to make sense to most API consumers.

There are certain things that need to be documented since simple inspection of an API can’t tell the consumer about them. For example, caching policies come to mind. If an API returns some object to the consumer, is that object cached internally? Should it be cached externally? There are many “policy” considerations that can only be sufficiently explained in API documentation.

Conclusion

An intuitive API is optimized for fast understanding by the consumer. Its purpose is to get out of the way as quickly as possible to allow the consumer to do their work. I’ve just scratched the surface here - a great way to get an understanding of what makes an intuitive API is to use lots of counterintuitive APIs.