I’ve left a few variables in the commands below, which you’ll need to replace with the actual paths and values:

• GLASSFISH_HOME should be replaced with the root directory of your Glassfish installation. In my case, that’s /opt/glassfish3, but your environment is probably different.
• GLASSFISH_DOMAIN should be replaced with the name of the Glassfish domain you’re configuring. That might be domain1, or if you’ve created your own domain with asadmin create-domain, use the name you used then.
• KEY_ALIAS should be replaced with some meaningful name for the SSL certificate pair. I used https-www.example.com, but you can use anything you like other than the two aliases used by Glassfish internally.

You will need the Java 6 tools installed, including keytool, as well as the asadmin command set up to manage your domain. You will also need write access to the keystore.jks file stored in the domain’s config directory.

The domain should be stopped while you modify its keystore. If the domain is running, restart the domain in step three rather than starting the domain.

Step one: convert the key/cert pair into a single PKCS#12 keystore

openssl pkcs12 -export \
	-in www.example.com.cert \
	-inkey www.example.com.key \
	-out https-certificates.pkcs12

The openssl command will prompt for a password to use when creating the new PKCS#12 keystore. This is not optional, but since we only need this keystore temporarily, setting it to an unsafe password like “password” is fine. Just be careful to delete the keystore away safely afterwards.

Step two: import the key/cert pair from the PKCS#12 keystore into Glassfish’s JKS keystore

keytool -importkeystore \
	-srckeystore https-certificates.pkcs12 \
	-srcstoretype PKCS12 \
	-srcstorepass password \
	-deststorepass changeit \
	-destkeypass changeit \
	-destkeystore $GLASSFISH_HOME/glassfish/domains/$GLASSFISH_DOMAIN/config/keystore.jks \
	-srcalias 1 \
	-destalias $KEY_ALIAS

(If you changed your Glassfish master password, you may need to change -deststorepass and -destkeypass to match. If you used a different password when creating the PKCS#12 keystore above, change -srcstorepass appropriately.)

Step 3: configure Glassfish

By default, Glassfish sets up a network listener called http-listener-2 to handle HTTPS connections on port 8181. We’ll need to tell it to use the new key/cert pair:

asadmin start-domain $GLASSFISH_DOMAIN
asadmin set server.network-config.protocols.protocol.http-listener-2.ssl.cert-nickname=$KEY_ALIAS

At this point, you should be able to point a browser at https://your-glassfish-server:8181/ and pick up the new key.

So I thought I’d write one.

This post will cover some introductory knowledge about LDAP. Subsequent posts will introduce some LDAP tools, go into more detail on the data stored in a directory and on the implementation of an authentication system.

“LDAP” is an extensive subject, and I’m not going to try to cover every aspect of it. (For that, see the links at the bottom of this post.) I’ll be demonstrating simple bind authentication, without SASL or Kerberos/GSSAPI, and I won’t be going into too much detail outside of users and groups. In particular, I won’t be covering too much history, and I won’t be covering ActiveDirectory (the other widely-deployed authentication and directory service built on LDAP).

LDIF

I’ll be using LDIF to describe LDAP data throughout this series. Don’t worry about the details too hard for now; I’ll cover it in more detail later on. For now, all you’ll need to understand is how attributes and objects will appear in simple common cases.

It’s A Database, Stupid

LDAP is a protocol and a data model for providing access to a hierarchal (tree-shaped) database. The database itself is normally referred to as a “directory” (hence the name), a bit of nomenclature inherited from its predecessor, X.500, populated with objects that are bundles of attributes. LDAP supports direct lookup of an object by name, searches through subtrees or through the whole directory for objects matching a sophisticated pattern language, atomic updates to part of or all of individual objects, and network federation.

Attributes

LDAP’s data model is built on top of attributes. Attributes form the basis of object names, store data for user applications, and store data maintained by and for the directory service itself.

Each attribute has a name and a value. Some attributes can appear multiple times in the same object; others can appear only once.

Attributes are written in LDIF at the start of line. The attribute name appears first, followed by a colon, followed by the attribute value. Attributes can span multiple lines; a line that begins with whitespace is assumed to be a continuation of the preceeding line’s attribute value.

This LDIF fragment describes a single attribute:

givenName: John

This is a pair of attributes:

givenName: John
surname: Franklin

This is an attribute with multiple values:

mail: john@example.com
mail: jfranklin@example.net

Attributes are divided into two groups: user attributes, which hold data for applications using the directory, and operational attributes, which hold data for use by the directory system.

Names

Every object in an LDAP directory has a distinguished name (DN) which identifies its location in the directory’s tree. An object’s name is a sequence of attributes, written with the root of the tree on the right. In a DN, attributes are written in name=value form and are separated by commas, rather than LDIF’s name: value form.

This is a DN:

givenName=John,o=Example

DNs have a parent-and-child relationship. In the example above, givenName=John,o=Example is the DN of a child of the object named o=Example. Directory trees can have very deep hierarchies of names, or very shallow hierarchies of names, depending on the needs of the application.

Every directory tree has a special DN called its base DN which acts as the root of the directory tree. The base DN of a directory tree may contain multiple segments, which would imply that it has parents, but the parent objects do not need to exist in any directory tree. Every directory service has a second special DN, identified by the empty string, naming the root of the directory service.

Objects

Objects hold the data stored in the directory. Each object has a name, distinct from the names of all other objects, and a suite of attributes describing the object.

In LDIF, objects are written as a list of attributes, starting with a dn attribute holding the object’s DN. Multiple objects can appear in a single LDIF file, separated by a blank line.

This is a pair of objects:

dn: o=Example
objectClass: organization
o: Example

dn: commonName=John Franklin,o=Example
objectClass: person
givenName: John
surname: Franklin
commonName: John Franklin

(The dn attribute in the LDIF description is not an attribute of the object, but rather part of the LDIF language. It must appear first in any LDIF representation.)

Every object has a user attribute named objectClass, which identifies what kind of data the object holds.

Schemata

Unlike more recent key/value databases, LDAP validates stored data against a schema. Each object’s objectClass attributes identify the schemata that apply to that object.

Object classes are broken down into two groups: structural classes and auxiliary classes. (There’s a third group, abstract classes, which we won’t cover here. Abstract classes do not usually appear in directory objects’ objectClass values.) Every object in the directory must have exactly one objectClass attribute identifying a structural class, and can have many (or no) objectClass attributes identifying auxiliary classes.

The schema for an object class includes the class’s names (most object classes have a single name, but aliases are permitted, with the first name in the schema being the canonical one), a description, the name of the class it expands on, the structural or auxiliary nature of the class, a list of attributes objects with the class must have, and a list of attributes objects with the class may have.

Each attribute in an object class definition has a corresponding attribute type definition. The attribute type definition includes the attribute’s names (as with object classes, attributes can have multiple names; this is relatively common), description, the rules for comparing two values for equality, ordering, or substring presence, the syntax rules for the attribute’s values, flags indicating whether the attribute can have multiple values, and flags indicating whether the attribute is for user or directory tree usage.

The LDAP specifications (RFC 4510 and friends) provide a set of common object classes and attribute types that are available on most LDAP servers. There are also common schemata for a number of use cases, including acting as an address book, authentication database, or network information service. Most directory servers also allow users to provide their own schemata to customize the directory for their particular needs.

Parent/child relationships between objects are not specified by schema information. Instead, each application and deployment works out the hierarchy that best fits the problem at hand. Several common hierarchy conventions are documented in RFCs and reference manuals, but most LDAP tools are flexible enough to work with a wide variety of tree shapes.

Finding Things

LDAP provides the ability to search for objects within a directory tree. A search request comes with four things:

• The search base DN.
• The search scope.
• A filter.
• A list of attributes to return.

The search base and the search scope combine to limit the parts of the tree that the search will examine. There are four scopes:

base

Return only the object identified by the search base DN, or no object.

one

Return only objects that are immediate children of the object identified by the search base DN.

subtree

Return objects that are descendents of the object identified by the search base DN, and the object identified by the search base DN.

children

Return objects that are descendents of the object identified by the search base DN. (This is a common extension, rather than part of the LDAP standard.)

The directory applies the search filter to the objects that are in the search scope, including only objects that match the filter in the final result. The filter is specified using a small expression language that permits several kinds of matching on attributes. Primitive filters, which match attributes against values or patterns, have the form (NAME OPERATOR VALUE), while composite filters, which combine filters together, have the form (OPERATOR FILTER [FILTER...]).

Some examples:

The full filter language includes support for partial matches, numerical comparison, presence or absence of an attribute, and several other features that I’m not going to go into here.

Finally, the directory server extracts each of the attributes named in the search request. Searches can ask for attributes by exact name, or by requesting the attribute * (which returns all user attibutes) or + (which returns all operational attributes). Attributes with multiple names are normally returned under their canonical name – for example, asking for the surname attribute will normally return attributes named sn under commonly used schemata. The final search result includes the matched objects’ DNs, as well as the requested attributes.

Indexing

Like any database system, LDAP directory services generally provide indexing to ensure common searches return quickly, without having to walk the entire directory tree to find the matches. I’ll cover this in more detail for OpenLDAP in a later post.

Next

Next time, we’ll look at OpenLDAP and the UNIX ldap utility programs, and construct a simple address book.

References

• RFC 4510 – Lightweight Directory Access Protocol: Technical Specification Road Map
• LDAP for Rocket Scientists

The following is a slightly expurgated version of the postmortem I sent the team. Table names have been changed to protect the guilty.

What Hotfixes?

In the deployment plan for Sinistar’s post-release migrations, there were two ALTER TABLE migrations slated to make structural changes to the customer and customer_login tables (changing the types of some columns, specifically). When we tested these migrations on a copy of the production system’s data, we discovered that they would have caused those tables to be inaccessible for two and five minutes respectively. Since customers are core to our applications, we determined that this qualified as “downtime” in the release plan.

I prototyped an alternate approach to the customer changes (outlined below) intended to make changes to the table without taking the app offline while they were happening. The alternate approach was considerably more complex than the original ALTER TABLE statement, as well as being considerably slower (~15 minutes for our largest shard, as compared to two minutes); we discussed whether it was worth taking the app offline at zero-dark-hundred on a weekend instead and concluded that we’d need to be able to make online changes to expensive tables eventually and opted to go ahead with the alternate approach as a learning experience.

How Did It Work?

The alternate approach we used relied on more MySQL-side tools to make the changes. Specifically, we:

1. Created a new, empty version of the customer table (the “target”) table based on the current customer table plus intended structural changes. Structural changes to an empty table that the app does not use are quite cheap.
2. Installed triggers on INSERT, UPDATE, and DELETE operations on the customer table that copied rows (using REPLACE) into the target table, applying the intended transforms to the affected columns.
3. Installed a server-side procedure (“the migration procedure”) that:
   1. Selected all of the keys from the original customer table.
   2. For each key, copied the corresponding row from the customer table to the target table using REPLACE.
4. Ran the migration procedure. (This part is where most of the time went.)
5. Verified that the contents of the target table were in 1:1 correspondence with the contents of the original customer table, where the only permitted differences between corresponding rows were the changes intended by the original migration.
6. Renamed the original customer table out of the way and renamed the target table to customer.
7. Verified that nothing had gone horribly wrong in our apps.
8. Dropped the renamed version of the original customer table, along with the triggers and the migration procedure.

The steps are grouped into four phases: prepare.xml (covering steps 1-3), execute.xml (covering step 4), cutover.xml (covering step 6) and cleanup.xml (covering step 8). The steps not covered by phases were performed semi-manually.

The changes to customer_login proceeded along the same paths.

What Went Right

Well, first of all, we noticed a potential app outage hazard before it went live.

We were able to back out and rewrite the migrations without affecting existing development environments, where the original ALTER TABLE might have already run, thanks largely to Liquibase‘s precondition support.

We ran through several versions of this plan, both on my development environment image and with QA (on our release candidate environment), which ferretted out a handful of subtle bugs in the migration procedure that would’ve damaged customer data. Our QA team’s extensive experience and automation tooling around our applications prevented any of those mistakes from going live.

Designing the hotfix in several phases permitted us to run the relatively safe phases over the weekend, with relatively little involvement from our operations team beyond monitoring.

The verification steps did not take out our applications in the process, and revealed a way to move at least some verification steps out of our applications’ scripts directory and into the core-dbs source tree.

Finally, it worked. Despite the relatively high complexity and the large collection of moving parts, this alternate plan worked very well. Rather than taking our applications offline, everything stayed up (and didn’t slow down too badly).

What Went Wrong, And What We Learned

Finding Problems Late is A Problem

We didn’t catch the initial problem before it went out to everyone’s development environments. I’ve habitually not worried about migration timing until code freeze, which means that potentially-troublesome migrations will have already run on our bleeding-edge environment (and possibly our release candidate environment) before we notice the problem. This lead to increased complexity in the alternate migration path’s Liquibase configuration – we made extensive use of preconditions to ensure that the alternate migration would be harmless on systems where the original ALTER TABLE had already run, which seems to have worked, but I’d rather not have had that problem in the first place as it’s another moving part that can have problems.

Concurrency is a Hard Problem

The first version of the migration procedure looked like this:

1. Select all of the rows from the original customer table.
2. For each row, insert that row into the target table using REPLACE.

The select and loop steps use a MySQL cursor to iterate over their results. While the MySQL documentation states that it’s undefined whether a cursor is over a snapshot of the query results or over the underlying tables, in practice we found that it was using a snapshot. Since the loop took considerable time to run, and since it (intentionally) did not prevent further changes to the original customer table from occurring after the snapshot, the loop would insert, into the target table, a “stale” version of any customer row that had been modified since the snapshot was taken (overwriting the correct version copied into the target table by the triggers on the original table).

We only noticed this on our release candidate environent. When we ran through the migration plan initially to ensure that the procedure did not block our applications’ normal usage, QA ran part of their Selenium suite, which produces a relatively high volume of test traffic. Out of ten thousand customers in the release-candidate databases, four customers had stale rows copied. This is not a bug we would ever have identified using only manual testing (even with a verification tool), so I’m intensely pleased that we had automation in place and very glad I got QA involved in testing the hotfix as extensively as we did.

Switching from a snapshot of the rows in customer to a snapshot of the keys in customer and selecting out the row itself only immediately before copying it to the target table addressed the problem; even in production, our verification script identified exactly zero mismatches between the original customer table and the target table (across ten million customers). It also made the procedure body somewhat shorter and simpler by reducing the number of fields copied from the cursor on each step of the iteration.

I’ve Got Ten Million Queries, Is This Bad?

Boy, is it slow. The migration procedure, which issues one query per affected customer row, takes approximately ten times as long as a straight ALTER TABLE, and produces considerably more IO load. This lead to the customer hotfix running into Friday morning (we kicked it off Thursday afternoon). I made some changes to the customer_login migration procedure to attempt to compensate – instead of issuing a REPLACE query for each row, we used a prefix of customer_login‘s primary key (customerid) instead to issue only one REPLACE query for each customer’s worth of rows. This seems to have helped, but not by much. Future iterations of this process might even go by larger groups of rows, depending on the average and worst-case number of rows per group.

The Best Laid Plans…

We ran through three iterations of the migration procedure for the customer table before we settled on a working version. While I did design recovery migrations to back out the broken versions of the procedure that were intended to leave trunk and other environments unaffected by the iterative testing process, they didn’t quite work as planned. Somehow (and we still don’t know how for sure), all of the customers on our bleeding-edge environment were deleted outright instead of being migrated over. We weren’t able to reproduce the problem with the final plan, and we do keep backups of our testing databases, so we were able to recover the missing customers, but it’s another thing that we only noticed because of QA’s intervention.

We avoided this with customer_login by not building the migrations for that table until we’d ironed out issues in the customer migrations, and in the future we’ll avoid rolling this sort of hotfix into master (for our bleeding edge environment and development environments) until we’re confident that it’s correct.

No Space Left On Device

This is one that caught me totally by surprise. When we ran the migration procedure for customer in production, we learned that passing 10m queries through the MySQL binary logs (for replication) uses up a lot of space. Fortunately, we keep the binlogs on a separate partition from the main data store, so this did not take MySQL out, but it could’ve if our Ops folks weren’t so sharp. This did, however, take replication out temporarily. Operations resized the partition the binlogs live on, which kept things running (slowly, but smoothly) while the customer_login migration procedure ran.

Read Locking Is Not Your Friend

Our initial version of a verification tool, a PHP script in one of our apps’ scripts/ directory that ran against the live databases on our master database, would’ve taken our applications offline in roughly the same way as the original ALTER TABLE statements would’ve. I only realized this when I ran the verification queries (outside of their script) against one of our replica databases’ shards and realized how long they took, shortly before we had planned to run the verification step in the hotfix plan. Fortunately, the verification used very little from our app’s environment; I ported it to a simple shell script and ran it against a replica (after replication caught up) instead. In the future, I’m going to try to get more of our verification scripts written that way where possible, since it does a good job of insulating our applications from side effects.

In Summary

Working with large data sets is hard, especially in MySQL (where even reads can cause locks). Making structural changes to those data sets is even harder, especially while the application’s in flight. However, I feel like this went relatively well even with the “creative” issues discovered during testing, and we’ll be using this approach for large tables more frequently in the future. If you’re making a change to a table and you’re not sure if it’s “large” or not, we’re here to help.

Imagine that you, and a few of your close friends, are practicing a juggling act that will involve throwing fine china plates, donated by volunteers in the audience, to one another. Knowing that this is an expensive and risky act, your troupe first procures a large stack of “practice” plates – they’re about the right shape, and about the right weight, and they’re cheap so that when you inevitably drop a few and smash them, they’re easy to replace.

Your troupe’s practice goes about as well as can be expected – incremental improvements, many smashed plates, a few complete reworks of your blocking and choreography, many smashed plates, a lot of feedback from a handful of folks you’ve dragged into rehearsals with you, and many smashed plates. However, every time the act falls apart and the plates hit the ground, the group of you shrug, pick up some more plates, and try again. These “practice” plates have no intrinsic value, so losing them all means nothing and costs very little.

Little by little, you refine your act, until you can do it without any dropped plates at all! You and your troupe can smoothly keep enough plates in the air to serve a five-course meal without having any mishaps.

Then comes opening night.

Suddenly, you’re up there with your expensive, real plates, in front of an audience of hundreds or thousands. These plates matter, and even if you drop one, you’ve still got many others in the air that also matter.

In fact, these plates matter so much that if you do drop one, you need to somehow put it back together and get it back into the show without dropping the others.

Two nights later, you discover that you need to change your juggling act, right in the middle of the show, without dropping any plates.

So it goes with software, too. While you’re building a new system, it’s relatively easy to gut your test data and start over, and nobody cares. Once your system holds real data, though, you need some way to keep the plates in the air while you make changes. Suddenly, the data matters, and the longer your system’s been around, the more it matters to everyone involved.

While there are other factors that drive up the cost of maintaining existing software, screwing up your users’ data is one of the fastest ways to destroy your users’ trust. How well does your system juggle plates?

All of these notes assume you’re setting up an LDAP replica on OpenLDAP running with cn=config configuration. I did this on Ubuntu 10.04, and many commands include Ubuntu- or Debian-specific assumptions. Translating to slapd.conf or to, say, Redhat Enterprise Linux is left as an exercise. Obviously, most of this is not safe to apply to an existing OpenLDAP server and there are a few places where we step outside of Apple’s comfortable padded box on the OpenDirectory side, too. I’ve tried to point those out, but try this at your own risk.

Directory Access

You can use ldapsearch and ldapmodify and so on to manipulate and search the OpenDirectory master. Apple configures slapd to allow access to the cn=config tree to anyone who connects via ldapi://%2fvar%2frun%2fldapi, even “anonymously”. For example, to read the cn=config entry you’d use

master$ sudo ldapsearch -x -H ldapi://%2fvar%2frun%2fldapi -b cn=config \
    -s base '*' '+'

Passwords

“Open Directory” passwords live in Apple Password Server, which I didn’t try to replicate over, rather than in the directory. For users who should be able to authenticate against the replica, I switched their passwords to Crypt passwords in Workgroup Manager. This has all of the downsides of Crypt password storage: 8-character limit, weak digest, and so on.

Open Questions

Does Workgroup Manager use the moral equivalent of ldappasswd(1) when setting users’ passwords? If so, configuring “Crypt” passwords to use salted SHA instead may be a matter of changing olcPasswordHash to {SSHA}. We weren’t brave enough to try.

Certificates

On the OpenDirectory master, I generated a self-signed “SSL Server” cert with a long expiry (ten years) and configured OpenDirectory to use it for SSL. The resulting .pem file is named after the server (ldap.example.com) and lives in /etc/certificates/ldap.example.com.A REALLY LONG HEX STRING.cert.pem. You can find the exact name by looking at the olcTLSCertificateFile entry in cn=config. If you’re using SSL for replication (and you should be), grab a copy of this file and put it on the replica:

master$ scp /etc/certificates/ldap.example.com.*.cert.pem \
    replica:/tmp/ldap.example.com_cert.pem

and

replica$ sudo cp /tmp/ldap.example.com_cert.pem /etc/ssl/certs/

Replication Provider

OpenDirectory compiles syncrepl directly into slapd, for use with OpenDirectory replica servers. However, you can abuse that support to provide replication to OpenLDAP as well. You’ll need to add the syncrepl configuration to your directory manually. Here’s the LDIF I used:

master$ sudo ldapadd -x -H ldapi://%2fvar%2frun%2fldapi <<'SYNCREPL'
dn: olcOverlay=syncprov,olcDatabase={1}bdb,cn=config
objectClass: olcSyncProvConfig
olcOverlay: syncprov
olcSpCheckpoint: 100 10
olcSpSessionlog: 100
SYNCREPL

You can use any of the syncrepl provider configuration options here.

Open Questions

This configuration uses basic syncrepl. In theory delta syncrepl should work as well, but I was trying to make as few changes to the OpenDirectory server as possible.

It’s unclear what changes in Server Admin will blow away the syncrepl overlay, but I suspect setting up a real OpenDirectory replica may affect this config.

Schemata

Hoo boy.

OpenDirectory’s directory server is a patched version of OpenLDAP. The patches include baked-in support for a handful of attribute types and object classes, which are omitted in Apple’s apple.schema and apple_auxiliary.schema files. I opted to use Apple’s schema files as-is to create the cn=schema,cn=config subtree:

master$ mkdir ~/schema
master$ cp /etc/openldap/schema/*.schema ~/schema
master$ cat > ~/schema/extensions.schema <<'EXTENSIONS_SCHEMA'
attributetype (
	1.3.6.1.4.1.63.1000.1.1.2.16.1
	NAME 'authAuthority'
	DESC 'password server authentication authority'
	EQUALITY caseExactIA5Match
	SUBSTR caseExactIA5SubstringsMatch
	SYNTAX 1.3.6.1.4.1.1466.115.121.1.26 )

attributetype (
	1.3.6.1.4.1.63.1000.1.1.1.21.1
	NAME 'apple-acl-entry'
	DESC 'acl entry'
	EQUALITY caseExactMatch
	SUBSTR caseExactSubstringsMatch
	SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 )

objectclass (
	1.2.840.113556.1.3.23
	NAME 'container'
	STRUCTURAL
	MUST cn )
EXTENSIONS_SCHEMA

Looking at OpenDirectory’s cn=schema,cn=config shows that Apple uses only nine schema files (and one of them is empty, so I omitted it). Use slaptest to convert the .schema files to a cn=schema directory:

master$ cat > ~/schema/convert.conf <<CONVERT_CONF
include $HOME/schema/core.schema
include $HOME/schema/cosine.schema
include $HOME/schema/nis.schema
include $HOME/schema/inetorgperson.schema
include $HOME/schema/misc.schema
include $HOME/schema/samba.schema
include $HOME/schema/fmserver.schema
include $HOME/schema/extensions.schema
include $HOME/schema/apple_auxillary.schema
include $HOME/schema/apple.schema
CONVERT_CONF
master$ mkdir ~/schema/output
master$ slaptest -f ~/schema/convert.conf -F ~/schema/output

This converts the whole configuration file into a cn=config tree, but we’re only interested in the cn=schema,cn=config subtree:

master$ (cd ~/schema/output/cn\=config/ && \
    tar czf ~/cn_schema.tar.gz cn\=schema)
master$ scp ~/cn_schema.tar.gz replica:/tmp/

On the replica, we’ll need to blow away OpenLDAP’s default schemata and replace them:

replica$ sudo -i # Yes.
replica# service slapd stop
replica# cd /etc/ldap/slapd.d/cn\=config/
replica# rm -rf cn\=schema
replica# tar xzf /tmp/cn_schema.tar.gz
replica# chown -R openldap:openldap cn\=schema
replica# chmod -R g+rX cn\=schema # Files 0640, dirs 0750
replica# service slapd start
replica# exit

If slapd fails to start, run it in debug mode as root to see why:

slapd -h 'ldap:/// ldapi:///' -g openldap -u openldap \
    -F /etc/ldap/slapd.d/ -d 5

If I’ve omitted a schema element, or if you need to recover a schema element from the master, use the following query:

master$ sudo ldapsearch -x -H ldapi://%2fvar%2frun%2fldapi \
    -b cn=Subschema -s base '*' '+' | less

The output will include many entries of the form

attributeTypes: ( 1.3.6.1.4.1.63.1000.1.1.2.16.2 NAME 'authAuthority' DESC 'pa
 ssword server authentication authority' EQUALITY caseExactMatch SUBSTR caseEx
 actSubstringsMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.15 )

and

objectClasses: ( 1.2.840.113556.1.3.23 NAME 'container' STRUCTURAL MUST cn )

These are easy to convert to OpenLDAP schema entries. (In fact, go back and look at the contents of extensions.schema – these two entries should look familiar.) Find the ones you’re looking for, add them to extensions.schema, regenerate the schema directory as above, and blow away and replace the replica’s cn=schema,cn=config tree with the updated version.

Replication Consumer

Once the schemata are reasonably sane and slapd starts properly with them installed, we can create the replica. We’ll also need to create a database to hold it; we can do both in one step.

replica$ sudo ldapadd -Y EXTERNAL -H ldapi:/// <<'REPLICA'
dn: olcDatabase=hdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcHdbConfig
olcDatabase: {1}hdb
olcSuffix: dc=example,dc=com
olcDbDirectory: /var/lib/ldap
olcRootDN: uid=diradmin,cn=users,dc=example,dc=com
olcRootPW: DIRADMIN_PASSWORD
olcDbConfig: set_cachesize 0 2097152 0
olcDbConfig: set_lk_max_objects 1500
olcDbConfig: set_lk_max_locks 1500
olcDbConfig: set_lk_max_lockers 1500
olcDbIndex: objectClass eq
olcDbIndex: uid eq
olcDbIndex: cn eq
olcDbIndex: member eq
olcDbIndex: entryUUID eq
olcLastMod: TRUE
olcDbCheckpoint: 512 30
olcAccess: to attrs=userPassword
    by dn="uid=diradmin,cn=users,dc=example,dc=com" manage
    by anonymous auth
    by self read
    by * none
olcAccess: to dn.base=""
    by dn="uid=diradmin,cn=users,dc=example,dc=com" manage
    by users read
    by * none
olcAccess: to *
    by dn="uid=diradmin,cn=users,dc=example,dc=com" manage
    by users read
    by * none
olcSyncRepl: rid=100
    provider=ldaps://ldap.example.com/
    bindmethod=simple
    binddn="uid=diradmin,cn=users,dc=example,dc=com"
    searchbase="dc=example,dc=com"
    credentials=DIRADMIN_PASSWORD
    schemachecking=off
    type=refreshAndPersist
    retry="60 +"
    interval="00:00:10:00"
    tls_cacert="/etc/ssl/certs/ldap.example.com_cert.pem"
REPLICA

Replication will use the diradmin login you set up when you created the OpenDirectory master. Obviously, you can use a different user, but if you want passwords to be replicated that user needs to be a directory administrator.

The interval and retry settings can be tweaked; we’ve found that ten minute refresh intervals plus one minute retries on failure are a reasonable balance of currency and bandwidth use. If you need to force an immediate refresh for any reason, restart slapd on the replica.

The ACL given by the olcAccess attributes allows users to authenticate against the replica and read information from it, but disallows anonymous access and changes (except to the root DN for the database, which provides an escape hatch if you need to fix something by hand).

The database should replicate over immediately. If it doesn’t, start slapd up in debug mode (see above) and look at the errors. In particular, if you see a message similar to

syncrepl_message_to_entry: rid=100 mods check (objectClass: value
#0 invalid per syntax)

it means you’re missing some schema entries. The DN of the object it was trying to replicate is on the preceeding line that looks like

<<< dnPrettyNormal: <cn=users,dc=example,dc=com>, <cn=users,dc=example,dc=com>

and you can check the affected attributes against the OpenDirectory master.

Open Questions

The same remark about delta syncrepl applies here, too.

Testing

If you have a user in your OpenDirectory master, you can test the replica by running searches against it:

replica$ ldapsearch -x -W -D uid=user,cn=users,dc=example,dc=com \
    -b dc=example,dc=com '(uid=user)'

This should prompt you for the user’s password, then print back the user’s entry.

PAM and NSS Notes

OpenDirectory’s extensions sit on top of a mostly-normal RFC 2307 directory structure. Users are inetOrgPerson, posixAccount, shadowAccount objects, with some ancillary schemata to support the Apple features. The subtrees for users, groups, and so on are named cn=users rather than ou=users as in the RFC, so you may have to twiddle ldap.conf to recognize the correct subtrees.

OS X home directories are usually in /Users, not /home. It’s not really worth arguing with it; just create /Users home directories on your linux client boxes as well.

Again, only users with Crypt passwords in the OpenDirectory master will be able to use the replica for authentication.

Switch is a holdover from lower-level languages (mostly, various assembly languages) where the destination of a jmp instruction could be computed or looked up in a table. In modern languages, there are usually more appropriate structures for determining which piece of code to jump to: that’s what virtual dispatch does, for example.

Unfortunately, it’s not always obvious how to “fix” code that makes heavy use of switch.

I normally try to avoid this, but for the sake of clarity I’ve omitted a couple of good style points. Online presentation makes certain Java conventions difficult to present; just because the following examples use a single compilation unit with default-visibility symbols doesn’t mean you should, too.

Starting Points

Let’s take a simple example that uses a switch statement to control the state of some service. The surrounding context involves a simple control protocol for a light dimmer: it can either be turned fully off, turned fully on, or set to a “dim” state halfway between the two.

class States {
    public final int OFF = 0;
    public final int DIM = 1;
    public final int ON = 2;
}
 
class DimmerController {
    public void setIntensity(int intensity) {
        doStuffBeforeChangingIntensity();
        switch (intensity) {
            case OFF:
                disable();
                break;
            case DIM:
                beginDimCycle();
                break;
            case ON:
                beginFullCycle();
                break;
        }
        doStuffAfterChangingIntensity();
    }
 
    private void disable() {
        /* ... */
    }
 
    private void beginDimCycle() {
        /* ... */
    }
 
    private void beginFullCycle() {
        /* ... */
    }
}

Eminent Domain

There are a few obvious problems with the version above. For starters, there’s nothing preventing someone from passing the wrong value, either from the wrong set of constants or by hard-coding the literal values instead of using the constants. We could add a default clause that raises an IllegalArgumentException, but we’d still only find out about the problem at runtime. We’d really like to find out about errors in our programs as early as possible: compile time, or even as we type them. Most languages provide a tool for doing that; since we’re using Java, we’ll use an enum.

enum State {
    OFF,
    DIM,
    ON
}
 
class DimmerController {
    public void setIntensity(State intensity) {
        doStuffBeforeChangingIntensity();
        switch (intensity) {
            case OFF:
                disable();
                break;
            case DIM:
                beginDimCycle();
                break;
            case ON:
                beginFullCycle();
                break;
        }
        doStuffAfterChangingIntensity();
    }
 
    private void disable() {
        /* ... */
    }
 
    private void beginDimCycle() {
        /* ... */
    }
 
    private void beginFullCycle() {
        /* ... */
    }
}

On Your Best Behaviour

That’s better, but it still has most of the fundamental problems switch statements bring with them. While I’m only showing one switch, apps that have them tend to have multiple switches over the same set of constants, which leads to code duplication throughout the app and makes it hard to determine what a given constant will cause the system to do. In an object-oriented system, we normally try to keep behaviour with the data it works with, and as a side effect of refactoring to using an enumerated type, we now have real objects to use. Let’s add some methods to our enumerated type to handle the per-case behaviour.

enum State {
    OFF() {
        public void apply() {
            disable();
        }
 
        private void disable() {
            /* ... */
        }
    },
    DIM() {
        public void apply() {
            beginDimCycle();
        }
 
        private void beginDimCycle() {
            /* ... */
        }
    },
    ON() {
        public void apply() {
            beginFullCycle();
        }
 
        private void beginFullCycle() {
            /* ... */
        }
    };
 
    public abstract void apply();
}
 
class DimmerController {
    public void setIntensity(State intensity) {
        doStuffBeforeChangingIntensity();
        intensity.apply();
        doStuffAfterChangingIntensity();
    }
}

Opening the Set

You can stop here and have a reasonably well-structured program. The specifics of what each constant means are stored with the constant, and it’s now impossible to pass an illegal value into setIntensity. However, we’ve still embedded into DimmerController the assumption that we will only ever be setting the intensity of a three-state light – but we don’t actually use that assumption anywhere. We can refactor further to eliminate this dependency by realizing that State‘s public interface really represents any arbitrary dimmer state callback:

 
interface State {
    public void apply();
}
 
enum DimmerState implements State {
    OFF() {
        public void apply() {
            disable();
        }
 
        private void disable() {
            /* ... */
        }
    },
    DIM() {
        public void apply() {
            beginDimCycle();
        }
 
        private void beginDimCycle() {
            /* ... */
        }
    },
    ON() {
        public void apply() {
            beginFullCycle();
        }
 
        private void beginFullCycle() {
            /* ... */
        }
    };
 
    public abstract void apply();
}
 
class DimmerController {
    public void setIntensity(State intensity) {
        doStuffBeforeChangingIntensity();
        intensity.apply();
        doStuffAfterChangingIntensity();
    }
}

Everything In Its Place

On the other hand, sometimes the heavy lifting (in our case, the work of actually changing the intensity of a light) actually belongs outside the enumeration. This is true most of the time: enumerations can’t hold per-use state, and creating a new State instance (from the interface-based example above) for each light isn’t always appropriate. We can pull those methods back out of the enumeration by introducing a third class to act as a callback for the apply method:

enum State {
    OFF() {
        public void applyTo(DimmerCallback dimmerCallback) {
            dimmerCallback.disable();
        }
    },
    DIM() {
        public void applyTo(DimmerCallback dimmerCallback) {
            dimmerCallback.beginDimCycle();
        }
    },
    ON() {
        public void applyTo(DimmerCallback dimmerCallback) {
            dimmerCallback.beginFullCycle();
        }
    };
 
    public abstract void applyTo(DimmerCallback dimmerCallback);
}
 
 
class /* or interface */ DimmerCallback {
    private void disable() {
        /* ... */
    }
 
    private void beginDimCycle() {
        /* ... */
    }
 
    private void beginFullCycle() {
        /* ... */
    }
}
 
class DimmerController {
    // Or from dependency injection, or...
    private final DimmerCallback dimmerCallback = new DimmerCallback();
 
    public void setIntensity(State intensity) {
        doStuffBeforeChangingIntensity();
        intensty.applyTo(dimmerCallback);
        doStuffAfterChangingIntensity();
    }
}

Ok, But I Have This Integer

In most of the cases where people feel compelled to use magic ints or an enumerated type, there’s an external force at work. The most common scenario involves file formats or network protocols where an integer or short known string needs to map directly to program behaviour. With the behaviour separated out into a real type with testable features, you lose the ability to near-blindly pass the protocol value directly into your logic.

Remember when I said that switch statements are really a giant lookup table? That should give you a hint. You can still use a lookup table to convert from an external representation to an internal one; depending on how far down the chain of refactorings you’ve gone, you may even get one for free. Consider the following, assuming we still have a State enumeration:

public State parseRequestedState(int state) throws ProtocolException {
    try {
        return State.values()[state];
    } catch (IndexOutOfBoundsException ioobe) {
        throw new UnexpectedValueException(ioobe);
   }
}

Java’s enumeration support gives us the lookup table we need for free. We can do the same thing with strings, if they happen to match the enum constants’ names; alternately, we can use an explicit lookup table stored in an array or a HashMap. There are also plenty of strategies for automatically populating a lookup table using mechanisms like classpath scanning, ServiceLoader, annotation processing tools, and so on.

There are lots of places to go from here. For example, enum-based implementations of the state object pattern begin with a very similar refactoring. In languages without strong enums, the structure outlined here can be implemented using normal classes with a fixed pool of instances, instead (Java’s enum support does exactly that, under the hood), skipping directly to the interface-based version near the end.

1
2
3
4
5

<?php
while ($read < $n && (false !== ($buf = fread($this->sock, $n - $read)))) {
    /* ... */
}
?>

contains a subtle bug. Go ahead, read the relevant function documentation and see if you can spot it.

PHP’s fread function directly maps the C library’s fread function. As TBlue explains, the C fread function returns 0 when reading at the end of a stream. PHP does the simplest possible thing with this and returns an empty string; to determine that the stream is at end of file, programs must check with feof as well.

Most modern languages realize that explicitly checking for errors is a fairly error-prone convention. Java’s InputStream.read(byte[]) methods return -1 on EOF (and only on EOF) and raise exceptions on errors. Python’s File-like objects convention supports mechanisms other than raw byte reads (primarily, iteration) that make stopping at end of file implicit, on top of only returning an empty string at end of input (as with Java, Python raises exceptions when a stream read fails). Not PHP, though! In PHP, the loop above has to look like

1
2
3
4
5
6

<?php
while ($read < $n && !feof($this->sock) && (false !== ($buf = fread($this->sock, $n - 
$read)))) {
    /* ... */
}
?>

Encountered in the wild: php-amqplib bug #12.

The Singleton pattern specifically refers to the object creational pattern where a class manages a single instance of itself and, usually, forbids other instances from existing. There’s also the “singleton” scope in Spring, which means something more limited: Spring will only create one instance of the bean. Don’t confuse the arguments that Singleton classes are a bad idea for arguments that objects you only have one of are a bad idea.

I mention this because Spring scopes come up in the following examples, and the default Spring scope is also called singleton. Now, with that out of the way, on with the show.

Moving a Singleton creation dependency to Spring

Injecting a singleton into its clients involves replacing code like this:

public class Mailer {
  private static final Mailer instance = new Mailer ();
 
  public static Mailer getInstance () { return instance; }
 
  private Mailer () { }
 
  public void sendMail (String recipient, String message) {
    System.out.printf ("I am sending %s to %s!\n",
      message, recipient);
  }
}
 
/* Using a Struts2 Action as an example "client" */
public class SendMailAction {
  public void setRecipient (String recipient) {
    this.recipient = recipient;
  }
 
  public void setMessage (String message) {
    this.message = message;
  }
 
  public void execute () {
    Mailer.getInstance ().sendMail (recipient, message);
  }
 
  private String message;
  private String recipient;
}

with code like this:

public class SendMailAction {
  public SendMailAction (Mailer mailer) {
    this.mailer = mailer;
  }
 
  public void setRecipient (String recipient) {
    this.recipient = recipient;
  }
 
  public void setMessage (String message) {
    this.message = message;
  }
 
  public void execute () {
    mailer.sendMail (recipient, message);
  }
 
  private final Mailer mailer;
  private String message;
  private String recipient;
}

The corresponding Spring configuration file looks like

<beans xmlns="http://www.springframework.org/schema/beans">
  <bean id="mailer"
        class="ca.grimoire.examples.Mailer"
        factory-method="getInstance" />
 
  <bean id="action"
        scope="request"
        class="ca.grimoire.examples.SendMailAction">
    <constructor-arg ref="mailer" />
  </bean>
 
  <!-- more definitions -->
</beans>

If getInstance, for some reason, needs parameters, they can be passed as elements nested inside its bean definition.

You can now write isolated tests for SendMailAction:

// Using JMock rather than a hand-crafted alternate implementation.
@RunWith(JMock.class)
public class SendMailTest {
  // This requires some extension to work with class mocks
  // See http://www.jmock.org/mocking-classes.html
  private final Mockery mockery = new JUnit4Mockery() {{
    setImposteriser(ClassImposteriser.INSTANCE);
  }};
 
  @Test
  public void sendsExpectedMessage () {
    final Mailer mailer = mockery.mock (Mailer.class);
 
    mockery.checking (new Expectations () {{
      oneOf (mailer).sendMail ("Recipient", "Hello World");
    }});
 
    SendMailAction action = new SendMailAction (mailer);
    action.setRecipient ("Recipient");
    action.setMessage ("Hello World");
 
    action.execute ();
  }
}

Prior to this change, writing a test for SendMailAction that did not also involve Mailer would’ve been difficult and error-prone.

Injecting dependencies into Singletons

You can also inject dependencies, including other Singleton instances, into a Singleton instance, which is helpful for breaking Singleton graphs. Let’s give Mailer another Singleton it depends on:

public class SmtpConnection {
  private static final SmtpConnection instance = new SmtpConnection ();
 
  public static SmtpConnection getInstance () { return instance; }
 
  public PrintStream getSmtpStream () {
    return System.out;
  }
}
 
public class Mailer {
  private static final Mailer instance = new Mailer ();
 
  public static Mailer getInstance () { return instance; }
 
  private Mailer () { }
 
  public void sendMail (String recipient, String message) {
    SmtpConnection connection = SmtpConnection.getInstance ();
    connection.getSmtpStream ().printf (
      "I am sending %s to %s!\n", message, recipient);
  }
}

You can move this dependency into Spring if you change Mailer:

public class Mailer {
  private static final Mailer instance = new Mailer ();
 
  public static Mailer getInstance () { return instance; }
 
  private Mailer () { }
 
  public void setConnection (SmtpConnection connection) {
    this.connection = connection;
  }
 
  public void sendMail (String recipient, String message) {
    connection.getSmtpStream ().printf (
      "I am sending %s to %s!\n", message, recipient);
  }
 
  private SmtpConnection connection;
}

The corresponding Spring configuration looks like:

<beans xmlns="http://www.springframework.org/schema/beans">
 
  <bean id="connection"
        class="ca.grimoire.examples.SmtpConnection"
        factory-method="getInstance" />
 
  <bean id="mailer"
        class="ca.grimoire.examples.Mailer"
        factory-method="getInstance">
    <property name="connection" ref="connection" />
  </bean>
 
  <!-- more definitions -->
</beans>

Even clients who aren’t using Spring to inject the Mailer Singleton can be used with this, so long as the Spring injection happens before anything tries to use the Mailer.

Without an interface limiting the operations, though, clients of Mailer may decide to call setConnection – don’t do this because it becomes a form of mutable global state. If you’re going to follow this route, either force clients to use a narrower interface without the setters or be very careful about code review.

The Singleton pattern is one of the common solutions identified in Design Patterns. At its core, a Singleton is a class that only allows one instance. All clients of the class will operate against the same instance. The implementation outlined in Design Patterns provides a class-level function replacing the class’ constructor in its public API which manages the existence of the singleton instance.

A fairly stock Java Singleton implementation for, for example, sending mail runs like this:

public class Mailer {
  private static final Mailer instance = new Mailer ();
 
  public static Mailer getInstance () { return instance; }
 
  /* Note the access specifier: no other class can access this
     constructor. */
  private Mailer () {
    /* ... */
  }
 
  public void sendMail (String recipient, String message) {
    /* ... */
  }
}

From an implementor’s point of view, Singleton classes cost almost nothing, either in terms of extra code to maintain or in terms of assumptions to test. In the average application, each class providing some service to the application only has one instance, so constraining the class to only one instance isn’t a huge leap. These kinds of classes tend to be stateless, or almost stateless, so clients don’t necessarily need to be aware that they’re sharing instances with other clients as there are no extra synchronization or garbage collection hazards to handle. There are even times when Singletons are the right thing: either they accurately model some aspect of the problem where only a single representative can exist (for example, Java’s Runtime class, which exposes services provided by the JVM running the program), or the resulting API is compellingly simple and easy to use (for example, Log4J’s Logger, which is effectively backed by a Singleton registry of Logger instances).

Implementing a Singleton correctly can easily depend on the dark corners of the language’s object model. The most common source of Singleton implementation bugs are caused by language or deployment models that scope “global” state to only part of the system. Networked systems can easily have an instance on every node, and applications that have multiple AppDomains or ClassLoaders can easily create multiple Singleton instances in a single address space. In multithreaded apps (that is, almost anything written this decade), Singletons can also be threading hazards: if they manage access to an external resource, then it’s very easy to build a synchronization bottleneck into your entire application in the name of keeping access to that resource thread-safe.

Singletons cause most of the problems associated with them on the consumer side. The most common implementation pattern for Singleton clients is very simple: every occurrence of

Mailer m = new Mailer (/* ... */);

gets replaced with

Mailer m = Mailer.getInstance ();

This wins nothing: the client is tightly bound to a number of assumptions about the identity and lifecycle of the provider. In fact, it loses a little: instead of giving each client its own isolated state, now there’s a guarantee that some state is shared between client instances, via the Singleton instance. There are also knock-on effects: because the client controls exactly which provider it uses, it’s difficult to isolate only the client class for testing. It also becomes difficult, if not impossible, to insert behaviour between the client and the singleton provider – no AOP, no proxies, and no remoting, without making invasive changes to the client or the singleton.

Over the long term, as software evolves, patterns tend to propagate. Inline getInstance() calls for Singleton clients tend to get repeated in new code. In a sufficiently evolved codebase that uses Singletons extensively, a badly-placed getInstance() call can easily cause half the objects in the system to spring into being and initialize themselves. Even when this doesn’t cause performance issues on some code paths, it can easily cause startup and initialization order to become unpredictable during apparently-safe changes. Remembering which code paths are “supposed” to initialize singletons becomes an extra form of friction for developers working on the codebase.

Singletons aren’t popular by accident, though. In most modern object-oriented systems, there is a constellation of core classes which, in the final system, only need to exist once. The Mailer example I’ve been using is actually a good example: if it manages connections to the underlying mail system itself in a sane way, then there probably only needs to be a single instance of Mailer in the deployed app. If Mailer handles any connection pooling, then ensuring the creation of new instances is carefully controlled becomes almost mandatory. Singletons provide a really simple way to provide a single instance to an application.

Most development time, thought, and effort goes into the code paths that get used in the final product, so there’s an easy conceptual leap to make from “the system I’m building only needs one instance” to “this object should only allow one instance.” Developers without fairly extensive experience with mocking and other unit-testing techniques, or who don’t value the ability to insert new behaviour without changing existing code, can easily see no problem with using Singletons to manage lifetime and to provide access to the instance, since it’s convenient.

There are alternatives to Singletons for managing service objects: the standard Java examples are EJBs, which delegates control over object lifecycles to the app container, and Spring, which takes responsibility for lifecycle management and wiring between providers and consumers within the app. Other languages have similar tools available, either in the language or as a library. However, these tools have a higher barrier to entry than “just another Singleton”: even the simplest inversion of control implementation is more lines of code than a Singleton.

The kinds of problems Singletons cause only really become problems in medium-sized and larger codebases. Not coincidentally, medium-sized and larger codebases tend to be the product of long evolution from a small, relatively clean initial product, which means there’s been lots of time for people to forget the program’s overall structure. Refactoring to clean up the problems isn’t difficult, but surprisingly little has been written about it.

For most Singletons, where obtaining an instance is a parameter-less call to a getInstance()-like method, refactoring that call out to an injected dependency via a constructor or method parameter is a low-risk change that can be made and tested for one class at a time. Most, if not all, getInstance() calls follow a very simple, predictable code path, which means there are no side effects to take care of, and no changes at all have to happen in the provider. The only potential “gotcha” is the first call, which may do initialization work. This kind of change also has high rewards: injected dependencies can be replaced with alternate implementations, which makes testing much easier and lets you augment or replace the implementation without altering the clients.

There’s more risk in changing the Singleton itself first. Refactoring the lifecycle management code out of the Singleton and into a higher layer does help, but while clients are calling getInstance() directly, any change to the lifecycle management means changing every client at the same time, which in turn means you need to re-test all of those clients. Once the clients have been refactored, though, it’s much safer and easier to change the lifecycle managment.

Singletons are not the Great Evil they’re sometimes depicted as. However, they’re used in a lot of situations as a crutch to avoid writing real inter-object dependency management, to the detriment of software using them. Their convenience can outweigh the problems, especially in small systems or prototypes, but you need to have a way out when they start to cause problems.

Next time: implementation examples, for Spring.

The next time you think to reach for the “reformat source code” button in your IDE, think twice before you press it. Automatic reformatting is a great way to turn a small merge hazard contained to a single method into a huge merge hazard affecting an entire file. Merge tools are relatively stupid: in general, they’re not aware of the syntactic structure of a source file, just the text by lines; as a result, it’s very easy to fool them into thinking a one-line change and a reformat are a conflict covering the entire file.

If you’re using automatic formatting tools, please check with the other developers on your project and make sure that your code formatter settings agree with theirs. If you can’t make that happen, then don’t reformat code unless everyone’s nobody but you is touching those files.

Please, think of the merges.