Repeatable performance tests: CPU options are best disabled

| |

In an effort to improve repeatability, the MongoDB Performance team set out to reduce noise on several performance test suites run on EC2 instances. After thinking about our assumptions and the experiment setup, we began by recording data about our current setup and found no evidence of particularly good or bad EC2 instances. In the next step, we investigated IO and found that EBS instances are the stable option for us. Having found a very stable behavior as far as disks were concerned, this third and final experiment turns to tuning CPU related knobs to minimize noise from this part of the system.

Repeatable performance tests: EBS instances are the stable option

| |

In an effort to improve repeatability, the MongoDB Performance team set out to reduce noise on several performance test suites run on EC2 instances. After thinking about our assumptions and the experiment setup, we began by recording data about our current setup and found no evidence of particularly good or bad EC2 instances. However, we found that the results of repeated tests had a high variance. Given our test data and our knowledge of the production system, we had observed that many of the noisiest tests did the most IO (being either sensitive to IO latency, or bandwidth). After performing the first baseline tests, we therefore decided to focus on IO performance through testing both AWS instance types and IO configuration on those instances.

Repeatable performance tests: EC2 instances are neither good nor bad

| |

In an effort to improve repeatability, the MongoDB Performance team set out to reduce noise on several performance test suites run on EC2 instances. At the beginning of the project, it was unclear whether our goal of running repeatable performance tests in a public cloud was achievable. Instead of debating the issue based on assumptions and beliefs, we decided to measure noise itself and see if we could make configuration changes to minimize it.

After thinking about our assumptions and the experiment setup, we began by recording data about our current setup.

Reducing variability in performance tests on EC2: Setup and Key Results

| |

On the MongoDB Performance team, we use EC2 to run daily system performance tests. After building a continuous integration system for performance testing, we realized that there were sources of random variation in our platform and system configuration which made a lot of our results non-reproducible. The run to run variation from the platform was bigger than the changes in MongoDB performance that we wanted to capture. To reduce such variation - environmental noise - from our test configuration, we set out on a project to measure and control for the EC2 environments on which we run our tests.

Causal guarantees are anything but casual

| | distributed systems

Traditional databases, because they service reads and writes from a single node, naturally provide sequential ordering guarantees for read and write operations known as "causal consistency". A distributed system can provide these guarantees, but in order to do so, it must coordinate and order related events across all of its nodes, and limit how fast certain operations can complete. While causal consistency is easiest to understand when all data ordering guarantees are preserved – mimicking a vertically scaled database, even when the system encounters failures like node crashes or network partitions – there exist many legitimate consistency and durability tradeoffs that all systems need to make.

MongoDB has been continuously running — and passing — Jepsen tests for years. Recently, we have been working with the Jepsen team to test for causal consistency. With their help, we learned how complex the failure modes become if you trade consistency guarantees for data throughput and recency.

Pruning Dynamic Rebuilds With libabigail

| | c++ open source optimization

Complex C++ projects frequently struggle with lengthy build times. Splitting a project into multiple dynamically-linked components can give developers faster incremental rebuilds and shorter edit-compile-test cycles than relying on static linking, especially when there are a large number of test binaries. However, build systems usually do not realize all of the possible gains in dynamic incremental rebuilds due to how they handle transitive library dependencies. Red Hat's ABI introspection library libabigail offers one possible path to eliminating unnecessary transitive re-linking for some classes of source modifications.

Considering the Community Effects of Introducing an Official MongoDB Go Driver

| | golang drivers open source

What do you do when an open-source project you rely on no longer meets your needs?  When your choice affects not just you, but a larger community, what principles guide your decision?

Submitting patches is often the first option, but you're at the mercy of the maintainer to accept them.  If the changes you need are sweeping, substantial alterations, the odds of acceptance are low.  Eventually, only a few realistic options remain: find an alternative, fork the project, or write your own replacement.  Everyone who depends on open source faces this conundrum at one time or another.

After relying for years on the community-developed mgo Go driver for MongoDB, MongoDB has begun work on a brand-new, internally-developed, open-source Go driver.  We know that releasing a company-sponsored alternative to a successful, community-developed project creates tension and uncertainty for users, so we did not make this decision lightly. We carefully considered how our choice would affect current and future Go users of MongoDB.

Investing in CS4All: One Year Later

| | cs4all education

When a couple of New York City high school teachers partnered with MongoDB to teach computer science, did they succeed? Their curriculum was untested, and they were teaching in difficult districts where most students are from poor and minority families. I talked with these two teachers, Jeremy Mellema and Timothy Chen, back in September, when they had completed a summer fellowship at MongoDB and had just started teaching their curriculum; at the end of the academic year this spring, I visited Jeremy and Tim again to find out the result.

Their successes were sparse and partial. They discovered that their students' poor reading skills were a barrier to learning to code, and that teaching new coders how to solve problems is, itself, an unsolved problem. With a coarse unit of iteration—a school semester—it is painfully slow to experiment and find teaching methods that work. But even partial wins make a difference for individual kids, and the support of professional engineers at companies like MongoDB can be a powerful accelerant.

Farewell, Solaris

| |

[Update 8/29/17 3:45pm: edited to contain more specifics regarding Solaris downloads and security patches.]

Solaris was the first “real operating system” I ever used.

The Brown University Computer Science Department was a Sun Microsystems shop when I was an undergraduate there in the late 90s. When I took the operating systems lab class, CS-169, we implemented a toy version of Sun’s research operating system Spring OS. Several of my contemporaries in the CS Department went on to work at Sun, and developed or advanced many of the technologies that made Solaris great, like ZFS, dtrace, libumem, mdb, doors, and zones. The Solaris Linkers and Libraries Guide remains one of the best ways to develop an understanding of shared library internals. The first startup I worked for developed on Solaris x86, because the team knew Solaris well. Today, many of my co-workers on the server engineering team here at MongoDB share that formative experience with Solaris. We have a great deal of collective nostalgia and appreciation for Solaris and the amazing engineering effort that went into its development.

So it is, for many of us at MongoDB, bittersweet to announce that MongoDB is terminating support for Solaris.

Effective immediately, we plan to cease production of new builds of MongoDB for Solaris, across all supported versions of MongoDB. Existing release artifacts for Solaris will continue to be made available, but no new releases will be issued, barring a critical issue raised under an existing support contract covering MongoDB versions 3.0 through 3.4 running on Solaris. We will continue to fix critical flaws for the community, regardless of where found or how reported. Anyone can report a security vulnerability by using our Security project to create an account, then a ticket, describing the vulnerability.

This was not an easy decision for us to make, and we feel that it is important to provide some background on why we have made what may seem at first to be a capricious decision.