Last week we launched a Cubelets Operating System update. While the point three makes it sound like a minor update, it’s actually kind of a big deal. OS 4.3 improves a bunch of little things, and it definitively fixes one particularly nefarious bug.
You know how we’re always talking about complexity and emergent behavior? About how Cubelets can be a great model for complex systems we see in the world like societies, ecosystems, and economies? About how play with Cubelets will help kids develop skills to solve problems in complex environments? Well, even though we play with Cubelets a lot around here, sometimes emergent problems in complex systems can be a real bear to figure out.
Take the Nefarious Bug, for instance. Since the release of OS4, we’ve seen a few Cubelets spontaneously lock up and stop functioning. It happened every once in a while in our factory, and we heard from a few customers that the same thing had happened out in the world. When we took these “bricked” Cubelets apart, we found that each one was missing a tiny little piece of memory — a random line of programming had somehow been erased. And while I can function pretty well having no idea what I ate for breakfast yesterday, the microcontrollers inside Cubelets are a little more brittle, their memories need to basically be perfect in order for them to function at all.
I’ll spare you the rundown of our debugging process. Suffice it to say that hypotheses were made, tested, and then re-made. Ideas and hopes and dreams were repeatedly generated and then dashed. It took weeks, but we finally figured it out. It was the combined behavior of a mechanical bug, an electronic bug, and a software bug, but we fixed it just with software.
The particulars of the problem are a bit obscure. Are you familiar with the Rotate Cubelet? If you’ve ever wondered how the rotating face can conduct power and data without a bunch of wires getting twisted up inside, the answer is that it uses a slip ring. Slip rings have contacts that slide around a rotating ring, using friction to make electrical contact. Sometimes this connection isn’t perfect (at least not as perfect as a soldered wire) so there tends to be some noise on the power line near Rotate Cubelets. While this shouldn’t be a problem, a big Cubelets robot has power flowing throughout all of the Cubelets and through all of the faces, and a bunch of noise on a bunch of connectors all at once ended up causing low voltage transients that wreaked a little havoc on those tiny, tiny brains.
Previously, Cubelets were riding a fine line between the voltage level at which they’d automatically shut down and the voltage level at which they could brick themselves and apparently (in less than 0.1% of Cubelets), noisy rotatey power connections could shock the Cubelet into crossing that line. Once we finally figured this out, it was pretty straightforward to repair by changing a few settings in software and testing profusely with the noisiest Rotate Cubelets we could find.