One of the common questions about switching to Karma Go is "Will I be able to keep my data?" The answer is a resounding yes. Data you've bought works with the existing Karma, the upcoming Karma Go, or even both at once if you'd like. The data is tied to you and your user account, never to a specific device. We're sorry if that wasn't more clear!

Specifically, data is tied to the user account you create with your email address and use to purchase that data. When you login with that email address to any Karma device, either your own device or someone else's, you can use data you've purchased for that account.

When you use Karma, the amount of data you consume is tracked by our backend servers so we can deduct that amount from your data balance. The device sends a message to the backend server to see if there’s still data left on the account of each user using that device. The message, sent every minute, is something like “Is Paul cool? And Sally, too? Keep giving them internet, yeah?” If Sally runs out of data, the device will be notified, and she'll be logged out, but Paul can keep browsing. Sally should’ve received an email or app notification of her impending data shortage, and as soon as she adds more to her account she can get back to browsing.

“Is Paul cool? And Sally, too? Keep giving them internet, yeah?”

I just want to clarify here, we don't want to stand between you and your internet connection. We're not routing your traffic through our servers, or looking at where you're going to on the internet, we're just tracking the quantity of bytes you consume so we know how much data you have left. In fact, we're so committed to being a "dumb pipe," as it's called, we have specific terms in our contract with Sprint that all data is treated as equal.

What glues this whole system together is the messages passed between the device and our backend servers. We call it a “heartbeat,” and it includes a lot more information than whether or not Paul and Sally are cool. For instance, info about the device’s battery level and signal strength are sent to the backend, which could then pass them on to the iOS or Android app for display to the user.

Because every device reports back to the mothership every minute, and we have thousands of devices doing this all at once, a big part of our engineering effort is transmitting, processing, and responding to this data. The messages not only need to be dependable, they need to be fast: because we use them to authenticate users, the device can't just fire messages off into the void, it needs a rapid response.

The messages not only need to be dependable, they need to be fast

For the original Karma, we used HTTP + TLS and JSON. HTTP is the same technology your browser used to access this webpage and TLS refers to the security protocol, while JSON is the most popular format for encoding information on the web. They're all marvelous and well-documented technologies, and have worked well enough so far, but for the sorts of messages we send they're a little heavy. We pay for the bandwidth required by the heartbeats, so the smaller the better.

So, for Karma Go, we've switched to MQTT and Protocol Buffers, alternatives to HTTP and JSON, respectively. The upshot of all this jargon is that we've reduced the bandwidth required by the heartbeats by roughly 50-60%.

"It allows us to scale into the future," explains Stefan Borsje, our CTO. As CTO, he makes the decision about this sort of architectural change, and, as CTO, he gets to do plenty of the implementation work. I sit a couple of desks away from Stefan at the office, and have had him explain MQTT to me on multiple occasions. As someone with little patience for technical challenges, I'm always surprised he's so excited about MQTT: it originated in the enterprise, with IBM, and therefore reference implementations are scarce, and documentation is lacking. Facebook uses it for its Messenger app, but we’re not Facebook. Somehow, I get the feeling that Stefan enjoys this challenge: building a Facebook / IBM-type solution for our little internet company.

And the challenges don't end with navigating the MQTT protocol and reading / writing Protocol Buffer messages. All that data has to flow into the rest of our backend, and needs a rapid response. We also have all of the legacy HTTP / JSON stuff around, to serve original Karma devices.

"There's a lot of moving parts involved," admits Stefan.

As the co-founder of Karma, Stefan naturally extols the merits of having a technical co-founder at a startup: "If you don't have someone on the technical side, it's really hard to figure out what's possible and impossible." Knowing that implementing MQTT was possible, and worth the challenge, got us where we are today.

The new, lightweight messages let us be more secure, and give us a finer-grained control of the service. The extra amount of moving parts adds complexity to our system, but it also gives us scalability. "Having so many parts makes different parts tweakable," explains Stefan. "So we can swap out a component if one turns out it doesn't scale, instead of having to find an entirely new solution."

Stefan detailed more of the reasoning behind this decoupling, and our overall architecture, in a blog post a few months ago.

I asked him how hard it would be to build this architecture from scratch. "It sounds ridiculous! This isn't something you can easily copy." We didn’t start out building a system like this, that would’ve been pointless at the beginning, but our original, simple systems were designed to be replaced by something more complex once it became necessary. Stefan pointed me to a recent post by Martin Fowler about "sacrificial architecture" that encapsulates this thinking.

"It sounds ridiculous!"

For Karma, MQTT was a natural progression. Adding it to our architecture was the next step we needed to add a new device to our system and serve a bunch of new users, and lets us continue to scale into the future.

"We could've done it just like everybody else does," explains Stefan. "But that's not our business model." Karma lets you run an open hotspot, lets only you use your data, and allows you to connect to any other Karma.

If all that was easy to do, maybe we wouldn't be the only ones.

About Paul Miller

That guy who left the internet for a year