Last week, some curious QR codes started to circulate on social media: obviously false EU digital covid passes, issued to Mickey Mouse and Spongebob Squarepants. Initial reactions were varied: some jumped to the conclusion that keys must have leaked, and predictably with a hot-button political issue, some commentators were quick to claim that this would surely spell the demise of the CoronaPass system. After all, why should anyone trust the passes anymore, now that it’s been shown that they can be forged?

If you’ll bear with me for a little while, I’d like to dive into how the CoronaPass system works on a technical level, look at what can be done to recover from the breach that allowed the false passes to be issued, and show the surprising parallels to IoT software update systems along the way, to see what lessons we can learn from the CoronaPass breach.

So first things first, let’s take a look at how the CoronaPass works. (It’s officially called the European Digital Covid Certificate, or DCC for short, but we’re going to keep on referring to it as a CoronaPass for simplicity.)

I’m going to try to keep this relatively brief, so I’m going to assume that you already have some basic familiarity with concepts like digital (i.e. cryptographic) signatures and JSON documents.

• The people checking the CoronaPass aren’t going to be highly trained health professionals. The verifiers are going to be busy mâitre d’s, ticket takers at gates of concerts and sporting events, and so on
• All 27 EU member states will need to be able to issue the certificates. The health care systems are wildly disparate across EU countries. Denmark, for example, has an extremely computerized, centralized system of health records. Just across the border in Germany, the health system is much more privatized, with paper-filled filing cabinets at local private practices still being the norm in many places
• There will be different rules for what makes a CoronaPass "green" in different countries, so the system needs to accommodate that without having to re-issue certificates
• The system needs to work offline: there are plenty of scenarios where you might need to check the CoronaPass where cell coverage is spotty
• The system should be able to issue long-lived certificates that can be printed on paper: people who don’t have smartphones or internet connections also need access to the CoronaPass
• The system needs to respect individuals' privacy: the goal is to make a system to allow people to check corona status in day-to-day life, not to make a big centralized database full of private health care information
• The system needs to be secure: if passes can be forged, the public health goals may be undermined.
• The system needs to be developed and deployed very, very fast, in 27 countries all at once, and it needs to be durable: once it’s rolled out, you won’t be able to make any major changes to the specification without inconveniencing hundreds of millions of people and reducing trust in the system

Given all of these constraints, especially the time constraint, the CoronaPass is an exemplary model of good security engineering and specification.

We can already see some similarities with the problems and design goals of a software update verification system: when we want to be able to send out a new version of an operating system or application package on an IoT device, we need the device to check and make sure it’s actually supposed to be installing the update. The update client on the device takes the role of the mâitre d', checking that the software it’s being told to install is actually valid.

The specification the EU came up with is effective, simple, and elegant. It’s easiest to show in a picture:

To make a CoronaPass, you create a JSON document in a specific format. The JSON document is pretty much just metadata about a test, vaccination, or recovery from covid. It’s structured in a format defined in an open specification from the EU. The JSON for a negative test looks like this, for example:

{
    "dob": "1980-01-14",
    "nam": {
        "fn": "Ebraert",
        "fnt": "EBRAERT",
        "gn": "Peter Paul Maria",
        "gnt": "PETER<PAUL<MARIA"
    },
    "t": [
        {
            "ci": "01BEVLBDIML2KR1IFKPPCSX8OQF3#K",
            "co": "BE",
            "dr": "2021-05-15T19:21:22+00:00",
            "is": "SC-BE",
            "sc": "2021-05-25T09:02:07+00:00",
            "tc": "Test centre west region 245",
            "tg": "840539006",
            "tr": "260415000",
            "tt": "LP6464-4"
        }
    ],
    "ver": "1.0.0"
}

Then, you add a signature, zip up the data, and encode it in base45 inside a QR code.

To verify the CoronaPass, you just need an app to unzip the data, check the signature to make sure it’s valid, and display the information from the JSON document in a way that’s easy for a human to read, usually along with a green check-mark showing that it’s valid under whatever rules and restrictions are currently active in that country (for example, at a port of entry from a high-risk zone, it might show green for a test certificate only if the test was taken in the last 48 hours).

The CoronaPass checks, both human-verified and app-verified, also bear a strong resemblance to the checks that a good IoT software update client should do: On the app side, we check the validity of the signature along with various other automated checks to see whether the update should be trusted or not. The device should also check to make sure that the update instruction was actually issued for this specific device, and not just some replayed set of installation instructions for a different device. That’s very similar to the human checks that CoronaPass verifiers are supposed to do: the person checking the certificate should check the photo ID of the person presenting the CoronaPass and make sure that the name and date of birth on the CoronaPass matches the photo ID of the person presenting it.

You might have noticed that in the previous section, I skipped over a really important question: who is it who signs the CoronaPass, and how do the verification apps know they should trust the signatures?

Well, as mentioned before, this needed to be a system that would work for 27 different countries with really different ways of doing things. It also needed to be deployed fast. The EU had to make some hard choices here, and here again I think they made the best choices they could under the circumstances. They used a system of X.509 PKI—the same thing that makes it possible to have a secure TLS/HTTPS connection with your bank or any other website—but designed in this case specifically for issuing CoronaPasses.

The implementation is very similar to what you’re used to if you know how Certificate Authorities work. Each individual country has a Country Signing Certificate Authority (CSCA), which issues Document Signer Certificates (DSCs). The DSCs issue the actual CoronaPasses:

Each country also needs to implement an API that lists which DSCs should currently be trusted, so that the verification apps can check and update their list of certificates.

1. The root certificate authority for each country (generally maintained and controlled extremely carefully, with private keys kept offline)
2. Individual signing certificates for each health authority that is allowed to issue CoronaPasses
3. The published list of the currently-trusted DSCs. There’s an unofficial collated list of all countries' trusted DSCs here. You can see the difference between Denmark and Germany’s health systems, for example: Denmark and Sweden have exactly one DSC each (the central health authority), while Germany has 57 and France has 33

So, for an app to trust a CoronaPass, it needs to check that the pass is signed by a DSC that the app knows about, and it also needs to make sure that the DSC is signed by a CSCA. It should check relatively frequently for updates to the list of CSCAs and DSCs, but exactly how often it should check this isn’t mandated by the spec.

Interesting side note for X.509 nerds: You might wonder why the EU settled on using this JSON API for publishing the list of DSCs to trust, instead of just using CRLs (certificate revocation lists). It appears to be because of uncertainty about whether implementors would actually get CRLs right: X.509 is a complicated standard, and every additional part of X.509 PKI you decide to support will always increase the risk of implementation problems. That’s also one of the reasons why TUF and Uptane--the gold-standard specifications for how to do software update security right--elected not to use X.509 PKI as the basis of trust. (There are other reasons too, but for a deeper dive into that topic you’ll have to come back later this month and read my forthcoming series of blog posts on OTA software update security in IoT.)

More parallels to be drawn here with IoT software updates. To keep your update system secure, you need to have a tiered system of trust, and a standardized and secure way to distribute the list of keys you trust for different roles. For CoronaPasses, there are only two tiers (and two roles) of signing authority: the root authority which issues the signing certificates for health authorities, and the health authorities that sign the CoronaPasses. On the Torizon Platform, we have a similar tiered setup, but with a few more roles that allow us to verify some extra security properties. We’ll get into those details more in the forthcoming series of blog posts about OTA software update security for IoT. Importantly, we also have a way to securely distribute new lists of trusted authorities. It’s more robust than the EU’s mechanism, because it’s in-band as part of the Uptane protocol, and that means that guarantees about checking for revocations or updates to the trust list are enforced on every update check.

• In Denmark (or at least, in Zealand and the capital region), most people use a smartphone app (and website) called MinSundhed. It hooks into your digitized, centralized health records, and provides a wide variety of services related to health care: push notifications and emails when it’s time to schedule a checkup, getting assigned a regular GP, booking any non-emergency appointments. So it’s not surprising that they also issue CoronaPasses directly through the app: they can verify with the central database whether you’ve been vaccinated, your login to the app is tied to your national ID, everything is quite straightforward. You don’t need to get a human to check your proof that you’ve been vaccinated or not, because the central system is trusted for everything.
• In Germany, where I live, there’s no such thing. I got my first vaccine dose at a small local doctor’s office, and my second one at an ice hockey arena that was turned into a mass vaccination center. Both times, it was recorded in my Internationale Bescheinigung über Impfungen und Impfbuch, a passport-like paper booklet where doctors can sign off on your vaccination record (for all vaccinations, not just Covid-19). To get my digital CoronaPass issued, I just needed to take the booklet to any pharmacy. The pharmacist took the booklet and my photo ID, verified the information in the booklet, and in about 3-5 minutes I had my QR code.

Now, I mentioned above that Germany has 57 different Document Signer Certificates issued. But there are a lot more than 57 pharmacies in Germany, and the fact that I was able to get my CoronaPass in under 5 minutes certainly suggests that there must be an automated system for signing them.

And that’s exactly the case. There’s an app/website, run by the Robert Koch Institute, where health professionals can get access to. I don’t know exactly how it works, but I think it’s safe to assume that they’re assigned individual credentials, and then they just copy the handwritten info (and the info encoded in the little QR code sticker that goes in the booklet) into that website, and an automated system on the backend generates the JSON, signs it, zips it, and spits out the QR code.

The parallels to doing OTA software updates in IoT—and specifically on the Torizon Platform—continue here. Torizon provides a web portal for you to log into, and through the web portal you can send software updates to devices, and our backend servers will automatically sign the individualized installation instructions for each device. In the default configuration, we can even sign software images you upload, authorized by your login account—but only if you decide to let us. You always have the power to revoke our keys, which means you can still use Torizon to deliver updates automatically…but new software can only be published when you decide to sign it with your own key.

1. She logs into the Robert Koch Institute portal for issuing CoronaPasses
2. She inputs my data into the system, and verifies that it matches what’s written in my vaccination booklet
3. She submits the data to the RKI site
4. The RKI site, once it’s validated that my pharmacist has logged on, accepts her request, and uses the private key on its Document Signer Certificate to sign my blob of JSON. Hopefully, it logs every certificate it issues—it can do that without saving any of my personal info on the server, because each certificate must have a unique ID.
5. The pharmacist prints out the QR code, and I add it to my app.

• Steal the login credentials of someone who has access to the RKI site, and use them to issue the bad certificates
• Get remote control over the pharmacist’s computer (via a virus let’s say) and then use her already-logged-in computer to issue some CoronaPasses
• Compromise (i.e. hack into) a signing server
• Steal a signing key and use it to sign whatever certificate they want

Let’s look at how we might be able to tell what happened, and figure out what response is most appropriate in each case. I’m going to start with the most serious, and move down from there. After I talk about how the CoronaPass system would need to respond, I’ll compare it to Torizon OTA with Uptane, and how you would recover from a similar situation.

The initial reporting on this issue claimed that private keys had been stolen or compromised. A user on 4chan claimed they had a signing cert/private key, and offered to issue a fake certificate under any name to prove it. But that doesn’t actually prove anything—compromised credentials to the signing portal would let you do that. If they actually had the private key and were interested in proving it, they could have done so very easily just by signing something that’s not a CoronaPass—say, the text of an article from today’s paper. The fact that they didn’t do so is strong evidence that no private keys were compromised.

But let’s imagine for a moment that the key really was stolen. How bad would that be?

Well, pretty bad. Whatever key was stolen, you’d need to remove it from the published list of trusted DSCs. That means that the verification apps would stop trusting every single certificate that signing authority issued. That could be literally millions of people! And every single one would have to go back through whatever process they did the first time to get a new one. For me, that would just mean digging out my vaccination booklet again, but what if there are people out there who got their digital certificate by presenting a note from their doctor, and then threw away the paper original once they got the digital one? Those people aren’t going to be happy.

This is an important lesson in how to deal with disaster recovery, and especially key rotation. The absolute worst-case scenario for a software update system is that an attacker manages to gain control of enough keys/capabilities that they can install any arbitrary malicious software on the fleet of devices. If this happens, and you don’t have any rock-solid way of guaranteeing what software the device is running, you might need to issue a recall—after all, a sophisticated adversary in full control of the software on the device would just modify the update client to lie to the server about what software it has installed.

How do we handle this in Torizon OTA? It’s a multi-pronged approach, and I’m going to give one last plug for the upcoming blog series on IoT software update security because I’ll be addressing it in more detail there. But the basic idea is this: resilience. Torizon OTA implements Uptane, and so it gives you the flexibility to adapt your security practices according to your stage of development. Once you’re in production and releasing new software updates less frequently, it might make sense to rotate your software signing key—perhaps to a hardware device like a Yubikey, that you can lock away in a safe deposit box somewhere, and only get out when you do the signing ceremony for a new release. (Uptane is designed for one of the most challenging, adversarial, and safety-critical security environments in the world: automotive software updating.) You can really get as paranoid as you like. But when you’re in development and pushing multiple builds every day, we can handle key management for you. Because Uptane has key rotation and role separation built into the protocol right from the start, you’re much less likely to have a problem like the one the CoronaPass system would have if they needed to revoke a certificate that had been used for millions of CoronaPasses. If you find you need to rotate out a key, your devices will automatically update to the new one once you’ve made it available on the server, using a chain of trust that extends all the way back to the root.

Let’s move to the next possibility down on the severity list: what if the signing server itself was hacked? Let’s assume the CoronaPass forger wasn’t able to extract the signing key because it was in an HSM, but they were in complete control of the server.

Unfortunately, this very well could have the same consequences for innocent CoronaPass holders as getting the private key stolen. Why? Logging. If the attacker was in complete control of the server, including being able to make signing requests to the HSM, they might be able to prevent the signing of the fraudulent certificates from being logged at all, or perhaps just hijack user sessions to make the fraudulent CoronaPasses blend into the background. That would put us back in the same scenario of losing the private key: there would be no way to know which ones were fake, so you’d have to just stop trusting that key and force everyone to get a new one once the server’s rebuilt. In this scenario, the devil would really be in the details. What if a thorough forensic analysis of the server revealed that only 400 fraudulent passes were issued out of 100,000? In that case, you might decide that the harm of 400 fake passes is less than the harm of inconveniencing 99,600 people, and decide to just live with it.

Now let’s imagine that Toradex’s server infrastructure was completely compromised, or a disgruntled employee with high-level access went rogue. (Of course, we have appropriate security measures in place to ensure that won’t happen. But let’s do the thought experiment anyway, because we think it’s important to always plan for the worst.)

In Torizon OTA, we mitigate against the possibility of server/infrastructure compromise—in a production-hardened configuration, you don’t have to worry. The worst thing the attacker could do is tell your fleet of devices to install older versions of your software. Why? There are two completely independent chains of trust in Torizon OTA.

The first chain of trust is the one that authorizes what the list of potentially valid software is. You can think of it as your own personal secure software repository—it might have old versions hanging around, but the only software in it is software that you decided (at one point or another) should be installed—that’s why you signed it. (And of course, you can always delete and revoke older versions that should never be installed anymore.) In this chain of trust, you can keep the keys offline, set up n of m multi-sig enforcement (e.g. have 7 signing keys and always require that at least 4 of them sign new software), and a whole lot more—and it will get securely verified by the update client on the device before it downloads a single byte of the actual binaries. (The binaries, of course, are checked once they’re downloaded, to make sure they match the verified metadata.)

The second chain of trust is the one that signs specific update instructions for specific devices. When you send an update to a device, we create and sign metadata on-demand that tells the device what it should install, and the device cross-checks that against the other independent chain of trust to make sure that it’s not being directed to install something invalid.

I’m going to lump a couple things together here: hacked/stolen account credentials, and fraudulent use by an authorized user. This is by far the most likely scenario for the forged CoronaPasses. There are over 50,000 pharmacists in Germany alone, so the chance that some of them leave their computer password on a post-it note on the monitor in an unlocked office is pretty high. The good news is, this ought to be fairly easy to track down, and then recover from.

The CoronaPass signing systems are required to log the IDs (a long semi-random string—no personal info) of every certificate they sign. So if it’s one or a few rogue user accounts issuing these fraudulent passes, the perpetrators can be tracked down, and only the passes issued by the rogue accounts need be revoked, affecting far fewer people.

In a similar scenario with Torizon OTA, you’d have an even easier time tracking everything down. If a rogue user were in control of your Torizon Platform OTA account, we’d be able to check our server logs to see exactly what they did--if they installed anything, for example--and quickly recover everything back to normal. Solid audit logging is a legal requirement for production automotive use cases, and so we intentionally don’t make it possible to delete logs of installation and update history, new software additions, and so on.

I had a lot of fun learning about how the CoronaPass works, and if you stuck with me all the way to the end, thanks for reading! I hope you found the parallels to software updating as interesting as I did, and maybe learned something about that domain as well. There’s one final thing I want to mention, though. I talked a lot in this piece about the technological measures that the EU implemented, and that our team at Toradex implemented while building Torizon OTA. But every system of trust, in the end, depends on a human. Cryptographic measures can protect, authenticate, or encrypt bits we send, but the systems will always depend on people following best practices (or at least simple instructions). When it comes to the CoronaPass, we depend on the people checking the pass to actually scan the QR code instead of just glancing at it and poking around the app on the customer’s phone, and to check IDs to make sure they match the data in the code.

If you want to dig deeper into the OTA topics, stay tuned here for content that takes it a bit slower and easier—I’ll be co-writing a series of shorter blog posts with my colleague Drew Moseley over the next months, introducing OTA security concepts for IoT from the ground up, and in more bite-size chunks.

Author:
Jon Oster, Platform Development Lead, Toradex

Leave a comment