Dark Matter and Measuring Security

I am occasionally asked by our clients to measure how secure a thing is. That is perfectly reasonable to want to know. Is it secure enough? Do we need to spend more on security to make it secure enough? Are we getting better or worse? And so, managers are surprised, as well as disappointed, to learn that measuring security is nearly impossible.

Security is not visible. It is kind of like dark matter, in that you cannot see it directly, you can only detect it by its absence. The absence of security, i.e. vulnerability, can be detected and measured in a multitude of ways, such as vulnerability counting, defect density, or days of vulnerability exposure. You can measure how vulnerable a thing is, but that is a lower bound on how vulnerable it is, it could be much worse.

What do we know if we measure vulnerability? We know that the thing has at least “this” set of vulnerabilities. But we don’t know how many others there might be, and we especially don’t know how many there are in the places we haven’t looked, or worse, places we haven’t thought of. Thus, the question itself of “how secure is it?” is effectively a divide-by-zero error, the ratio of its attack surface divided by its vulnerabilities, and when we get to “secure” (no vulnerabilities) the result is defined as not-a-number.


A common way to try to measure product vulnerability is to count the number of vulnerability disclosures against a product, and that data is easily available. However, that data produces skewed results. The number of vulnerabilities disclosed in a year is as much a function of how much security attention the products received as it is the vulnerability of the product. Thus, counting vulnerabilities is considered a very bad security metric because it measures the environment more than it does the products.

whack a mole.png

You can measure how often intrusion attempts are caught by automation, but this is very similar to measuring vulnerability disclosure rates. What this is really measuring is how much pre-attack research the attacker invested in before attempting to breach. An attacker who is bulk rattling door knobs will very often trip automated alarms, and not care, because they just move on to find someone else. An attacker with a very focused interest in a specific target will learn what automation is in place before attempting anything, and thus is unlikely to trip automated alarms.

faster than.jpg

This is not to say that you can’t measure some aspects of security. For instance, you can measure how quickly a vendor responds to vulnerabilities found in their products, how quickly an IT department responds to security incidents, and the volume of attack incidents. But these are just instances of measuring vulnerability, none of them measure your security. As David LeBlanc says, there is no “secure”, there is only “insecure” and “good enough for now.”

So, if you ask how secure a thing is, and the answer is a vulnerability assessment, then that begs the question “did you check all the things?” Did whoever did the vulnerability assessment really check everything that could be a threat?

“Everything” is a long list. Borrowing a glib phrase from Andy Lowry, “Had the Sliver Surfer not befriended Galactus, Galactus would have consumed the Earth, causing most terrestrial computers to fail.” Thus, getting a usable “everything” requires us to construct a reasonable threat model that hypothesizes an attacker and their capabilities (what the attacker already has access to, such as your web site) and what the attacker’s goals are (e.g. steal your password list).

If a product or service is online, then the threat model must at least include “the Internet”, because all attackers have Internet access. If the service offers free accounts, then the threat model must assume that the attacker has numerous accounts on the system, giving them access to all the resources that normal users have, and the ability to collude multiple “users” to achieve effects that a single user can’t, e.g. artificially voting up the number of stars on a particular thing.

One can’t just “block the bad guys” because attackers rarely attack from their own computer, they attack some oblivious victim and use that other person’s computer to attack you. If one tries to investigate the attacker or “hack back” what we find is a wiped computer and a confused victim who doesn’t know what all the shouting is about.

who me.jpg

If a software product is for sale or offered for download, then the threat model must assume that the attacker has obtained copies of the product, and tested their attacks against it, and possibly reverse-engineered the product with a de-compiler. That means that if they attempt to attack this product in the wild, they have already bypassed the product’s built-in defenses.

This list of real threats goes on for a while, the attackers have lots of opportunities to attack any product or service available to the general public, or even a moderately large set of selected people. To ensure that “everything” that a skilled attacker might exploit has been checked, a similarly skilled penetration tester must enumerate, inspect, and validate all possible attack surfaces.Which is why security consulting services such as the Leviathan Security Group exist, to provide that kind of penetration testing expertise on demand. Large companies often have internal penetration test teams that continuously attack their own systems, searching for vulnerabilities so that they can close them before attackers find them. Smaller firms lack the resources to justify maintaining their own penetration test teams, and so will contract with firms like Leviathan for outsourced penetration testing. Even large firms that do have internal penetration testing teams will often outsource external penetration testing to get a different perspective on “everything” to be checked. If you don’t know whether your product or service is secure enough, let us help.

ASLR Protection for Statically Linked Executables

We present new research that details crucial security weaknesses in Linux software that has been statically linked. We also provide a solution to temporarily resolve these security issues. Finally, we conclude by demonstrating how to have both RELRO [1] and ASLR [2] security mitigations working with static linked executables in the ELF format.

The Calculus of Threat Modeling

I have been designing secure and security products for 20 years. I always thought of this as “architecture” and it took me a long time to realize that a major part of what I was doing was threat modeling. There are many established approaches to threat modeling, but because I backed into the field, I had rolled my own. This post is to explicitly describe what I have been doing.

A Minimum Viable Risk Management Program

Risk management is a fundamental requirement for all major information security frameworks, but there is little practical guidance for implementing a risk management program at small or young organizations. Existing risk management practices require varying levels of staff, expertise, tooling, and time — all expensive — as well as a mature concept of risk, when none of these necessities may be available. Consequently, there is an industry-wide need for a “minimum viable program” that allows organizations to manage risk despite lacking the prerequisites for more full-featured risk management programs. This white paper outlines such a program.

White Paper Link

Temporary workarounds shouldn’t last longer than permanent solutions

You’ve got frustrated users, availability and confidentiality issues. All from a temporary workaround that wasn’t fixed when it was relatively easier. Welcome to technical debt and the interest is accruing. Where non-kludged systems can be patched and upgraded within regular service windows without the entire IT department on call, fixing this monster will require serious planning.

On Changing Password Guidance: A Good First Step From Microsoft

On Changing Password Guidance: A Good First Step From Microsoft

Passwords, as a security solution, have become untenable. Whereas 15 years ago you might only have needed to remember two passwords, your ISP or your work password, now we have a plethora of passwords to keep track of. Power utility websites, water utility websites, bank websites, online payment websites, Facebook, Instagram, Twitter, Pinterest or any number of other sites that have our information. The guidance has often been to make passwords complex, alphanumeric, contain special characters, be as long as possible, oh and make them easy to remember but don’t reuse passwords and never write them down.