Malware Detection Based on Suspicious Behavior Identification: How API Call Sequences Give Malware Away

For years, antivirus engines have leaned on one trick: signatures. Take a sample, disassemble it, pull out a unique chunk of bytes, and add it to a blacklist. It worked fine when malware authors were lazy. It stopped working the moment they weren’t.

Today a single piece of malware can be repacked, re-encrypted, and re-obfuscated hundreds of times a day without changing what it actually does. The bytes change. The behavior doesn’t. That gap — between “looks different” and “acts the same” — is exactly what a 2009 paper out of China’s National Digital Switching System Engineering & Technology Research Center set out to exploit. The paper, Malware Detection Based on Suspicious Behavior Identification by Cheng Wang, Jianmin Pang, Rongcai Zhao, Wen Fu, and Xiaoxian Liu, describes a prototype system called RADUX (Reverse Analysis for Detecting Unsafe eXecutables) that detects malware not by what it looks like on disk, but by what API calls it makes and in what order.

I want to walk through this paper properly — not as a dry summary, but as a working explanation of how behavior-based detection actually functions under the hood, why static API-sequence analysis beats raw PE-header statistics, and where the approach still runs into trouble. If you’ve spent any time picking apart binaries with tools like pdf-parser or binwalk, the underlying philosophy here — that structure and behavior reveal more than surface bytes — will feel familiar.

Why Signatures Stopped Being Enough

Signature-based detection is fast and cheap, which is why every commercial AV product still ships with a signature engine. But it has a structural weakness: it needs to have already seen the malicious code, or something close enough to it, before it can flag it. Change a handful of bytes, pack the executable, insert junk instructions, or recompile with a different compiler, and the signature no longer matches even though the payload is identical.

The paper’s authors point out that some researchers tried to get around this with data mining — pulling features out of the PE (Portable Executable) file header, counting DLLs, counting the number of imported API functions, and feeding all of that into a classifier such as naive Bayes. That approach, from earlier work by Schultz, Eskin, and Zadok, produced decent results. But it has its own flaw: PE header metadata is trivially easy to fake or strip. A program can declare very few imports in its header while still resolving and calling dozens of functions dynamically at runtime through GetProcAddress. The header, in other words, describes what the program claims to do — not what it actually does when it runs.

So the RADUX team made two corrections to that line of work:

1. Instead of trusting the PE header, they statically decompile the executable to recover its true function calls.
2. Instead of treating those calls as an unordered bag, they reconstruct the sequence of calls using control-flow analysis, and match that sequence against a database of known suspicious behavior patterns.

That second point is the real contribution. A single API call rarely means anything on its own. CreateFile is used by virtually every legitimate Windows program. It’s the sequence — CreateFile → WriteFile → CloseHandle, or RegCreateKey → RegSetValue → RegCloseKey — that starts to look like a fingerprint of intent.

The Architecture of RADUX

The prototype system is organized around five stages, and the paper lays them out as a pipeline diagram. Here’s the same flow redrawn:

flowchart TD
    A[Executable Program] --> B{Packed?}
    B -- Yes --> C[Unpack]
    C --> D
    B -- No --> D[① Decompilation]
    D --> E[② Behavior Identification]
    E --> F[③ Program Detection]
    F --> G[Result Output]

    H[Function Calls Analysis] --> D
    I[Control Flow Analysis] --> D
    J[Build API Calls Sequence] --> E
    K[Suspicious Behavior Database] <--> E
    L[Sample Space] --> M[Behavior Suspicious-Degree]
    M --> N[Bayes Algorithm]
    N --> O[Program Suspicious-Degree]
    O --> P[Set Critical Threshold]
    P --> F

Two modules matter most here, and the paper is explicit that they’re the actual research contribution rather than the whole engineering stack:

• Module ② — Behavior Identification. This is where known-malicious API call sequences get distilled and added to a “suspicious behavior database.” The system then builds the API call sequence of a new, unknown program via control-flow analysis and checks it against that database.
• Module ③ — Program Detection. This is where a Bayes algorithm, trained on a large labeled sample space, computes a “malicious degree” score for the unknown program and classifies it against a tunable threshold.

Unpacking happens first, because if the binary is packed, static decompilation won’t see the real code — it’ll see a decompression stub. This is the same reason malware analysts reach for entropy analysis in tools like binwalk before trusting any static extraction: packed or encrypted regions have to be identified and dealt with before the “real” content becomes visible.

What Counts as “Suspicious Behavior”?

This is the part of the paper I find most useful, because it’s concrete. The authors don’t just wave at “malicious API calls” — they define nine specific behavior categories, each backed by a real Windows API sequence:

None of these calls is malicious in isolation. VirtualAlloc is used by ordinary compilers and runtimes constantly. RegSetValue is how any installer writes a configuration key. What makes these behaviors suspicious rather than malicious-by-definition is context: behavior 2 (searching for files to infect) plus behavior 4 (writing to those files) plus behavior 6 (resetting the modified timestamp back to its original value, so the infection doesn’t show up as “recently modified”) is a pattern almost no legitimate installer needs. That last one — quietly restoring the file time after tampering with it — is a classic self-concealment trick, and it’s a good example of why sequence matters more than any single call.

Detecting Sequences with a Finite Automaton

To actually catch these patterns inside a stream of decompiled function calls, RADUX represents each suspicious behavior as a small finite-state automaton. The paper gives the registry-manipulation behavior as its worked example. The automaton accepts any number of irrelevant calls (labeled λ, meaning “any action except the ones we care about”), but the moment it sees the specific sequence RegCreateKey → RegSetValue → RegCloseKey, it transitions into a Bad state — the accepting state that flags the behavior as present.

stateDiagram-v2
    [*] --> S0
    S0 --> S0: λ
    S0 --> S1: RegCreateKey
    S1 --> S1: λ
    S1 --> S2: RegSetValue
    S2 --> Bad: RegCloseKey
    Bad --> [*]

This is a neat, lightweight design decision. Rather than doing expensive sequence-alignment or edit-distance comparisons against a huge database of call traces, each behavior becomes its own tiny automaton, and the decompiled call stream is fed through all of them in parallel. If any automaton reaches its Bad state, that behavior is flagged as present in the program. It’s essentially the same idea as a regular-expression scanner, just applied to API calls instead of characters.

The Bayes Layer: Turning Behaviors into a Malice Score

Detecting individual suspicious behaviors isn’t the same as deciding whether a program is malware. A single behavior might show up in benign software too — a legitimate installer touches the registry, after all. So RADUX layers a naive Bayes classifier on top of the behavior-identification stage to turn a handful of yes/no behavior flags into a probability.

The starting point is the standard form of Bayes’ theorem:

$$P(C \mid F) = \frac{P(F \mid C) \times P(C)}{P(F)}$$

Here $C$ represents the class “malicious,” and $\overline{C}$ represents “benign.” The system builds two hash tables — one from a training set of known malicious programs, one from a training set of known benign programs — and for each suspicious behavior $\omega_i$ it computes:

$$P(\omega_i \mid C) = \frac{\text{frequency of } \omega_i \text{ in malicious training set}}{\text{size of malicious hash table}}$$

$$P(\omega_i \mid \overline{C}) = \frac{\text{frequency of } \omega_i \text{ in benign training set}}{\text{size of benign hash table}}$$

The paper gives real numbers for two behaviors from its own experiments. “Search files to infect” appears in 127 out of 282 malicious-set observations but only 19 out of 289 benign-set observations — roughly a 6-to-1 ratio favoring malware. “Distribute virtual memory” is far less discriminating: 72/282 in malware versus 55/289 in benign code, which makes sense, since allocating virtual memory is something almost every nontrivial Win32 program does.

That asymmetry is the whole point of a naive Bayes design — some behaviors carry a lot of signal, others carry almost none, and the math weighs them accordingly instead of treating every “hit” the same way.

For a full program, the authors extract a set of $n$ behavior indicators, $\omega = {\omega_1, \omega_2, \dots, \omega_n}$ (in their experiments, $n = 9$, matching the nine behaviors in the table above), and combine the naive Bayes assumption of conditional independence to get the malicious degree of the whole program:

$$P(C \mid \omega) = \frac{\prod_{i=1}^{n} P(\omega_i \mid C) \times P(C)}{\prod_{i=1}^{n} P(\omega_i \mid C) \times P(C) + \prod_{i=1}^{n} P(\omega_i \mid \overline{C}) \times P(\overline{C})}$$

By construction, this score sits between 0.5 and 1 whenever the observed behavior set is more probable under the malicious model than the benign one. To spread that narrow range out into something closer to a usable 0–1 probability, the authors apply an exponential remapping:

$$f(x) = e^{x}$$

applied to the raw $P(C \mid \omega)$ output to sharpen the separation between benign and malicious scores before thresholding.

The Experiment: Does It Actually Work?

The authors didn’t just propose the architecture — they built and tested it. Their sample space totaled 914 programs: 461 benign programs pulled directly from a stock Windows XP installation directory, and 453 malicious programs pulled from the VX Heavens malware repository. They split this 80/20 into training and testing sets.

To score how well the classifier performed, they used the standard four-outcome confusion-matrix breakdown: True Positive (malware correctly flagged), True Negative (benign correctly cleared), False Positive (benign wrongly flagged), and False Negative (malware wrongly cleared). Detection precision was computed as:

$$DP = \frac{TP + TN}{TP + TN + FP + FN}$$

Then they swept the classification threshold and measured how detection precision moved:

The peak, 93.98% detection precision at a threshold of 0.8, is the headline number of the paper. The authors compare this directly against an earlier support-vector-machine-based approach from Zhang et al. (2006), which reported 89.07% precision on unknown malware, and frame their result as a meaningful improvement.

It’s worth reading that comparison with a bit of care rather than taking it purely at face value — the two studies almost certainly used different sample spaces, different malware families, and different eras of malware (2006 vs. 2009 threat landscapes aren’t identical), so the comparison is suggestive rather than a controlled head-to-head. Still, a ~5-point improvement over a reasonable prior baseline, on a 183-sample held-out test set, is a solid result for a technique built on such an interpretable foundation.

Also worth noting: the threshold sweep shows the familiar precision/recall tradeoff in miniature. Push the threshold up to 0.85 and false positives drop to 4 (good for user trust — fewer benign programs get quarantined), but false negatives rise to 8 (bad for security — more real malware slips through). There’s no threshold that maximizes both simultaneously; 0.8 is simply where this particular dataset’s tradeoff curve peaked for the combined DP metric.

Where This Approach Shines — and Where It Strains

The core insight of this paper has aged well, honestly. Every modern EDR (Endpoint Detection and Response) product does some version of behavioral sequence analysis today, just with vastly more compute and far larger behavior libraries. A few things stand out as genuinely strong design choices:

• Static analysis of true calls, not declared imports. Pulling calls from actual control flow rather than trusting the PE header closes an easy evasion path.
• Sequences over single events. A finite automaton per behavior is cheap to evaluate and directly encodes the intuition that order matters — RegCreateKey before RegSetValue before RegCloseKey tells a coherent story that no single call does alone.
• Probabilistic scoring instead of a hard yes/no. The Bayes layer lets an analyst tune sensitivity to the operating environment — a threshold of 0.7 for a consumer AV product might be too permissive for a bank’s endpoint policy.

But there are real limitations, some acknowledged by the paper itself and some that just come with hindsight:

• Packing and obfuscation still have to be solved first. The whole pipeline assumes decompilation succeeds. If a sample is packed with something the unpacker doesn’t recognize, or protected with anti-decompilation tricks, the “true call sequence” extracted upstream might be garbage or simply the unpacking stub itself.
• Static analysis can’t see runtime-resolved calls perfectly. Malware that dynamically resolves API addresses via LoadLibrary + GetProcAddress using computed or encrypted strings can hide the target function name from a purely static call-graph walk, even after successful decompilation. This is exactly the kind of case where dynamic analysis or hybrid sandboxing — the same philosophy behind tools that trace live network behavior, such as tcpdump for network-level behavioral capture — fills the gap that pure static analysis leaves open.
• The naive independence assumption. Treating each behavior $\omega_i$ as conditionally independent of the others is what makes the math tractable, but it’s not strictly true. “Search files to infect” and “file write” are behaviors that tend to co-occur precisely because they’re part of the same infection routine, not independently.
• A relatively small, dated sample space. 914 samples pulled from a single malware repository (VX Heavens) in 2009 is a reasonable proof of concept, but it’s nowhere near the scale or diversity of a modern malware corpus, and the threat landscape — ransomware, fileless attacks, living-off-the-land binaries — has shifted substantially since.

Why This Still Matters for Practical Analysis

If you do any hands-on malware triage — whether that’s dumping strings, tracing API calls in a debugger, or hashing files to check for tampering the way hashdeep does — the mental model in this paper is worth internalizing even outside the specific 2009 prototype. When you’re looking at an unfamiliar binary, don’t just ask “what strings does this contain” or “what does the header claim.” Ask what it actually calls, in what order. A program that resolves FindFirstFile, walks a directory, opens each match for writing, and then resets the file’s timestamp is telling you something about its intent that no signature match ever could.

That’s really the lasting contribution of this paper: behavior, expressed as an ordered sequence of system calls, is a far more durable signal than any static byte pattern. Sixteen years on, with adversaries wrapping the same tricks in ransomware-as-a-service kits and fileless PowerShell loaders, that core idea — watch what it does, not what it looks like — is more relevant than it was in 2009, not less.

Reference

Wang, C., Pang, J., Zhao, R., Fu, W., & Liu, X. (2009). Malware Detection Based on Suspicious Behavior Identification. In 2009 First International Workshop on Education Technology and Computer Science, pp. 198–202. IEEE. DOI: 10.1109/ETCS.2009.306.