Detecting Kernel-Level Rootkits Through System Call Table Redirection: A Look Back at a Foundational Methodology

Every rootkit tells the same lie. It sits between an application and the truth, and it decides what that application gets to see. Ask the kernel for a directory listing, and the rootkit quietly removes its own files from the answer. Ask for a process list, and it drops its own PID before handing the results back. The application never knows it was lied to, because from its point of view, the kernel answered normally.

That single idea — intercepting the boundary between user space and kernel space — is the whole game for kernel-level rootkits, and it’s the subject of a paper I keep coming back to: A Methodology to Detect and Characterize Kernel Level Rootkit Exploits Involving Redirection of the System Call Table, written by John Levine, Julian Grizzard, and Henry Owen at Georgia Tech’s School of Electrical and Computer Engineering. It’s an older paper, built and tested against Red Hat 8.0 running kernel 2.4.18-14, and it looks at rootkits like KNARK, SuckIT, and zk. But the underlying reasoning — trust the hardware, not the software, when you’re hunting for a compromise — still holds up, and it’s worth walking through carefully, both on its own terms and against what Linux rootkits actually look like today.

What a Rootkit Actually Is (and Isn’t)

The paper opens with a definition worth restating because people get it wrong constantly: a rootkit does not get an attacker into a system. It keeps them in. A hacker needs root-level access before a rootkit can be installed. Once they have it, the rootkit’s job is to let them come back later without tripping any alarms, and to hide the evidence of their presence — files, processes, network connections — from the legitimate administrator.

The authors borrow a taxonomy from earlier work on Trojans by Thimbleby, Anderson, and Cairns, who split malicious masquerading software into four categories: direct masquerades (pretending to be a normal program), simple masquerades (pretending to be a plausible program that doesn’t actually exist), slip masquerades (using a name close enough to an existing one to fool a quick glance), and environmental masquerades (already-running processes that a user wouldn’t easily recognize as foreign). Kernel rootkits fall squarely into the first category — a trojanized version of a real system function, standing in for the original.

The Kernel Is the Real Target

Traditional binary rootkits swap out user-space utilities like ls, ps, or netstat with trojanized versions that lie about their output. That’s detectable, in principle, by comparing file hashes or checksums against a known-good baseline — which is exactly what file integrity checkers like Tripwire and AIDE were built to do.

Kernel-level rootkits sidestep that defense entirely by attacking a layer below any user-space binary: the system call table. Every request an application makes to the kernel — read a file, fork a process, allocate memory — goes through a system call, or sys_call. The kernel keeps a table of addresses in memory, one per system call, pointing to the actual function that handles the request. If an attacker can quietly redirect entries in that table, they control what the kernel tells every application on the system, without touching a single file that a checksum tool would notice.

flowchart LR
    A[User Application] -->|"sys_call(#N)"| B[System Call Table]
    B -->|normal system| C[sys_fork / sys_read / sys_execve ...]
    B -->|"rootkit redirects entry #N"| D[Attacker's Replacement Function]
    D -->|filters output, hides files/PIDs| E[Falsified Result Returned]
    C -->|honest result| F[Real Result Returned]

The paper describes two distinct techniques attackers used to pull this off on Linux 2.4:

1. Modifying individual entries in the system call table. A Loadable Kernel Module (LKM) — a legitimate Linux feature for extending kernel functionality at runtime without recompiling — is repurposed to overwrite selected addresses in the existing system call table with pointers to the attacker’s own functions. The table itself stays where it always was; only specific entries inside it point somewhere new. KNARK, released by CREED in 2001, is the paper’s example of this style.

2. Redirecting the entire system call table to a new location. This is subtler. Rather than editing the original table in place, the rootkit builds a brand-new table somewhere else in kernel memory — containing a mix of the attacker’s malicious replacements and the original, untouched addresses for anything it doesn’t care about — and then overwrites the pointer that the kernel uses to find the table in the first place. One way to do this on the target systems studied is by writing directly to /dev/kmem, the device file that exposes raw kernel memory. Crucially, this method leaves the original system call table completely intact. Anyone who checks the original table for tampering finds nothing wrong, because the kernel isn’t even looking at that table anymore.

Why This Beats Traditional Detection

That second technique is why kernel rootkits were such a headache for the tools available at the time. The paper walks through a live comparison, and it’s a genuinely useful case study in how a defense-in-depth mindset can still fail against the right attacker.

They tested a Red Hat 8.0 system, first clean, then infected with the SuckIT rootkit — a program developed by “sd” and “devik,” documented in Phrack 58, notable for patching the kernel on the fly without needing LKM support at all. Three detection tools were run against the infected system:

• chkrootkit, a signature-based shell script that checks system binaries and looks for known rootkit fingerprints, failed to detect SuckIT while it was actively running. It only caught traces of the rootkit after it had been uninstalled, because SuckIT’s file-hiding trick (renaming the real /sbin/init to a hidden filename and quietly substituting a lookalike) evaporated the moment the rootkit removed itself.
• AIDE (Advanced Intrusion Detection Environment), a file-integrity checker, noticed that attributes on /sbin/telinit had changed — but had no way to explain why, and gave no indication that the kernel itself, or the system call table, had been compromised. That’s the structural limit of any checksum-based tool: it tells you that something changed, never what kind of something.
• kern_check, a small Samhain Labs utility built specifically to compare the live system call table against the /boot/System.map file generated at kernel compile time, also missed SuckIT entirely — because SuckIT redirects the table pointer rather than editing the original table, and Samhain Labs themselves acknowledged the tool wasn’t built to catch that.

None of the three tools available at the time could reliably say “this system has a kernel rootkit, and here specifically is how it works.” That gap is exactly what the paper set out to close.

Querying the Hardware Instead of Trusting the OS

The methodology’s central trick is elegant: instead of asking the (possibly compromised) kernel where its own system call table lives, ask the CPU directly.

Every x86 system call passes through a specific interrupt — int $0x80 — and the processor’s Interrupt Descriptor Table (IDT) holds the true address of the routine that handles it. That IDT entry points to the system_call() function, and buried inside that function’s disassembled machine code is an indirect call instruction: call *sys_call_table(,%eax,4). That instruction contains, as literal encoded bytes, the actual address the kernel is using right now to look up system calls — regardless of what any file on disk claims that address should be.

The authors use the sidt x86 assembly instruction (the same trick SuckIT itself uses internally) to retrieve the IDT, walk to the system_call() entry point, and extract the operand of that call instruction directly from live kernel memory. That gives them the address the CPU is genuinely dereferencing on every system call — a ground truth that no amount of table redirection can hide, because the rootkit still has to funnel every syscall through this exact instruction to keep the system functioning normally.

sequenceDiagram
    participant CPU as CPU / IDT
    participant SC as system_call() function
    participant SCT as Live System Call Table
    participant Map as /boot/System.map

    Note over CPU: sidt retrieves IDT base
    CPU->>SC: Locate system_call() entry point
    SC->>SCT: Extract address from<br/>"call *sys_call_table(,%eax,4)"
    Note over SCT: This is the address<br/>the kernel actually uses
    SCT->>Map: Compare against compiled-in address
    alt Addresses match, but individual entries differ
        Note over Map: Rootkit modifies table in place
    else Addresses do not match at all
        Note over Map: Rootkit redirects to new table
    end

Once that live address is in hand, it gets compared against the address recorded in /boot/System.map, the symbol map generated when the kernel was originally compiled. The comparison produces one of two diagnostic outcomes:

• If the retrieved address matches /boot/System.map, but individual entries within the table point somewhere unexpected — a rootkit is modifying the original system call table in place (the KNARK-style attack).
• If the retrieved address does not match /boot/System.map at all — the entire table has been redirected to a new location in kernel memory (the SuckIT-style attack).

Either way, the analyst now knows not just that the system is compromised, but which class of kernel rootkit is responsible. The authors modified Samhain’s original kern_check utility to implement this comparison and ran it against a SuckIT-infected system, correctly flagging 25 redirected system calls and reporting the new table’s address in kernel memory — a result the original, unmodified kern_check was structurally incapable of producing.

A Mathematical Framework for “Is This the Same Rootkit?”

Detecting a compromise is only half the paper’s contribution. The other half is a framework for answering a more subtle question: once you’ve found a rootkit, is it one you’ve already seen, a modified version of one you’ve seen, or something genuinely new?

The authors build this on set theory. Let $p_1$ be the set of original, legitimate programs, and $p_2$ be the malicious replacement installed by the rootkit. Because a functioning rootkit must preserve all of the original program’s behavior while adding its own hidden capabilities, $p_1$ is necessarily a proper subset of $p_2$:

$$p_1 \subset p_2, \quad \text{since } p_1 \subseteq p_2 \text{ and } p_1 \neq p_2$$

The interesting object is the delta, written $\nabla$ — the set of elements present in $p_2$ but absent from $p_1$:

$$p_2 \setminus p_1 = p’ = \nabla$$

$\nabla$ is effectively the rootkit’s fingerprint: the specific added functionality — new hooked syscalls, backdoor triggers, hidden PID logic — that distinguishes the trojanized program from the legitimate one it’s impersonating. Once you have a $\nabla$ for a known rootkit, you can classify any newly discovered suspect $p_3$ against it:

• If $p_3 – (\nabla \cap p_3) = p_1$, then $p_3$ contains exactly the same added elements as the known rootkit — it’s the same exploit.
• If some but not all elements of $\nabla$ appear in $p_3$ (written $\nabla \in p_3$), it’s a modification of the known rootkit.
• If none of $\nabla$’s elements appear in $p_3$ (written $\nabla \notin p_3$), it’s classified as an entirely new rootkit.

The paper puts this to work on a genuinely interesting comparison: SuckIT versus a related rootkit called zk, whose own documentation admits it borrowed SuckIT’s kernel-patching approach. Running the modified kern_check against a zk-infected system showed the exact same 25 system calls being subverted as SuckIT — strong evidence of shared lineage. But zk’s uninstall command failed where SuckIT’s succeeded, which was the first real behavioral $\nabla$ between the two. Digging into zk’s source with grep for the string "password" turned up a hardcoded uninstall password — "kill me" — buried in client.c, something SuckIT’s own equivalent file didn’t contain. That password became a usable signature for identifying zk specifically, separate from SuckIT, even though the two share the bulk of their kernel-patching mechanics.

Why the System Call Table Method Eventually Stopped Being Enough

The paper is honest about its own limits, and reading it today, those limits map almost exactly onto how the Linux kernel evolved afterward. The authors note that kern_check already failed against the Linux 2.6 kernel, because 2.6 stopped exporting the system call table’s address via query_module specifically to prevent race conditions during dynamic replacement of syscall addresses by loadable modules. That single kernel change quietly broke an entire generation of detection tools built around comparing exported symbols.

That trajectory has continued for two decades. Direct writes to /dev/kmem — the technique SuckIT relied on — have been closed off entirely on modern Linux; /dev/kmem was removed from the mainline kernel years ago, and /dev/mem access to general kernel memory is restricted by CONFIG_STRICT_DEVMEM. Kernel module signing, Secure Boot chains, and Linux Security Modules make an unsigned, unverified LKM far louder and harder to load quietly than it was in 2003. According to recent security research from Elastic Security Labs, classic syscall-table-hooking LKM rootkits like Diamorphine (2016) and Reptile (2020) are still found in the wild — but attackers increasingly treat raw syscall table overwrites as a last resort rather than a first move, precisely because that technique is now well-instrumented and reliably flagged by tainted-kernel checks, /proc/modules auditing, and specialized LKM scanners.

Where the field has actually moved is toward attacking legitimate kernel instrumentation frameworks instead of the syscall table directly. eBPF — originally built for safe, verified packet filtering and performance tracing — lets a program attach to kprobes, tracepoints, and LSM hooks without loading a kernel module at all, which sidesteps module-signing defenses entirely. Rootkits such as PUMAKIT (documented in late 2024) and a 2025 in-the-wild implant called LinkPro use eBPF programs to intercept calls like getdents (to hide files and processes) and even to hide their own eBPF programs from introspection via sys_bpf. Some go further still, abusing io_uring — the asynchronous I/O interface introduced in Linux 5.1 — to batch operations in ways that generate almost no observable syscall events at all, which is what an experimental 2025 proof-of-concept called RingReaper demonstrated against tools that assume syscall-level tracing catches everything. And on the defensive side, a late-2025 ScienceDirect paper on a system called HKRD proposes using eBPF itself, defensively, to validate syscall table addresses continuously and catch direct kernel object manipulation — essentially the same “check the live address against the true one” instinct from the Georgia Tech paper, reimplemented with a modern, verifier-checked kernel framework instead of raw /dev/kmem access.

The throughline across all of this — 2004 to 2026 — hasn’t actually changed: a kernel rootkit has to funnel legitimate operations through some interception point to hide anything, and that interception point, however cleverly hidden, has to be reachable at the hardware or kernel-instrumentation level by anyone willing to go around the operating system’s own reporting mechanisms rather than trust them. What’s changed is which interception point attackers use, and how much cover modern kernel instrumentation gives them while doing it.

Practical Takeaways for Anyone Doing This Kind of Work Today

A few lessons from this paper are worth carrying forward regardless of which kernel version or hooking technique is in play:

• File integrity checkers tell you something changed, not what changed it. AIDE flagging /sbin/telinit in the paper’s SuckIT test is a perfect illustration — true, useful, and completely insufficient on its own. Pairing integrity monitoring with something that inspects live kernel state (or, today, host-based eBPF telemetry from tools designed for exactly this) closes that gap.
• Signature-based scanners are only as good as their signature database, and rootkits actively exploit that fact. chkrootkit missing SuckIT while it was running, then catching it only after uninstall, is a direct consequence of a rootkit author designing specifically around a known detector’s blind spots. The same dynamic plays out today between eBPF-based rootkits and eBPF-naive monitoring stacks.
• Prefer ground truth from hardware or a verified low-level source over anything the OS chooses to report. Querying the IDT directly, rather than trusting query_module or a userland utility, is the same instinct behind modern memory forensics approaches that read kernel structures out of a raw memory image rather than trusting a live, possibly-compromised OS to describe itself accurately.
• Behavioral deltas are durable signatures even when byte-level signatures aren’t. The zk-versus-SuckIT comparison in this paper — spotting a hardcoded password difference through source review rather than a static hash match — is conceptually close to the API-call-sequence approach used in behavior-based malware detection more broadly. If you’re comparing binaries for common ancestry, tools like hashdeep are a fast first pass for flagging exact matches, but genuine lineage analysis — the $\nabla$ concept from this paper — usually needs a deeper look with something like Autopsy or manual source-level review, exactly as the authors did when they grepped client.c for the string that gave zk away.
• Registry and configuration hive artifacts matter on the Windows side of this same problem. Kernel-mode rootkit concealment isn’t a Linux-only concern, and when an incident spans a mixed environment, offline artifact review — the same philosophy behind tools like chntpw for pulling data out of a Windows registry hive without trusting a live, possibly-compromised OS to hand it over honestly — follows the identical logic as querying the IDT instead of query_module: don’t ask a system that might be lying to describe itself.

Closing Thought

What makes this paper worth reading in 2026 isn’t the specific tools it built — kern_check for the 2.4 kernel is a historical artifact at this point, and neither SuckIT nor zk pose any real threat to a modern, signed, lockdown-mode kernel. What’s worth keeping is the underlying discipline: when you suspect a rootkit, don’t ask the operating system to grade its own homework. Find the one piece of ground truth the attacker cannot fake without breaking the very functionality they’re trying to preserve, and check that instead. Twenty years on, whether the interception point is a syscall table pointer, an eBPF program hooked into getdents, or a batch of disguised io_uring operations, that’s still the only reliable way to catch something that was specifically engineered to lie to you.

Reference

Levine, J., Grizzard, J., & Owen, H. (2004). A Methodology to Detect and Characterize Kernel Level Rootkit Exploits Involving Redirection of the System Call Table. School of Electrical and Computer Engineering, Georgia Institute of Technology.