How Are System Calls Implemented in Assembly Language?

The moment I finally wrote a “Hello, World!” program in raw x86-64 assembly without linking against the C standard library, something clicked: printing text to the screen isn’t magic, it’s a carefully choreographed handoff between my program and the operating system kernel, executed through a single, deliberate instruction. That handoff is called a system call, and understanding it at the assembly level demystifies almost everything about how user programs talk to the OS.

What Is a System Call?

A system call (syscall) is a controlled entry point that lets a user-mode program request a service from the operating system kernel — reading a file, writing to the console, allocating memory, creating a process, and so on. User-mode code cannot directly touch hardware or privileged kernel data structures; it must ask the kernel to do it on its behalf.

At the assembly level, making a system call means:

1. Loading specific registers with a syscall number and its arguments.
2. Executing a special instruction that transitions the CPU from user mode to kernel mode.
3. The kernel does the work and returns a result, and the CPU transitions back to user mode.

Why Assembly-Level System Calls Matter

Every high-level function you call — printf, malloc, open() — eventually bottoms out in one of these low-level syscall instructions. Understanding this layer helps you:

• Write freestanding programs without a C runtime.
• Debug and reverse-engineer binaries by recognizing raw syscall patterns.
• Understand security boundaries (why user programs can’t just read arbitrary kernel memory).
• Optimize syscall-heavy code, since each syscall involves a costly mode transition.

The Historical Mechanism: Software Interrupts

On older x86 systems, syscalls were triggered via the INT 0x80 software interrupt instruction on Linux (32-bit), which explicitly invoked interrupt vector 128, pointing to the kernel’s syscall handler.

; Linux x86 (32-bit) - write() syscall via int 0x80
mov     eax, 4          ; syscall number for sys_write
mov     ebx, 1          ; file descriptor 1 (stdout)
mov     ecx, msg        ; pointer to buffer
mov     edx, len        ; buffer length
int     0x80            ; trigger the syscall

INT 0x80 is flexible but relatively slow, because software interrupts involve looking up an interrupt descriptor table (IDT) entry, saving a substantial amount of state, and performing a full privilege-level switch through a general-purpose mechanism designed for many kinds of interrupts, not just syscalls.

The Modern Mechanism: SYSCALL/SYSRET

On x86-64, a dedicated, faster instruction pair — SYSCALL (to enter the kernel) and SYSRET (to return) — replaced the older interrupt-based approach specifically for syscalls, because it avoids the general interrupt-handling overhead.

; Linux x86-64 - write() syscall via syscall instruction
section .data
msg     db "Hello, World!", 0xA
len     equ $ - msg

section .text
global _start
_start:
    mov     rax, 1          ; syscall number for sys_write (x86-64 table)
    mov     rdi, 1          ; file descriptor 1 (stdout)
    mov     rsi, msg        ; pointer to buffer
    mov     rdx, len        ; buffer length
    syscall                 ; enter kernel mode

    mov     rax, 60         ; syscall number for sys_exit
    mov     rdi, 0          ; exit code 0
    syscall

x86-64 Linux Syscall Calling Convention

Note the quirk: the 4th argument uses R10 instead of RCX, because the SYSCALL instruction itself uses RCX internally to save the return address.

ARM: The SVC (Supervisor Call) Instruction

On ARM (both AArch32 and AArch64), the equivalent mechanism is the SVC (SuperVisor Call) instruction, historically called SWI (Software Interrupt) on older ARM cores.

; Linux ARM64 (AArch64) - write() syscall via svc
.global _start
.section .data
msg:    .ascii "Hello, World!\n"
len = . - msg

.section .text
_start:
    mov     x0, #1          ; file descriptor 1 (stdout)
    ldr     x1, =msg        ; pointer to buffer
    mov     x2, #len        ; buffer length
    mov     x8, #64         ; syscall number for sys_write (AArch64 table)
    svc     #0              ; trigger the syscall

    mov     x0, #0          ; exit code
    mov     x8, #93         ; syscall number for sys_exit
    svc     #0

ARM64 Linux Syscall Calling Convention

Internal Working: What Actually Happens During a Syscall

sequenceDiagram
    participant App as User Program (Ring 3 / EL0)
    participant CPU as CPU Mode Switch Logic
    participant Kernel as Kernel (Ring 0 / EL1)

    App->>App: Load syscall number and arguments into registers
    App->>CPU: Execute SYSCALL / SVC instruction
    CPU->>CPU: Save user-mode context (PC, flags, stack pointer info)
    CPU->>CPU: Switch privilege level (Ring 3 -> Ring 0, or EL0 -> EL1)
    CPU->>Kernel: Jump to fixed kernel entry point
    Kernel->>Kernel: Look up syscall number in syscall table
    Kernel->>Kernel: Execute requested kernel service
    Kernel->>CPU: Prepare return value, restore user context
    CPU->>CPU: Switch privilege level back (Ring 0 -> Ring 3)
    CPU->>App: Resume execution after the syscall instruction

The key detail is the privilege level switch. The CPU has hardware-enforced rings (x86) or exception levels (ARM), and only a small, fixed set of entry points are allowed to move code from user mode into kernel mode. This prevents arbitrary user code from jumping into the kernel at an unexpected address.

Comparison: Interrupt-Based vs. Dedicated Syscall Instructions

Practical Use Cases

• Writing freestanding “no libc” programs: Useful for OS development, bootloaders, and minimal-footprint tools.
• Reverse engineering and malware analysis: Recognizing raw syscall patterns (syscall, svc #0) in disassembly reveals exactly what a binary is doing at the OS-interaction level, even if higher-level library calls are stripped or obfuscated.
• Sandboxing and security tooling: Technologies like seccomp-bpf on Linux filter which syscall numbers a process is allowed to invoke, directly operating at this assembly-visible boundary.
• Performance-sensitive I/O: Understanding syscall overhead motivates techniques like batching I/O operations or using io_uring to minimize the number of expensive mode transitions.

Debugging System Calls

$ strace ./my_program
execve("./my_program", ["./my_program"], 0x7ffd...) = 0
write(1, "Hello, World!\n", 14)         = 14
exit(0)                                 = ?

strace (Linux) intercepts and logs every syscall a program makes, which is invaluable for understanding what a compiled or hand-written assembly binary is actually asking the kernel to do, without needing to single-step through the assembly by hand.

In GDB, you can also set a breakpoint directly on the syscall instruction and inspect registers right before the mode switch:

(gdb) break *0x401020
(gdb) info registers rax rdi rsi rdx

Optimization Considerations

• Batch syscalls where possible: Each syscall incurs a real cost from the privilege-level switch and kernel-side processing; reducing syscall frequency (e.g., buffered writes instead of one syscall per byte) is a major performance lever.
• Use vDSO where available: Linux provides a “virtual dynamic shared object” for certain syscalls like gettimeofday() that avoids a full mode switch entirely by mapping a fast, read-only implementation directly into user-space memory.
• Avoid unnecessary syscalls in hot loops: Profiling tools like strace -c summarize syscall counts and time spent, quickly identifying syscall-heavy hot paths.

Common Mistakes

• Forgetting that the syscall number tables differ between architectures (x86, x86-64, ARM32, ARM64 all have different numbering) — code that hardcodes syscall numbers is not portable.
• Clobbering RCX/R11 unexpectedly on x86-64, since SYSCALL uses them internally to save return context.
• Mixing up the argument-passing convention for syscalls versus the standard C calling convention — they’re similar but not identical (notably the 4th argument register difference on x86-64).

Best Practices

• Prefer the modern dedicated syscall instruction (SYSCALL/SVC) over legacy interrupt mechanisms unless targeting very old systems.
• When writing freestanding assembly, always double-check the target architecture’s official syscall table for correct numbering.
• Use strace/ltrace during development to validate that your hand-written syscalls behave as expected.

FAQs

Do Windows and Linux use the same syscall mechanism? The underlying CPU instructions (SYSCALL on x86-64) are similar, but the syscall numbers, calling conventions, and stability guarantees differ completely — Windows discourages direct syscalls in favor of its documented Win32/Native API layers, since raw syscall numbers can change between Windows versions.

Is a syscall the same as a function call? No — a function call (CALL/BL) stays within user-mode privilege and doesn’t change CPU privilege level, while a syscall (SYSCALL/SVC) triggers a hardware-enforced privilege transition into the kernel.

Why is INT 0x80 still around on Linux x86? Purely for backward compatibility with old 32-bit binaries; new code should use the faster SYSCALL mechanism.

Summary and Key Takeaways

• System calls are the controlled boundary between user-mode programs and the kernel, implemented via dedicated CPU instructions.
• x86-64 Linux uses SYSCALL, ARM uses SVC, and legacy x86 used INT 0x80.
• Arguments are passed via specific registers following architecture-defined conventions, with the syscall number in a dedicated register (RAX on x86-64, X8 on ARM64).
• Tools like strace and GDB make syscall behavior visible and debuggable at the assembly level.
• Minimizing syscall frequency is a core performance optimization technique in systems programming.

References

• Linux man 2 syscall and architecture-specific syscall tables (arch/x86/entry/syscalls, arch/arm64/include/asm/unistd.h in the Linux kernel source)
• Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 2 (SYSCALL/SYSRET instruction reference)
• Arm® Architecture Reference Manual for A-profile architecture (SVC instruction and exception levels)
• GNU Assembler (GAS) and Linux System Call documentation (syscalls(2) man page)