How Does Assembly Language Interact with the Operating System?

Writing pure Assembly code that just crunches numbers in registers is one thing. Getting that code to actually print something to the screen, read a file, allocate memory, or exit cleanly is a completely different challenge — and it’s where Assembly meets the operating system. Understanding this boundary was, for me, the point where Assembly stopped feeling like an academic exercise and started feeling like real systems programming.

In this post, I’ll walk through exactly how Assembly programs talk to the operating system: system calls, calling conventions, privilege levels, interrupts, and the practical mechanics on both Linux (x86-64) and ARM-based systems.

Why Assembly Needs the Operating System at All

An Assembly program, on its own, only has direct control over the CPU’s registers and whatever memory it’s been given. It cannot, by itself, write to a file, allocate more memory from the system, create a new process, or even reliably print to a terminal — all of these require interacting with hardware resources that are managed and protected by the operating system’s kernel.

The OS exists specifically to mediate access to shared hardware resources (disk, network, memory, other processes) so that programs can’t interfere with each other or the system as a whole. Because of this, Assembly programs must go through a well-defined, controlled interface to request OS services — this interface is the system call.

Privilege Levels: User Mode vs. Kernel Mode

Modern CPUs implement hardware-enforced privilege levels to protect the operating system kernel from user programs.

• On x86-64, these are called rings — Ring 0 (kernel mode, full privilege) through Ring 3 (user mode, restricted privilege). Most operating systems only use Ring 0 and Ring 3.
• On ARM, these are called Exception Levels — EL0 (user applications), EL1 (OS kernel), EL2 (hypervisor), EL3 (secure monitor).

Assembly code you write and run as an ordinary program executes in the least privileged level (Ring 3 / EL0). It cannot directly execute privileged instructions (like manipulating page tables, or directly accessing certain hardware I/O ports) — attempting to do so triggers a fault, and the kernel decides how to handle it (usually terminating the offending process).

flowchart TD
    A[User Mode: Ring 3 / EL0] -->|syscall / SVC instruction| B[Trap into Kernel]
    B --> C[Kernel Mode: Ring 0 / EL1]
    C --> D[Kernel executes requested service]
    D --> E[Return to User Mode]
    E --> A

System Calls: The Core Interface

A system call is a controlled, well-defined entry point that lets user-mode Assembly code request a service from the kernel — reading/writing files, allocating memory, creating processes, networking, and so on.

Making a System Call on Linux x86-64

On Linux x86-64, system calls are invoked using the syscall instruction. Arguments are passed in specific registers according to a defined convention:

Example — writing “Hello, world!” to standard output and exiting cleanly:

section .data
    msg db "Hello, world!", 0xA
    len equ $ - msg

section .text
global _start
_start:
    mov rax, 1          ; syscall number for write
    mov rdi, 1          ; file descriptor 1 = stdout
    mov rsi, msg        ; pointer to the message
    mov rdx, len        ; length of the message
    syscall             ; trap into the kernel

    mov rax, 60         ; syscall number for exit
    mov rdi, 0          ; exit code 0
    syscall

When syscall executes, the CPU switches from user mode to kernel mode, the kernel looks at RAX to determine which system call is being requested, performs the requested action using the arguments in the other registers, places a return value back into RAX, and then switches back to user mode, resuming execution right after the syscall instruction.

Making a System Call on ARM64 (Linux)

The mechanism is conceptually identical but uses the SVC (Supervisor Call) instruction and a slightly different register convention:

Example on ARM64 Linux:

.data
msg: .ascii "Hello, world!\n"
len = . - msg

.text
.global _start
_start:
    mov x8, #64          ; syscall number for write
    mov x0, #1            ; file descriptor 1 = stdout
    ldr x1, =msg          ; pointer to message
    mov x2, #14            ; length of message
    svc #0                 ; trap into the kernel

    mov x8, #93           ; syscall number for exit
    mov x0, #0
    svc #0

Older Mechanisms: Interrupts

Before the dedicated syscall/sysenter instructions existed, x86 systems used software interrupts to invoke system calls — most famously int 0x80 on 32-bit Linux:

mov eax, 4          ; syscall number for write (32-bit ABI)
mov ebx, 1
mov ecx, msg
mov edx, len
int 0x80             ; trigger software interrupt to enter kernel

int 0x80 works by triggering a software interrupt, which causes the CPU to look up a corresponding entry in the Interrupt Descriptor Table (IDT), transferring control to the kernel’s interrupt handler. It’s slower than the dedicated syscall/sysenter instructions (which were introduced specifically to speed up this transition), which is why 64-bit Linux moved to syscall as the standard mechanism.

Calling Conventions and the OS/Application Boundary

Beyond raw system calls, Assembly also interacts with the operating system indirectly through calling conventions — standardized rules about how functions pass arguments, return values, and preserve registers. This matters because:

• Dynamically linked libraries (like the C standard library, libc) are provided by, or closely tied to, the operating system’s runtime environment.
• Calling a library function like printf or malloc from Assembly requires respecting the calling convention (e.g., the System V AMD64 ABI on Linux, or the AAPCS64 on ARM64) so that arguments end up where the library function expects them.

Signals and Asynchronous OS Notifications

Beyond synchronous system calls that a program explicitly requests, the operating system can also asynchronously interrupt a running program via signals (on Unix-like systems) or equivalent mechanisms on other platforms. A signal handler is itself just a regular function — often written in Assembly or C — that the kernel invokes on behalf of the process when a specific signal (like SIGSEGV for a segmentation fault, or SIGINT for a keyboard interrupt) occurs.

From the Assembly programmer’s perspective, registering a signal handler still goes through a system call (rt_sigaction on Linux x86-64), but the actual invocation of the handler happens asynchronously, at a point in time the program itself doesn’t directly control — the kernel essentially “injects” a call to the handler function into the program’s execution flow, saving and later restoring the interrupted context.

; simplified concept: registering a SIGINT handler
mov rax, 13              ; syscall number for rt_sigaction
mov rdi, 2                ; SIGINT
mov rsi, sigaction_struct   ; pointer to a struct describing the handler
mov rdx, 0
mov r10, 8
syscall

Windows System Calls: A Brief Contrast

Everything above focuses on Linux, since it’s the most common target for hand-written Assembly system programming, but it’s worth noting that Windows handles this boundary differently. Rather than exposing raw syscall numbers as a stable, documented interface the way Linux does, Windows funnels almost all OS interaction through the Windows API (Win32 API), implemented in user-mode DLLs like kernel32.dll and ntdll.dll. The actual low-level system call instruction (syscall on x86-64 Windows) exists and is used internally by these DLLs, but Microsoft does not guarantee syscall numbers or the raw calling convention as a stable public interface the way Linux does — which is why Assembly programmers targeting Windows almost always call into these DLL-provided functions rather than issuing raw syscalls directly.

; Calling ExitProcess from Windows x64 Assembly (via kernel32.dll)
sub rsp, 28h
mov ecx, 0            ; exit code
call ExitProcess       ; resolved via import table, ultimately backed by a syscall internally

Process Memory Layout and the OS

The operating system is also responsible for setting up the memory layout your Assembly program runs within — the stack, heap, and various segments (.text, .data, .bss) are all placed into virtual memory by the OS loader before your program’s entry point even executes. Assembly programs interact with the OS here too, indirectly, whenever they:

• Request more heap memory (via brk/mmap system calls on Linux, or VirtualAlloc on Windows).
• Rely on Address Space Layout Randomization (ASLR) placing code and data at semi-random addresses each run.
• Use stack memory managed and protected by OS-enforced guard pages.

Context Switching: The OS Managing Assembly-Level State

Whenever the operating system needs to pause one process (or thread) and resume another — a context switch — it must save the complete Assembly-visible CPU state (general-purpose registers, flags, program counter, stack pointer, and often floating-point/SIMD registers) for the outgoing process, and restore that same set of state for the incoming one. This entire mechanism, while orchestrated by kernel code, ultimately operates directly on the exact same registers and concepts an Assembly programmer works with daily.

This is worth understanding because it explains an important guarantee: from the perspective of your Assembly program, a context switch is completely invisible. Your registers, stack, and program counter are exactly as you left them whenever your process resumes execution — even though, physically, the CPU may have executed thousands of other instructions belonging to entirely different processes in between. The OS achieves this illusion of continuous execution purely through careful, systematic save/restore of Assembly-level state during every context switch.

System Call Overhead and Why It’s Minimized in Practice

Because crossing from user mode to kernel mode (and back) involves real hardware-level overhead — saving/restoring registers, potentially flushing certain CPU speculation structures for security reasons (relevant to mitigations for vulnerabilities like Meltdown and Spectre), and updating privilege state — well-optimized Assembly and systems code tries to minimize the number of system calls made, even when each individual call is cheap. This is exactly why, for example, file I/O libraries buffer reads and writes internally rather than issuing a write syscall for every single byte, and why high-performance networking code often uses batch-oriented syscalls (like sendmmsg on Linux) that handle multiple operations in a single kernel transition instead of many small ones.

Debugging OS Interaction

Tools like strace on Linux are invaluable for understanding exactly which system calls your Assembly program makes, in what order, and with what arguments:

strace ./my_assembly_program

This shows you the raw system call sequence — write, exit, mmap, open, etc. — which is often the fastest way to diagnose why a low-level program is behaving unexpectedly, especially when something as simple as a wrong file descriptor or a malformed syscall argument is at fault.

Virtual Memory: Another Layer of OS-Managed Abstraction

Every memory address an Assembly program works with — whether it’s a base register holding a pointer, a stack address, or a global variable’s location — is actually a virtual address, translated transparently by the CPU’s Memory Management Unit (MMU) into a physical address, using page tables that the operating system’s kernel sets up and maintains. This translation is invisible to ordinary Assembly instructions; a mov/LDR referencing a virtual address behaves exactly as if it were referencing physical memory directly, but under the hood, the OS is responsible for mapping that virtual address to actual RAM (or triggering a page fault if the memory isn’t currently resident, prompting the kernel to load it from disk via a swap/demand-paging mechanism). This is yet another example of how deeply Assembly-level execution depends on, and is shaped by, services the operating system quietly provides beneath the surface.

Common Mistakes

• Using the wrong syscall number for the target architecture/OS — syscall numbers differ between x86-64 Linux, ARM64 Linux, and other operating systems entirely (they are not universal).
• Forgetting that some registers are call-clobbered by the kernel during a syscall (e.g., syscall on x86-64 clobbers RCX and R11).
• Mixing up 32-bit (int 0x80) and 64-bit (syscall) conventions, which use entirely different argument-passing registers and syscall numbers.
• Not aligning the stack properly before calling into C library functions, which can cause crashes in SIMD-optimized library code that assumes 16-byte alignment.

Best Practices

• Always consult the syscall table for your specific target OS and architecture rather than assuming numbers/conventions carry over from a different platform.
• Use strace/ltrace (Linux) early when debugging unexpected behavior in a program that interacts with the OS.
• When mixing Assembly with C, respect the platform’s official calling convention exactly (register usage, stack alignment, callee-saved registers).
• Prefer using syscall/SVC (the modern mechanisms) over legacy interrupt-based syscalls unless you have a specific compatibility reason not to.

FAQs

Can Assembly programs run without any operating system at all? Yes — this is common in embedded systems and OS kernel development itself (“bare-metal” programming), where the Assembly code directly manages hardware without any underlying OS. In that context, there are no system calls at all; the Assembly code is effectively the lowest software layer, interacting directly with hardware registers.

Is a system call the same as a function call? No. A function call transfers control within the same privilege level (user mode to user mode). A system call transfers control across privilege levels (user mode to kernel mode), involving hardware-level protection mechanisms, which is significantly more expensive in CPU cycles.

Why do syscall numbers differ between architectures? Because each OS/architecture combination defines its own syscall table independently — there’s no universal standard number for “write” or “read” across all platforms, which is why cross-platform Assembly code needs architecture-specific syscall handling.

Why doesn’t Windows expose raw syscall numbers the way Linux does? Microsoft treats the raw syscall interface as an internal implementation detail that can change between Windows versions, and instead guarantees stability at the Win32 API / DLL level. This gives Microsoft flexibility to change internals without breaking existing software, at the cost of Assembly programmers needing to go through DLL-exported functions rather than issuing syscalls directly.

What happens if I execute a privileged instruction in user mode by mistake? The CPU raises a general protection fault (on x86) or an equivalent exception (on ARM), which the operating system’s kernel intercepts. Typically, the kernel terminates the offending process (often with a segmentation fault or illegal instruction signal) rather than allowing the privileged operation to proceed.

Summary and Key Takeaways

• Assembly programs interact with the OS primarily through system calls, which cross the boundary from user mode to kernel mode.
• On Linux x86-64, this is done via the syscall instruction with arguments in RAX, RDI, RSI, RDX, R10, R8, R9.
• On ARM64 Linux, the equivalent is SVC #0 with arguments in X8, X0–X5.
• Privilege levels (rings on x86, exception levels on ARM) enforce that user programs can’t directly access protected hardware or kernel memory.
• Calling conventions and ABI rules govern how Assembly interacts with OS-provided libraries beyond raw syscalls.
• Tools like strace make the entire OS interaction of a program visible and are essential for debugging.

References

• Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 3: System Programming Guide
• AMD64 Architecture Programmer’s Manual, Volume 2: System Programming
• Arm® Architecture Reference Manual for A-profile Architecture
• Linux Kernel System Call Table Documentation (kernel.org)
• GNU Binutils / GAS Documentation (as.info)