Subroutines in Assembly Language: A Complete Guide From Beginner to Advanced

When I first started learning Assembly language, the idea of a “subroutine” felt oddly abstract compared to functions in C or Python. There’s no neat def keyword, no curly braces, no built-in return statement. Instead, a subroutine in Assembly is something you build yourself out of jumps, stack operations, and a handful of disciplined conventions. Once it clicked for me, though, I realized subroutines are really the backbone of every non-trivial Assembly program — and understanding them deeply is what separates someone who can read Assembly from someone who can actually write it.

In this post, I’ll walk through what a subroutine is, how the CPU and stack cooperate to make it work, how it’s implemented on x86, x86-64, and ARM, and how to avoid the mistakes that trip up almost everyone the first time they write one.

What Exactly Is a Subroutine?

A subroutine (also called a procedure, function, or in some older texts, a “closed subroutine”) is a named, reusable block of code that performs a specific task and then returns control to wherever it was called from. The key idea is reusability without duplication — instead of copying the same ten instructions in five places in your program, you write them once and jump to them whenever needed.

In high-level languages, this concept is hidden behind syntax. In Assembly, you’re responsible for:

1. Saving the return address (where to go back to after the subroutine finishes)
2. Transferring control to the subroutine’s first instruction
3. Preserving register values the caller still needs
4. Passing arguments in some agreed-upon way
5. Returning a result (if any)
6. Restoring control back to the caller at the exact right instruction

That’s a lot of bookkeeping for something that looks like one line of code in Python. But this is also why Assembly gives you such precise control over performance — you decide exactly how much overhead a function call costs.

The Core Mechanism: CALL and RET

On x86 and x86-64, subroutines are invoked using the CALL instruction and exited using RET. These two instructions are mirror images of each other.

When CALL executes:

• It pushes the address of the instruction immediately following the CALL onto the stack (this is the return address).
• It then jumps to the target label/address.

When RET executes:

• It pops the return address off the stack.
• It jumps to that address, resuming execution right where the caller left off.

; x86-64 NASM syntax
section .text
global _start

_start:
    mov rdi, 5
    call square        ; call the subroutine
    ; rax now holds the squared result
    mov rax, 60         ; sys_exit
    mov rdi, 0
    syscall

square:
    mov rax, rdi
    imul rax, rax
    ret                 ; return to caller

I like to think of CALL and RET as a matched pair — almost like opening and closing a bracket. Every CALL pushes one return address; every RET pops one. If those two ever get out of balance (say, you manually push something onto the stack inside the subroutine and forget to pop it before RET), the CPU will “return” to garbage, and your program crashes in a way that’s often very hard to debug.

The Role of the Stack

The stack is what makes subroutines possible. It’s a region of memory that grows (usually downward in x86/x86-64) as data is pushed and shrinks as data is popped, tracked by the stack pointer (ESP/RSP on x86, SP on ARM).

Here’s a simplified diagram of what happens in memory during a subroutine call with two local variables:

High Address
+-------------------+
| Caller's frame     |
+-------------------+
| Return Address      |  <- pushed by CALL
+-------------------+
| Saved RBP (old)     |  <- pushed by function prologue
+-------------------+
| Local variable 1    |
+-------------------+
| Local variable 2    |  <- RSP points here
+-------------------+
Low Address

This structure is called a stack frame (or activation record). Every time a subroutine is called, a new frame is pushed on top of the stack; when it returns, that frame is discarded.

The Prologue and Epilogue

Most non-trivial subroutines follow a standard pattern:

my_function:
    push rbp            ; save old base pointer (prologue)
    mov rbp, rsp         ; set new base pointer
    sub rsp, 16           ; reserve space for locals

    ; ... function body ...

    mov rsp, rbp         ; epilogue: restore stack pointer
    pop rbp               ; restore old base pointer
    ret                    ; return to caller

RBP (base pointer) acts as a fixed reference point within the function, so local variables and parameters can be accessed at consistent offsets ([rbp-4], [rbp+16], etc.) even as RSP moves around during the function’s execution.

Subroutines on ARM

ARM doesn’t use CALL/RET — instead it uses BL (Branch with Link) and BX LR / RET.

; ARM (AArch32) example
square:
    MUL R0, R0, R0      ; R0 = R0 * R0
    BX LR                 ; return using Link Register

main:
    MOV R0, #5
    BL square             ; branch and store return address in LR
    ; R0 now holds 25

BL copies the return address into the Link Register (LR) instead of pushing it onto the stack automatically. This is a deliberate design choice — it makes simple leaf functions (functions that don’t call other functions) extremely fast because there’s no stack traffic at all. If a function needs to call another function, though, it must manually push LR onto the stack first, since the next BL will overwrite it.

; AArch64 (ARM64) example
square:
    MUL X0, X0, X0
    RET                    ; returns to address in LR (X30)

nested_function:
    STP X29, X30, [SP, #-16]!   ; save frame pointer and LR
    BL square
    LDP X29, X30, [SP], #16     ; restore
    RET

Passing Arguments and Returning Values

There’s no universal rule for how arguments are passed — it depends on the calling convention in use.

Understanding the calling convention matters enormously when you’re linking Assembly with C code, writing OS kernels, or reverse-engineering compiled binaries — get it wrong and your program will silently corrupt data or crash.

Internal Working: Step-by-Step Flow

Here’s a mermaid diagram showing the full lifecycle of a subroutine call on x86-64:

sequenceDiagram
    participant Caller
    participant Stack
    participant Subroutine

    Caller->>Stack: CALL pushes return address
    Caller->>Subroutine: Jump to subroutine label
    Subroutine->>Stack: PUSH RBP (save old base pointer)
    Subroutine->>Subroutine: MOV RBP, RSP (new frame)
    Subroutine->>Subroutine: Execute function body
    Subroutine->>Stack: POP RBP (restore base pointer)
    Subroutine->>Stack: RET pops return address
    Stack->>Caller: Resume execution after CALL

Nested and Recursive Subroutines

Because each call pushes a fresh stack frame, subroutines can call themselves — this is how recursion works in Assembly.

; x86-64: recursive factorial
factorial:
    push rbp
    mov rbp, rsp
    cmp rdi, 1
    jle base_case
    push rdi              ; save current n
    dec rdi
    call factorial          ; recursive call
    pop rdi                 ; restore n
    imul rax, rdi
    jmp done
base_case:
    mov rax, 1
done:
    pop rbp
    ret

Every recursive call consumes stack space, which is why deep, unbounded recursion in Assembly (just like in C) can trigger a stack overflow — the stack pointer runs past the memory region allocated for the stack and crashes into other data.

Practical Use Cases

• Modularizing OS kernels and bootloaders, where code reuse and precise control over registers matters
• Interrupt Service Routines (ISRs), which are technically subroutines invoked by hardware/software interrupts rather than CALL
• Optimized inner loops in performance-critical libraries (codecs, cryptography, DSP) where a hand-written subroutine outperforms compiler-generated code
• Reverse engineering and malware analysis, where recognizing subroutine boundaries (prologues/epilogues) is the first step to understanding a disassembled binary

Debugging Subroutines

When debugging with GDB or a similar debugger, a few commands become second nature:

• break function_name — set a breakpoint at a subroutine’s entry
• step / stepi — step into a call
• finish — run until the current subroutine returns
• info registers — inspect register state, including RSP/RBP
• x/10i $rip — disassemble instructions near the current position

A classic debugging scenario: your program crashes with a segmentation fault right after a RET. Nine times out of ten, this means the stack was unbalanced somewhere inside that subroutine — an extra PUSH without a matching POP, or a RET that fires before the stack is properly restored.

Optimization and Performance Considerations

Function calls aren’t free. Each CALL/RET pair involves memory writes and reads on the stack, and can disrupt the CPU’s branch predictor and return address stack predictor (a small hardware cache that predicts where RET will jump to). A few practical considerations:

• Inlining: For very small, frequently-called subroutines, compilers often inline the code instead of generating a CALL. In hand-written Assembly, you can do this manually with macros.
• Leaf function optimization: On ARM, a leaf function (one that doesn’t call anything else) can skip saving LR to the stack entirely, since it never gets overwritten.
• Register preservation overhead: Following the calling convention properly (saving callee-saved registers like RBX, RBP, R12–R15 on x86-64) adds a few instructions per call — necessary for correctness, but a cost nonetheless.

Comparing Subroutine Techniques

Common Mistakes

1. Forgetting to balance the stack — every PUSH inside a subroutine needs a matching POP before RET.
2. Clobbering caller-saved registers without warning — always check your calling convention’s register rules.
3. Ignoring return value width — moving a 32-bit result into EAX automatically zero-extends into RAX on x86-64, but this behavior differs across instruction forms.
4. Recursive subroutines without a base case, leading to stack overflow.
5. Overwriting LR on ARM without saving it first when the subroutine itself calls another subroutine.

Best Practices

• Always document your subroutine’s calling convention (which registers hold arguments, which hold return values, which are preserved).
• Use comments to mark the prologue and epilogue clearly.
• Keep subroutines focused on a single task — the same “single responsibility” principle from high-level programming still applies.
• Use consistent naming conventions for labels so a disassembler or a teammate can follow the logic.

FAQs

Q: Is a subroutine the same as a function? Conceptually, yes — “subroutine” is simply the Assembly-level term for what higher-level languages call a function or procedure.

Q: Can a subroutine return multiple values? Yes, by using multiple registers (e.g., RAX and RDX on x86-64) or by passing a pointer to a memory location where results are written.

Q: What happens if I call a subroutine without enough stack space? You’ll eventually hit a stack overflow, which typically manifests as a segmentation fault when the stack pointer moves outside the memory region the OS has mapped for the stack.

Q: Why does ARM use a Link Register instead of pushing to the stack automatically? It’s a performance decision — many function calls in real programs are to leaf functions, and avoiding automatic stack traffic for those calls saves cycles.

Summary and Key Takeaways

Subroutines are how Assembly language achieves code reuse, modularity, and (eventually) recursion — all without any of the syntactic sugar that higher-level languages provide. The stack (or, on ARM, the Link Register) is the mechanism that makes returning to the right place possible, and calling conventions are the “social contract” that lets different pieces of code — even code written in different languages — cooperate correctly.

If you take one thing away from this post, let it be this: every CALL must be matched by a RET, and every register you touch that the caller expects to survive must be saved and restored. Master that discipline, and subroutines stop being scary — they become just another tool in your toolbox.

References

• Intel® 64 and IA-32 Architectures Software Developer’s Manual, Volume 1: Basic Architecture — intel.com/sdm
• AMD64 Architecture Programmer’s Manual — amd.com
• ARM Architecture Reference Manual (ARMv8-A) — developer.arm.com
• Procedure Call Standard for the Arm Architecture (AAPCS) — developer.arm.com
• GNU Assembler (GAS) Documentation — sourceware.org/binutils