Abstract Syntax Trees: The Program’s Internal Map
To transcend the limitations of simple regex match-and-replace operations, modern static code analysis tools require a deeper understanding of the code’s intrinsic properties. This includes distinguishing between a function and a variable, grasping class inheritance hierarchies in object-oriented languages, and so forth. This nuanced understanding is typically expressed in the form of an Abstract Syntax Tree (AST). An AST is a tree-like representation of the syntactic structure of a program’s source code, abstracting away from the concrete syntax of the language and focusing on the essential elements of the program.
ASTs serve a far more fundamental purpose than just code analysis: compilers routinely use ASTs as an intermediate representation of source code. This intermediate form allows compilers to quickly perform crucial optimizations and syntax checks before the code is ultimately compiled down to machine code.
You can readily visualize an AST using Python’s built-in ast module. To try this out for yourself, save the following code in a script named ast_example_1.py:
import ast # Import Python's built-in Abstract Syntax Tree module
# Python source code (as a string) to be parsed into an AST
code = """
name = 'World'
print('Hello,' + name)
"""
# Parse the source code string into an AST tree
tree = ast.parse(code)
# Pretty-print the AST tree structure
print(ast.dump(tree, indent=4))Run the script to convert the source code into its AST representation:
python ast_example_1.pyYou should receive output similar to this:
Module(
body=[
Assign(
targets=[
Name(id='name', ctx=Store())
],
value=Constant(value='World')
),
Expr(
value=Call(
func=Name(id='print', ctx=Load()),
args=[
BinOp(
left=Constant(value='Hello,'),
op=Add(),
right=Name(id='name', ctx=Load())
)
],
keywords=[]
)
)
],
type_ignores=[]
)
The output is clearly organized in a tree structure. The Module node serves as the root, from which child nodes like Assign, Expr, and Call branch off, representing assignments, expressions, and function calls, respectively.
Now, let’s consider a practical scenario for static analysis. Suppose that print is identified as a dangerous sink function—a function that, if supplied with untrusted input, could lead to a security vulnerability (e.g., command injection, information disclosure). You want to determine if executing the following Python code will actually call print in a way that could be problematic:
Python
def old_greet(name): # ¶ Define a function named old_greet that takes one argument 'name'
print('Hello, ' + name) # Print a greeting message by concatenating 'Hello, ' with the given name
yell = print # • Assign the built-in print function to a new variable called 'yell'
yell('HELLO, WORLD') # Call 'yell' (which is just print), so this prints: HELLO, WORLDThe source code defines a simple function old_greet that prints a string. The code then assigns the built-in print function to the yell variable before calling yell with a string argument.
A naive approach to finding calls to print would be to use a regular expression, such as /print\([^)]*\)/g. However, this approach would lead to both a false positive and a false negative.
- It would produce a false positive for the
printcall inside theold_greetfunction (marked¶). Althoughold_greetcallsprint, theold_greetfunction itself is never actually invoked in the provided script, so that specificprintstatement would not be executed. - It would result in a false negative for the call via
yell(marked•). Theyellvariable does ultimately callprint, but due to the reassignment (yell = print), a simple regex looking for “print(” would entirely miss this execution path.
A regular expression capable of dealing with all possible edge cases, even for such simple code, would quickly become incredibly complex, unwieldy, and notoriously difficult to debug and maintain.
Instead, a more robust and accurate approach is to traverse the AST to identify all the Call nodes that will actually occur based on the semantic meaning and execution flow dictated by their parent nodes. Let’s use the ast module again to convert the code into its AST. Save the sample code above in a file named sample_code.py, then create a script called ast_example_2.py in the same directory with the following contents:
import ast # Import the Abstract Syntax Tree module for parsing and analyzing Python code
import os # Import os module to work with file paths and directories
# Get the directory path where the current script is located
cur_dir = os.path.dirname(os.path.abspath(__file__))
# Open the file named 'sample_code.py' located in the same directory as the current script
with open(os.path.join(cur_dir, 'sample_code.py')) as f:
# Read the file's contents and parse it into an AST tree
tree = ast.parse(f.read())
# Pretty-print the AST tree structure using indentation
print(ast.dump(tree, indent=4))The output of ast_example_2.py should look something like this:
Module(
body=[
FunctionDef(
name='old_greet',
args=arguments(
args=[
arg(arg='name')]),
body=[
Expr(
value=Call(
func=Name(id='print', ctx=Load()),
args=[
BinOp(
left=Constant(value='Hello, '),
op=Add(),
right=Name(id='name', ctx=Load()))]))]),
Assign(
targets=[
Name(id='yell', ctx=Store())],
value=Name(id='print', ctx=Load())),
Expr(
value=Call(
func=Name(id='yell', ctx=Load()),
args=[
Constant(value='HELLO, WORLD')]))])

Armed with knowledge of what each node in the AST represents, you can efficiently traverse the tree by intelligently deciding which branches to follow. For instance, you would focus on nodes like Assign (where variables are assigned values) and Expr (representing expressions that are evaluated), while largely ignoring FunctionDef nodes and similar constructs unless the defined function is explicitly called elsewhere in the execution flow of the script. By diligently tracking variables that are affected by Assign operations (like yell = print), you can semantically follow the data flow. This allows you to correctly identify that the execution path in the tree genuinely reaches a Call node whose func attribute’s effective value is indeed print.
The inherent tree structure of ASTs facilitates the use of various optimized algorithms to query for specific information, avoiding the waste of compute cycles on “pruned branches” that are irrelevant to the current analysis goal.
Beyond ASTs, other powerful code representations exist for automated analysis:
- Control Flow Graph (CFG): A CFG models the potential paths through a program during execution. It represents the flow of control in a program, where nodes are basic blocks (sequences of instructions with one entry and one exit point) and edges represent possible transfers of control. This allows for even more advanced and targeted queries on the code, such as reachability analysis, which determines which parts of the code can actually be reached during execution, helping to discard unexecutable code paths.
- Data Flow Graph (DFG): While CFGs are primarily concerned with the order of execution in a program (e.g.,
if-elsestatements, loops), DFGs focus on the propagation and transformation of data. DFGs model how data (including variables and expressions) moves through the program and how it’s modified. Both CFGs and DFGs are invaluable representations of code for conducting sophisticated automated analysis.
All this theoretical background is critical for truly understanding how modern static code analysis tools function under the hood. Any abstraction, by its very nature, inevitably loses some level of detail compared to the raw source code. While manual code analysis may offer greater comprehensiveness in this regard, it is often simply not feasible to manually review millions of lines of code, a common characteristic of complex software. In such cases, static code analysis tools are extraordinarily useful. A solid understanding of their underlying strengths (like AST traversal and data/control flow analysis) and their inherent weaknesses (e.g., difficulty with dynamic features, path explosion) will enable you to deploy them more effectively to support and enhance your overall code analysis strategy.
Not all source code analysis tools are created equal. Significant differences in their underlying abstractions and querying methods directly impact how effectively a tool can search for specific patterns and identify vulnerabilities in code.
CodeQL: Database-Driven Code Analysis
CodeQL is a powerful code analysis engine with deep roots in academia. It was originally developed by a research team at Oxford University, which created an innovative object-oriented query language (initially named .QL) capable of querying a relational database containing a meticulously modeled representation of source code. This fundamental database-centric approach is one of the key distinctions between CodeQL and more lightweight tools like Semgrep; CodeQL necessitates building a comprehensive database of the target codebase before any queries can be performed.
For compiled languages such as C and C++, CodeQL integrates seamlessly with the language’s build system (e.g., make). During the compilation process, CodeQL “observes” the build and extracts relevant information to populate its database. For non-compiled languages like Python, CodeQL employs specialized extractors that parse the code and then store the extracted information into the database.
Unsurprisingly, CodeQL’s query language bears many similarities to traditional database query languages like SQL. For example, consider this CodeQL query designed to find calls to the print function in Python:
import python
from Call call, Name name
where call.getFunc() = name and name.getId() = "print"
select call, "call to 'print'."
The CodeQL classes (Call, Name) share the exact same names as the types found in Python’s built-in ast module. This is because CodeQL’s Python extractor leverages the ast module, along with its own extended semmle.python.ast class, to parse Python codebases. Similarly, many of CodeQL’s other language extractors integrate deeply into their target programming language’s specific contexts. For instance, CodeQL’s Go extractor also utilizes the Go standard library’s go/ast package (https://github.com/github/codeql/blob/820de5d/go/extractor/extractor.go). This highly customized extraction approach for each language allows CodeQL to construct exceptionally comprehensive databases that capture intricate data flow and control flow relationships within the code.
With CodeQL’s in-depth analytical capabilities, you can craft incredibly powerful global taint tracking queries to uncover complex source-to-sink vulnerabilities. Furthermore, CodeQL’s object-oriented query language promotes excellent code reuse, allowing you to build modular and maintainable security queries. The following example effectively illustrates CodeQL’s strengths in this domain.
Multifile Taint Tracking Example
Consider a Node.js web API server built using the popular Express framework, which consists of two separate files: index.js and utils.js. This web API exposes a single /ping endpoint that is designed to ping any IP address provided in the ip query parameter. Unfortunately, the developer has inadvertently introduced a remote code execution as a service feature due to a subtle command injection vulnerability:
const express = require("express"); // ¶ Import the Express framework for building web apps
const { ping } = require("./utils.js"); // Import the 'ping' function from the local 'utils.js' file
const app = express(); // Create a new Express application instance
// Define a GET route at /ping
app.get("/ping", (req, res) => {
const ip = req.query.ip; // • Get the 'ip' query parameter from the request (user-controlled input)
// Call the ping function with user input and send back the result
res.send(`Result: \n${ping(ip)}`);
});
app.listen(3000); // Start the server and listen on port 3000You cannot definitively determine whether this vulnerability exists by merely analyzing the code of index.js in isolation. While this file clearly introduces a source of user-controlled data in req.query.ip (marked •), you crucially need to check whether the ping function, which is imported from utils.js (marked ¶), then passes this ip argument to a dangerous sink function.
const { execSync } = require("child_process"); // Import execSync to execute shell commands synchronously
// Export a function named 'ping' that takes an IP or hostname as input
exports.ping = (ip) => {
try {
// ¶ Execute the ping command using the user-supplied 'ip' value
// Sink: `ip` is directly interpolated into the shell command — this is dangerous!
// If `ip` = "8.8.8.8; rm -rf /", it could execute unintended or malicious commands
return execSync(`ping -c 5 ${ip}`);
} catch (error) {
// Return the error message if the ping fails
return error.message;
}
};Unfortunately, the ping function does indeed pass the ip parameter directly into the execSync function (marked ¶). execSync executes a shell command using its first argument. This means an attacker can execute any arbitrary command by simply crafting a malicious ip query parameter, such as ;whoami. Although this is a relatively straightforward code review task for a human, the process of tracing the source-to-sink flow in a multifile context often stumps most regex-based searches, as they cannot easily correlate imported functions and data flow across different files.
Fortunately, CodeQL excels at this kind of analysis because it models the code as a Data Flow Graph (DFG) and further extends this with taint tracking capabilities. While general data flow analysis follows the propagation of data (such as the value of a variable), it doesn’t necessarily keep track of other variables that might become “tainted” due to operations on the original tainted data. The separate DataFlow and TaintTracking libraries provided by CodeQL reflect this distinction.
Additionally, CodeQL provides convenient, pre-defined classes for common sources (where untrusted data enters the application) and sinks (where untrusted data could be used in a dangerous way), including generic remote user input and various command execution functions. As such, a potent global taint tracking rule for the previously identified vulnerable Node.js server can be concisely written like this:
/**
* @id remote-command-injection
* @name Remote Command Injection
* @description Passing user-controlled remote data to a command injection.
* @kind path-problem
* @severity error
*/
import javascript
module RemoteCommandInjectionConfig implements DataFlow::ConfigSig {
predicate isSource(DataFlow::Node source) {
¶ source instanceof RemoteFlowSource // Defines remote user input as a source
}
predicate isSink(DataFlow::Node sink) {
• sink = any(SystemCommandExecution sys).getACommandArgument() // Defines command arguments as a sink
}
}
module RemoteCommandInjectionFlow = TaintTracking::Global<RemoteCommandInjectionConfig>;
import RemoteCommandInjectionFlow::PathGraph
from RemoteCommandInjectionFlow::PathNode source, RemoteCommandInjectionFlow::PathNode sink
where RemoteCommandInjectionFlow::flowPath(source, sink)
select sink.getNode(), source, sink, "taint from $@ to $@.", source.getNode(), "source", sink, "sink"
For now, don’t get bogged down by the exact details of CodeQL’s syntax. Instead, focus on the general structure of the query, particularly the taint tracking configuration. This configuration clearly defines what constitutes a source (in this case, instances of RemoteFlowSource, marked ¶, representing remote user input) and what constitutes a sink (any command argument passed to a SystemCommandExecution instance, marked •). This configuration is essentially all you need to define the conditions for tracking the flow of attacker-controllable data to a vulnerable function call. The actual select query then checks whether there is a complete “flow path” from any defined source to any defined sink. If such a path exists, CodeQL outputs the results in a structured format that it can parse into comprehensive, step-by-step paths of the tainted data flow.
"results" : [
{
--snip--
"codeFlows" : [
{
"threadFlows" : [
{
"locations" : [
{
"location" : {
"physicalLocation" : {
"artifactLocation" : {
"uri" : "index.js",
"uriBaseId" : "%SRCROOT%",
"index" : 1
},
"region" : {
"startLine" : 7,
"startColumn" : 16,
"endColumn" : 28
}
},
"message" : {
¶ "text" : "req.query.ip"
}
}
},
--snip--
{
"location" : {
"physicalLocation" : {
"artifactLocation" : {
"uri" : "index.js",
"uriBaseId" : "%SRCROOT%",
"index" : 1
},
"region" : {
"startLine" : 8,
"startColumn" : 32,
"endColumn" : 34
}
},
"message" : {
• "text" : "ip"
}
}
},
--snip--
{
"location" : {
"physicalLocation" : {
"artifactLocation" : {
"uri" : "utils.js",
"uriBaseId" : "%SRCROOT%",
"index" : 0
},
"region" : {
"startLine" : 5,
"startColumn" : 21,
"endColumn" : 38
}
},
"message" : {
‚ "text" : "`ping -c 5 ${ip}`"
}
}
}
]
}
]
}
]
}
]
CodeQL accurately tracks the tainted data from its origin as the req.query.ip request query parameter value (marked ¶) to the ip variable (marked •), and finally to the template string (ping -c 5 ${ip}) passed as an argument to execSync in utils.js (marked ‚). If you were to run a simpler global data flow analysis by replacing TaintTracking::Global with DataFlow::Global in the configuration, you would likely get no results for this specific vulnerability. This is because standard data flow analysis strictly follows only the preserved data value of req.query.ip. The use of a template string within the argument passed to execSync means that the literal value of req.query.ip is no longer directly “preserved” as a distinct variable; it has been transformed into part of a new string. This transformation typically terminates a pure data flow path. If, however, utils.js had simply used execSync(ip) (without the template string), the data flow analysis would have worked just as well.
The immense power of global taint tracking in CodeQL comes with significant trade-offs: migrating from local to global analysis, as well as from pure data flow to full taint tracking, is inherently more computationally expensive and can sometimes be less precise (leading to more false positives if rules are not carefully crafted). Additionally, the CodeQL rule syntax itself is fairly complex. CodeQL rules are written in QL, which is an object-oriented programming language designed specifically for making queries. This explains why the initial part of RemoteCommandInjection.ql resembles typical object-oriented code, complete with classes and method overrides, while the final query section at the end is reminiscent of a traditional database query language, featuring from, where, and select clauses.
To use CodeQL effectively, you essentially need to learn a new programming language (QL) and become intimately familiar with the extensive CodeQL standard libraries. While this represents a significant learning curve, many security researchers find this a worthwhile pursuit because the query-oriented nature of QL allows you to express incredibly complex relationships and predicates, enabling the creation of truly powerful global taint tracking queries. The CodeQL developers have continuously augmented the tool with many helpful classes specifically tailored for common frameworks such as Express (Node.js), Spring (Java), and Ruby on Rails (Ruby). For example, instead of using the generic RemoteFlowSource in the example query, you could specifically employ Express::RequestSource to precisely track inputs originating from an Express framework web request. On the other hand, a potential drawback is the constant context switching required as you mentally toggle between analyzing the target codebase and formulating the intricate CodeQL query.
VS Code Extension: Streamlining CodeQL Development
To significantly minimize the friction involved in developing and refining CodeQL queries, you can leverage the CodeQL extension for Visual Studio Code. This powerful extension seamlessly integrates with the CodeQL CLI (Command Line Interface), adding a suite of intuitive UI elements and features directly into the VS Code editor. The result is a comprehensive Integrated Development Environment (IDE) specifically tailored for writing queries in QL.
Although the CodeQL VS Code extension comes bundled with its own internal CodeQL CLI, its official documentation provides a crucial caveat:
The extension-managed CodeQL CLI is not accessible from the terminal. If you intend to use the CLI outside of the extension (for example to create databases), we recommend that you install your own copy of the CodeQL CLI.
Since you will be actively creating your own CodeQL databases locally (a common practice in vulnerability research), it’s essential that you first install your own standalone copy of the CodeQL CLI.
Installation Steps (for Kali Linux example):
- Download CodeQL CLI: Download the latest release of the CodeQL CLI bundle from the official GitHub releases page:
https://github.com/github/codeql-action/releases.- For example, in Kali Linux, you might use
wget:
(Note: Replace$ wget https://github.com/github/codeql-action/releases/download/codeql-bundle-v2.20.2/codeql-bundle-linux64.tar.gzv2.20.2with the latest version available.)
- For example, in Kali Linux, you might use
- Extract the Archive: Extract the downloaded tarball to a convenient location:
$ tar -xzvf codeql-bundle-linux64.tar.gz - Add to PATH: Add the extracted CodeQL directory to your system’s
PATHenvironment variable so you can runcodeqlcommands from any terminal. If you’re usingzsh(common in Kali):
(Adjust$ echo "export PATH=\$PATH:$(pwd)/codeql" >> ~/.zshrc $ source ~/.zshrc~/.zshrcto~/.bashrcor your shell’s configuration file as needed.) - Verify Installation: Confirm the CLI is correctly installed and accessible:
(The$ codeql version --snip-- Unpacked in: /home/kali/Desktop/codeql Analysis results depend critically on separately distributed query and extractor modules. To list modules that are visible to the toolchain, use 'codeql resolve packs' and 'codeql resolve languages'.Unpacked in:path will vary based on where you extracted the archive.)
After successfully installing the CodeQL CLI, proceed to set up the VS Code environment:
- Download & Install VS Code: Download Visual Studio Code from
https://code.visualstudio.com/download. For Kali Linux, download the.debpackage and install it:$ sudo apt install - Install CodeQL Extension: Open VS Code.
- Use the keyboard shortcut
CTRL + Pand typeext install GitHub.vscode-codeql. - Alternatively, click the Extensions button (the square icon) in the Activity Bar on the left, search for “CodeQL” in the marketplace, and then click “Install”.
- Use the keyboard shortcut
- Clone CodeQL Starter Workspace: Git clone the CodeQL starter VS Code workspace to a working directory of your choice. It’s crucial to recursively clone Git submodules to ensure you don’t miss important CodeQL library dependencies:
If you forget$ git clone --recursive https://github.com/github/vscode-codeql-starter--recursive, you may find that you’re missing important CodeQL libraries later on, leading to query compilation errors. - Open Workspace: Finally, open the CodeQL starter workspace file (
vscode-codeql-starter.codeworkspace) in VS Code viaFile▶Open Workspace from File...(orFile▶Open Workspace...and select the.codeworkspacefile directly).
This comprehensive setup will provide you with a robust CodeQL development environment where you can draft and test your QL queries.
However, before you can run your queries, you need to specify a CodeQL database generated from the target’s source code. While the VS Code extension offers the convenience of downloading pre-built CodeQL databases from remote sources like GitHub, for practical learning and customized analysis, you’ll create a database yourself using the multifile taint tracking example from the previous section.
Creating a CodeQL Database:
- Create Project Directory: Create a new project directory outside of your CodeQL starter VS Code workspace. Move the example code files (
index.jsandutils.js) into this new project directory. If you’re following along with the book’s example code repository (https://github.com/spaceraccoon/from-day-zero-to-zero-day), this project directory is located atchapter-03/command-injection-example/app. - Navigate to Parent Directory: Open your terminal and navigate to the parent directory of your project directory (e.g.,
cd chapter-03/command-injection-example). - Create Database: Execute the
codeql database createcommand:
This command instructs CodeQL to create a database named$ codeql database create --language javascript --source-root app example-database --snip-- Finished writing database (relations: 13.30 MiB; string pool: 4.78 MiB). TRAP import complete (2.1s). Finished zipping source archive (243.70 KiB). Successfully created database at /home/kali/Desktop/from-day-zero-to-zero-day/chapter-03/command-injection-example/example-database.example-database, targetingjavascriptcode, with the source files located in theappsubdirectory.
Now that you’ve successfully generated a CodeQL database from the example Node.js code, you can proceed to test the RemoteCommandInjection.ql query on it directly within VS Code:
Running a CodeQL Query in VS Code:
- Open CodeQL Sidebar: Back in your VS Code workspace, click the CodeQL button (the icon resembles a magnifying glass over a database) in the Activity Bar on the left. This will open the CodeQL sidebar, which contains several useful views, including “Databases,” “Variant Analysis Repositories,” “Query History,” and “AST Viewer.”
- Add Database: In the “Databases” view, click the
+button (or the icon to “Add a CodeQL database from a folder”). Navigate to and select theexample-databasedirectory you created earlier. - Add Database Source to Workspace: Once the database is loaded, right-click
example-databasein the “Databases” view within the CodeQL sidebar, then select “Add Database Source to Workspace.” This makes the original source code (from which the database was built) visible in your VS Code Explorer. - Switch to Explorer View: Switch to the “Explorer” view by clicking the files icon in the Activity Bar. You should now see a new folder in the file explorer sidebar, typically named
example-database source archive, which contains the originalindex.jsandutils.jssource code files. - View AST: Right-click on
index.js(located in the new source archive folder) and select “CodeQL: View AST.” This will open the “AST Viewer” view in the CodeQL extension sidebar, visually representing how the CodeQL database has parsed and modeled the code’s Abstract Syntax Tree. - Explore AST: Click any item within the “AST Viewer” view. This action will automatically open the corresponding source code file (
index.jsorutils.js) and highlight the relevant code segment. Conversely, you can select code directly in the editor to automatically locate and highlight the matching node in the AST Viewer. For example, if you select the lineres.send(Result: \n${ping(ip)});, the AST Viewer will inform you that this is anExprStmtnode with a childMethodCallExprnode. This interactive exploration is incredibly helpful for understanding CodeQL’s model and selecting the correct classes when you’re writing or refining a query. - Create/Copy Query File: Switch back to the “Explorer” view. Create (or copy from the book’s code repository) the
RemoteCommandInjection.qlquery file into thecodeql-custom-queries-javascriptdirectory within your CodeQL starter workspace. You cannot create a QL query file in isolation, as QL queries depend on aqlpack.ymlfile in the same directory (or a parent directory) to determine which CodeQL library dependencies to include. In this specific case, your query needs to include thecodeql/javascript-alllibrary. - Run Query: Finally, right-click the
RemoteCommandInjection.qlquery file in the Explorer and select “CodeQL: Run Queries in Selected Files.”
The VS Code extension will then trigger the CodeQL CLI to execute your query against the example-database and parse the results. If everything is configured correctly, you should see a nicely formatted results view emerge in the editor region on the right side of VS Code.
If you expand a row in the results, you’ll be presented with a detailed list of each taint step from the source of the untrusted data to the identified sink. Clicking on any individual step within this list directly links you to the exact location in the source code corresponding to that taint step. This feature is invaluable for thoroughly analyzing query results, verifying findings, and debugging any draft queries you are developing.