In Part 1, we explored Recast & Detour, the library used by Unreal Engine for NavMesh generation.

Building upon that foundation, we now delve into the implementation details within Unreal for static NavMesh generation. As a reminder, a static NavMesh does not support runtime modifications to its geometry (in contrast to a dynamic NavMesh).

Topics covered in this article:

• Overview of systems and configuration options involved in the generation process.

• Code analysis (UE 5.2).

While extensive, this article will not cover dynamic NavMeshes, world partitioning, runtime pathfinding, and other important topics. These areas will be addressed in subsequent posts, giving them the focus they deserve.

For now, let’s focus on static NavMesh generation, from which all future topics will build upon.

Overview

Agents

An agent defines an entity that will navigate a NavMesh.

Navigation data is generated for each agent you configure.

You can configure as many agents as your project requires, each represented by a set of characteristics such as a radius, height and max step height. These properties are used as part of the generation process, such as for eroding walkable areas.

Each agent configuration sets a navigation data type, with RecastNavMesh being the default (hence Part 1 discussing Recast & Detour). Unreal gives you the flexibility of implementing your own type (e.g., navigation data not generated using Recast), though that is outside the scope of this series.

Boundaries

A NavMeshBoundsVolume defines the boundaries where NavMesh data is generated.

By mapping agents to a NavMeshBoundsVolume, we define what navigation data type is generated within its boundaries. A volume can support multiple agents.

Navigation Data

With agents and boundaries defined, the NavMesh generation process is able to generate navigation data.

The process takes into account the intersecting geometry within the defined boundaries, the agent’s generation parameters (e.g., agent radius, height, etc.), and the chosen navigation data type (e.g., Recast library).

For each supported agent, a unique instance of ARecastNavMesh stores the navigation data tailored to their characteristics.

Tiles and Layers

The 3D boundaries of a NavMeshBoundsVolume are made up of tiles of a fixed size, with each tile storing a portion of the navigation data. These tiles are further categorized into layers based on the height range they cover.

This subdivision allows for Unreal to leverage its Task system, a mechanism specifically designed for asynchronous work. This enables concurrent processing of tiles, effectively reducing generation runtime.

Within the Recast library, dtMeshTile represents a tile and holds the polygons and vertices of the NavMesh area they cover. They are stored as a linked list within a parent dtNavMesh.

Resolution

The NavMesh generation process, as explained in Part 1, starts with the generation of a heightmap (used for voxelization), with its cell size assuming a crucial role. All subsequent steps rely on this initial step.

Unreal provides low, default, and high resolution presets for configuring the heightmap’s cell dimensions.

The resolution presets are configured in the project settings and are applied per-tile during the NavMesh generation process.

Resolution selection at runtime is determined by NavModifierVolumes overlapping generation boundaries, e.g., NavMeshBoundsVolume.

The choice of resolution depends on gameplay and performance requirements. Opting for higher resolutions, achieved by using smaller cells, results in more detailed and precise NavMeshes. However, they also lead to increased memory usage, generation time, impacted runtime performance (due to having more data to process), and overfitting (e.g., minor geometry details are considered, yet are not necessarily relevant for navigation).

Code Analysis (UE 5.2)

Class Overview

UNavigationSystemV1

• Oversees the entire navigation system within Unreal, including all instances of ARecastNavMesh.

FNavDataConfig

• Describes an agent for which NavMesh data can be generated.

  • Name, radius, height, navigation data type, etc.

• Owned by UNavigationSystemV1.

ARecastNavMesh

• High-level manager class serving as a bridge between Unreal and the Recast library.

• Provides access to generated Recast data, accessible from its owned FPImplRecastNavMesh instance.

• Offers utility functions for various navigation-related operations.

  • ex: GetPolyNeighbors(), GetPolyEdges(), etc.

• Owned by UNavigationSystemV1.

FPImplRecastNavMesh

• Handles the private implementation details of ARecastNavMesh.

• Wraps access to NavMesh data through owned pointers to Recast types (e.g., dtNavMesh, dtMeshTile).

• Owned by ARecastNavMesh.

FRecastNavMeshGenerator

• Manages the tile-based generation process by overseeing instances of FRecastTileGenerator. It leverages Unreal's Task system to parallelize the execution of all FRecastTileGenerator instances, maximizing performance during the generation process.

• Owned by ARecastNavMesh.

FRecastTileGenerator

• Responsible for generating navigation data for layered tiles within specific boundaries using the Recast library

• Owns an array of FNavMeshTileData, which stores the navigation data for all layered tiles corresponding to the position covered by this generator.

• Owned by FRecastNavMeshGenerator.

FNavMeshTileData

• Holds the data generated by Recast for a single layered tile.

• Owned by FRecastTileGenerator.

dtNavMesh

• A low-level Recast type holding a NavMesh’s entire data set, including all layered tiles (e.g., dtMeshTile).

• Owned by FPImplRecastNavMesh.

dtMeshTile

• A low-level Recast type that stores all the polygons and vertices for a layered tile. It utilizes a header (e.g., dtMeshHeader) to store metadata that uniquely identifies its corresponding tile (e.g., x/y position, layer index, etc.).

• Holds a pointer to the next layered tile.

• Owned by dtNavMesh.

Logic & Data Flow

Non-essential statements have been excluded from code snippets to highlight only statements that facilitate the comprehension of the static NavMesh generation logic.

Function names remain unchanged and can serve as references to the original code accessible from Epic's Git repository (v5.2).

The goal is not to provide a per-statement analysis of the code, but rather describe the overall logic and data flow.

The (x) syntax references a numbered statement in the code snippet being discussed.

Entry Point

UNavigationSystemV1::Build() serves as the entry point to the generation process.

First, the function checks if there are any ANavMeshBoundsVolume instances with valid boundaries (1). These volumes define areas where NavMesh data will be generated.

If at least one valid boundary exists, SpawnMissingNavigationData() (2) creates an ARecastNavMesh instance per supported agent type. An agent type is considered supported if a ANavMeshBoundsVolume is configured with support for it.

ARecastNavMesh is a high-level manager class that functions as a bridge between Unreal and Recast. To facilitate quick lookup, a mapping is maintained between agent types and their corresponding ARecastNavMesh instance (3).

ARecastNavMesh does not directly interact with the Recast library. Instead, it delegates this responsibility to its unique instance of FRecastNavMeshGenerator, which is constructed by invoking the RebuildAll() function (4, 5).

We’ll cover the EnsureBuildCompletion() function (6) when we discuss FRecastTileGenerator.

FRecastNavMeshGenerator

FRecastNavMeshGenerator is unique to each instance of ARecastNavMesh (7, 8). Its primary purpose is to coordinate the parallelization of agent-specific work for all tiles within the boundaries where NavMesh data should be generated.

Reminder: Each supported agent type has its own dedicated ARecastNavMesh. As FRecastNavMeshGenerator is owned by an instance of ARecastNavMesh, the data produced by the generator is also specific to that particular agent.

The generator’s Init() function (9) configures the properties that will be used during the generation process. It also defines the boundaries within which navigation data will be generated. Finally, it determines the maximum number of Tasks that will execute parallel work.

Following initialization, the RebuildAll() function is invoked (10):

• (11) ConstructTiledNavMesh():

  • Allocates memory for low-level Recast objects (e.g., dtNavMesh, dtMeshTile) to store generated NavMesh data.

  • These objects will be used by Unreal at runtime to execute navigation-related queries.

• (12) MarkNavBoundsDirty():

  • Creates an FPendingTileElement for each tile requiring NavMesh data generation.

  • Those instances are stored in the PendingDirtyTiles array.

All that’s left is to run asynchronously the generation of navigation data for all dirty tiles.

Tile Generators

At this stage, all the necessary ARecastNavMesh and FRecastNavMeshGenerator instances have been created, with each generator holding a collection of tiles to process (e.g., PendingDirtyTiles).

The execution flow now returns to the entry point of the generation process, UNavigationSystem::Build(), where EnsureBuildCompletion() is called on each ARecastNavMesh instance (6).

ARecastNavMesh forwards the call to its FRecastNavMeshGenerator, which starts processing its PendingDirtyTiles collection by invoking ProcessTileTasksAndGetUpdatedTiles() (13).

Since there might be more tiles to process than the maximum number of tasks permitted to run in parallel (NumTasksToProcess), this function is invoked as many times as needed until data generation is completed for all tiles.

ProcessTileTasksAndGetUpdatedTiles() creates a unique instance of FRecastTileGenerator per tile requiring generation (14).

FRecastTileGenerator will interface directly with the Recast library.

Each FRecastTileGenerator is then wrapped within an FRecastTileGeneratorTask to handle its asynchronous execution (15).

Launching the task is done through StartBackgroundTask(), with progress monitored by adding an FRunningTileElement entry to the RunningDirtyTiles collection (16). After processing, the corresponding tile entries are removed from PendingDirtyTiles (17).

For debugging purposes, the global variable GNavmeshSynchronousTileGeneration can be set to force synchronous execution of tile generators.

To understand how navigation data is generated by FRecastTileGenerator you can examine its DoWork() function. The code interfaces with the Recast library and largely follows the same steps discussed in Part 1:

• Rasterization of geometry triangles into a 3D grid of voxels (e.g., heightfield).

• Filtering of generated heightfield and conversion into a compact heightfield.

• Creation of a distance field from heightfield data.

• Generation of layered data, e.g., rcBuildHeightfieldLayers()

With generation completed, FRecastTileGenerator stores the navigation data of each layered tile within instances of FNavMeshTileData. These are stored in its NavigationData array.

The final step involves FRecastNavMeshGenerator retrieving the content of the NavigationData array from all of its completed FRecastTileGeneratorTask instances. This data is then used to populate the dtNavMesh Recast object previously created in step (11).

The dtNavMesh object will be utilized by Unreal at runtime to execute navigation-related queries.

This article has provided a comprehensive overview of static NavMesh generation in Unreal Engine. We've explored the key components and processes involved, from defining agents, boundaries and resolutions to the coordination of asynchronous tile-based generation using ARecastNavMesh, FRecastNavMeshGenerator and FRecastTileGenerator.

Through detailed code analysis and logic flow, we've gained insight into how Unreal leverages its Task system to efficiently generate navigation data tailored to each agent's characteristics.

While this article focused on static NavMesh generation, it's only the beginning of our journey into Unreal Engine's navigation capabilities.

In the next installment, we will delve into the dynamic NavMesh generation process, where we'll tackle real-time modifications, obstacles, and other dynamic aspects essential for responsive navigation in dynamic game environments.

• Part 2 - Static Generation

A navigation mesh (NavMesh) is a set of interconnected polygons that are generated through an analysis of the environment. They represent the walkable surfaces.

The NavMesh serves as a guiding tool for AI-controlled entities, allowing them to navigate the game world while avoiding obstacles. It provides spatial information about the environment, such as the type of walkable surfaces, height and elevation data, and other relevant attributes.

This information can be used by AI systems to make informed decisions, such as determining the nearest cover location or planning optimal paths.

Unreal Engine supports two types of NavMeshes:

• Global (also known as Unified)

• World-Partitioned (experimental as of UE 5.2)

The topic of world partitioning will be tackled in a follow-up article.

A Global NavMesh fully covers an area, it is not divided into smaller parts (i.e.: partitioned).

In order to reduce memory usage and improve performance, the polygons that make up the NavMesh are grouped together to form tiles, which provide additional granular control over which polygons are loaded or unloaded at runtime based on the proximity of the player or AI agents.

The quantity of tiles and polygons generated is determined by several factors, including the dimensions of the covered area, the complexity of the environment, and the generation configuration.

Before delving deeper into the NavMesh generation process within Unreal Engine (a topic for the next article), it is essential to grasp the inner workings of the library on which it relies: Recast & Detour.

The Recast & Detour library shipping with UE had modifications made to it.

We will be covering “vanilla” v1.6.0 of Recast (latest version as of this writing).

The next article will discuss the modifications made by Epic, as well as how the library is integrated within UE.

Recast & Detour

Generating a NavMesh in Unreal creates an actor of type ARecastNavMesh. The Recast prefix originates from Epic’s use of the Recast & Detour open source library.

The library is made up of two toolsets: Recast, which handles the generation of the NavMesh, and Detour for pathfinding using the data generated with Recast.

The Recast generation pipeline consists of several steps. Initially, triangles within the NavMesh's generation boundaries are classified as walkable or non-walkable based on their slope, with the maximum slope angle being adjustable.

// Source: https://github.com/recastnavigation/recastnavigation
// File: Recast/Source/Recast.cpp

/* ========================================================================
   Triangle Categorization (i.e.: walkable or not)
   ======================================================================== */

void rcMarkWalkableTriangles(rcContext* context, 
                             const float walkableSlopeAngle,
                             const float* verts, const int numVerts,
                             const int* tris, const int numTris,
                             unsigned char* triAreaIDs)
{
	(...)

	const float walkableThr = cosf(walkableSlopeAngle / 180.0f * RC_PI);

	float norm[3];

	for (int i = 0; i < numTris; ++i)
	{
		const int* tri = &tris[i * 3];
		calcTriNormal(&verts[tri[0] * 3], &verts[tri[1] * 3], &verts[tri[2] * 3], norm);
		// Check if the face is walkable.
		if (norm[1] > walkableThr)
		{
			triAreaIDs[i] = RC_WALKABLE_AREA;
		}
	}
}

The triangles are converted (rasterized) into a 3D grid of voxels. This 3D grid, known as a heightfield, represents the environment by capturing both the ground surface and other structures or objects within it.

• A heightfield is composed of cells, which are square units on the XZ plane. The dimensions of a cell can be configured. The smaller the cell, the higher the resolution of the heightfield.

• As we iterate over all cells, we determine their overlap with the triangles making up the environment’s mesh - this overlap represents obstructed space. This is accomplished by taking a cell on the XZ plane and projecting it upwards. Anything that overlaps does so for a certain Y range. This range is used to create spans, representing the height variations where obstructions exist within the cells.

• A cell contains [0, X] spans.

• Based on the slope of the triangles overlapping with the cells, spans are marked as walkable or non-walkable.

• The elevation data from the spans, along with the position of the cells on the XZ plane, enables the construction of the 3D grid of voxels, accurately representing the terrain and obstacles within the environment.

// Source: https://github.com/recastnavigation/recastnavigation
// File: Recast/Source/RecastRasterization.cpp

bool rcRasterizeTriangles(...)
{
    // Invokes rasterizeTri()
}

static bool rasterizeTri(...)
{
    // Invokes dividePoly() and addSpan()
}

static void dividePoly(...)
{
}

static bool addSpan(...)
{ 
}

The heightfield is subjected to a series of filtering operations which aim to refine its data, removing any artifacts or noise, as well as enhance its accuracy.

// Source: https://github.com/recastnavigation/recastnavigation
// File: Recast/Source/RecastFilter.cpp

void rcFilterLowHangingWalkableObstacles(...)
{
    // If an unwalkable span has a walkable span directly under it, and the 
    // height difference (climb) between them is under a configured threshold, 
    // the unwalkable span is marked as walkable.
}

void rcFilterLedgeSpans(...)
{
    // Marks spans as unwalkable if close to a ledge or on a steep slope.
}

void rcFilterWalkableLowHeightSpans(...)
{
    // Remove walkable flag from spans which do not have enough
    // space above them for the agent to stand there.
}

Next, the heightfield undergoes a conversion process to become a compact heightfield, aimed at optimizing processing speed. As part of this optimization, the dynamically allocated memory used by the non-compact heightfield is reorganized into blocks of sequential memory.

By arranging the data to fit in blocks of memory that the computer's processor can retrieve at once from main memory (i.e.: cache lines), cache locality is improved. Improved locality reduces the occurrence of cache misses, where data needs to be fetched from main memory because it is not found in the CPU's caches (L1/L2/L3). This results in faster memory access times and overall improved performance, as the CPU spends less time waiting for data from main memory.

// Source: https://github.com/recastnavigation/recastnavigation
// File: Recast/Source/Recast.cpp

bool rcBuildCompactHeightfield(...)
{
}

Walkable areas are then eroded based on a walkable radius config. This ensures that the data takes into account the physical constraints of the entities that will eventually make use of the NavMesh. A larger walkable radius is used to accommodate larger entities, ensuring they have enough room to navigate the environment.

// Source: https://github.com/recastnavigation/recastnavigation
// File: Recast/Source/RecastArea.cpp

bool rcErodeWalkableArea(rcContext* ctx, int radius, rcCompactHeightfield& chf)
{
    // Any spans that are closer to a boundary or obstruction than the 
    // specified radius are marked as unwalkable.
}

With filtering complete, a distance field is created from the heightfield data.

The algorithm operates in two passes. In the first pass, it traverses the heightfield from the top left to the bottom right, calculating the walkable distance for each spans of a cell. The value is determined by considering the number of neighboring spans that are walkable. To ensure complete coverage, the process is repeated in a second pass, this time traversing the heightfield from the bottom right to the top left.

The distances are then “blurred” to smooth the transitions between values.

// Source: https://github.com/recastnavigation/recastnavigation
// File: Recast/Source/RecastRegion.cpp

bool rcBuildDistanceField(...)
{
}

static unsigned short* boxBlur(...)
{
}

The heightfield is then partitioned into regions. Recast offers three different partitioning algorithms: Watershed, Monotone and Layer. Only Watershed relies on the previously described distance field. For all algorithms, the goal is to create simple regions by grouping cells based on their walkable property and environment connectivity.

Each algorithm has it’s advantages and disadvantages, detailing them is outside the scope of this article.

// Source: https://github.com/recastnavigation/recastnavigation
// File: Recast/Source/RecastRegion.cpp

bool rcBuildRegions(...)
{
}

bool rcBuildRegionsMonotone(...)
{
}

bool rcBuildLayerRegions(...)
{
}

Next, contours are extracted from the regions by identifying boundaries.

// Source: https://github.com/recastnavigation/recastnavigation
// File: Recast/Source/RecastRegion.cpp

bool rcBuildContours(...)
{
}

Finally, polygons are generated from the contours to form the NavMesh.

We’ve covered quite a few things today and while certain details were briefly mentioned, we will delve further into them in the next post, which will focus on how Unreal Engine leverages the Recast library. This upcoming article will provide a more comprehensive exploration of the topic.

I hope you found this post informative, and I look forward to sharing more insights in the next installment.

My name is Hussein - I’ve been working in the gaming industry for over a decade as a programmer for studios such as Eidos, Ubisoft and Gearbox.

Codebases, regardless of their size, often have complex logic and interdependencies that may not be immediately evident. This is compounded by code frequently switching hands, with the original developer often not involved in subsequent changes. As the initial vision evolves and new ideas are integrated, the complexity of the architecture can grow, sometimes stretching beyond its original design intentions.

This led me over the years to develop my own approach for delving into unfamiliar (and sometimes unwelcoming) code and trying to make sense of it all.

It starts with documenting systems in manageable pieces that we can always go back to in order to get a quick refresher. The goal is not to understand everything from the get-go, but rather to establish a foothold within the codebase which slowly expands, one step at a time, to eventually cover as much as possible.

This leads to understanding the overall logic, runtime data flow, and technical limitations. This in turn helps us recognize the impact of our changes, how to interface with the systems, as well as schedule work with a higher degree of confidence in estimates.

The official Unreal Engine documentation, while certainly useful, is not geared towards programmers and doesn’t provide that level of detail which I find important. The exercise of opening the codebase and analyzing it remains a requirement to understanding what makes the engine tick under the hood.

As such, I will be going through that exercise and documenting my own journey - hoping that some of my findings prove useful to you.

Join me as we uncover everything Unreal Engine has to offer.

Subscribe now