Simulation Engine

Simulation Engine#

Central to the POLARIS framework is the POLARIS discrete event engine (DEVE). This provides an API to: create an agent of any kind (be it a person, a vehicle, or even a weather system), describe when it wants to act, what it does when it does act, and then set it on its way to perform independently.

The agents created in the DEVE organize themselves first among iterations and then within sub-iterations. Iteration in this case provides a useful analogue for a clock ticking along at a fixed rate, while sub-iteration can be used to describe the sequencing of operations within this particular slice in time.

Agents can talk about what they want to do using “events”. These callback functions serve the purpose of allowing the agent to say when they want to be visited next, whether they want to do anything at this particular time, and what action the agent would like to take. It may be swapped in and out over the course of the simulation creating an agent which can react in a multi-faceted ways. External entities can affect the scheduled behavior of an agent through calling a reschedule function.

Memory Allocation Blocks#

The POLARIS memory allocator utilizes management techniques which are targeted at improving performance for agent-based simulations running in a discrete event paradigm. Agent-based code can differ from other types of applications as there is a very high demand for allocations/deallocations of homogenous objects which have the same type (for instance, traveler agents for each member of a population)

In general, within POLARIS, all allocated objects will be sub-classed from two types of object:

  1. Execution_Object - used for modelling simulation agents

  2. Data_Object - used for storing data or utility classes

When an instance of an Execution_Object is allocated (via Allocate<MyObject>), a slice of memory within a Execution_Block<MyObject> is marked and then used to initialize an object. If no blocks of MyObject exist yet, one will be created and marked as active.

During each simulation timestep, individual worker threads within the thread pool will iterate through each active block and then through each Execution_Object within the block. If an object is marked as requiring computation at the current simulation timestep, its Execute method will be called, when all objects and blocks have been processed control will return to the main thread and the simulation will move to the next timestep.

This is obviously a huge simplification of the entire process, but the benefits of the above approach are:

  1. By ensuring that all agents of a particular type are located closely together in memory, we can take advantage of cache locality effects to reduce the amount of time the CPU spends waiting for data.

  2. By doing record keeping of the next_timestep at the block level, we can potentially skip entire blocks of objects at some timesteps.

  3. By allocating in blocks we avoid the overhead of multiple small allocation calls and the memory fragmentation that would otherwise result.

This also comes with some caveats that users should be aware of when setting up their agent allocation strategy. The main difficulty comes from the tradeoff that must be made between small agents that will naturally be evenly distributed across threads and the overhead of having many small agents.

For example, if you allocate 20 agents and these agents all fit within a single block - that block will only ever be processed by a single thread. Thus if the amount of work being done by each agent is large, a significant bottleneck is introduced.

The solution to this problem is to set the “objects per block” explicitly - this can be done by calling MasterType::My_Agent::Average_Execution_Objects_Hint(num) before starting to allocate objects. A common strategy is to allocate only as many agents as there are threads and then use 1 object per block. This, however, requires that there is an alternate way to evenly divide the work across these agents. The skimming methods accomplish this, for example, by evenly dividing the zones across the agents.

However, we can’t always know ahead of time how many agents we will require and can’t set this explicitly. In such cases it becomes harder to provide concrete advice other than testing the performance of various strategies to ensure even CPU utilisation.

Hierarchy#

There are a number of terms thrown around in the code refering to various levels of management of the above concepts. Broadly those are as follows:

  • EX - Execution - World::Instance()->simulation_engine()->ex_next_revision(update_revision);

  • TEX - Type-Execution - DataType::component_manager->tex_next_revision(update_revision);

  • PTEX - Page-Type-Execution - _execution_block->ptex_next_revision(update_revision);

  • OPTEX - Object-Page-Type-Execution - These are actual instances (objects) that can be executed.

These terms are used to describe the places where event tracking takes place. Individual objects (OPTEX) track the next simulation step that they need to fire at, Pages of Objects (i.e. Execution Blocks) keep track of the next simulation step that ANY of the objects in that page need to fire, and so on up the chain.

Debugging#

If you set the EVENT_DEBUGGING preprocessor definition, then the event loop will spam your log file with the following records which are useful for tracking down changes in simulation ordering. It is only recommended to enabled this for small models (like Grid) as it generates a very large amount of data.

The syntax of emitted messages is events_fired: <iteration>,<sub-iteration>,<num-events>,<event-name>

2023-11-16 18:07:07,103 Thread: 140325060429568 | [INFO] events_fired: 198,4,2,Activity_Components::Implementations::ADAPTS_Activity_Plan_Implementation<MasterType, polaris::TypeList<polaris::NULLTYPE, polaris::NULLTYPE>, void>
2023-11-16 18:07:07,104 Thread: 140325060429568 | [INFO] events_fired: 198,4,1,Activity_Components::Implementations::ADAPTS_Activity_Plan_Implementation<MasterType, polaris::TypeList<polaris::NULLTYPE, polaris::NULLTYPE>, void>
2023-11-16 18:07:07,104 Thread: 140325060429568 | [INFO] events_fired: 198,4,1,Activity_Components::Implementations::ADAPTS_Activity_Plan_Implementation<MasterType, polaris::TypeList<polaris::NULLTYPE, polaris::NULLTYPE>, void>

This can be enabled from the command line (CLI) build scripts by adding “event_debugging” to your enable_libs:

python build.py -c -b -e "event_debugging"

or adding it in your build-config.json:

───────┬────────────────────────────────────────────────────────
       │ File: build-config.json
───────┼────────────────────────────────────────────────────────
   1   │ {
   2   │     "linux": {
   3   │         "deps_dir": "/opt/polaris/deps",
   .   |         ...
  11   │         "enable_libs": [
  12   │             "cplex",
  13   │             "event_debugging"
  14   │         ],
  ..   |         ...
  20   │         "working_dir": "/home/jamie/git/polaris-linux"
  21   │     }
  22   │ }
───────┴────────────────────────────────────────────────────────