Model Outputs

Model Outputs#

MOE Results as HDF5 Outputs#

Polaris uses HDF5 to store some of its largest outputs. HDF5 files (sometimes called just H5) store multidimensional arrays/matrices which can be organized into folders and given attributes (strings, numbers, arrays) to describe them.

Currently we store Link and Turn MOE results in the “{Model}-Result.h5” file. These are organised in a “LinkMOE” and “TurnMOE” folder respectively. Each folder has multiple tables (one per attribute), each of which is organized as a matrix with dimensionality Records x Timesteps. When writing to these tables, POLARIS appends a row of link/turn records for each timestep as it runs.

For example, if our timestep is 60 seconds, and we have 1500 links, all of your link tables (except for the ID/Length tables, which are arrays!) would have 1500 columns and 1440 rows (i.e. the number of minutes in the simulation)

LinkMOE Attributes	Description	Units
link_travel_time	Average travel time experienced by travelers to traverse the link in the time slice	seconds
link_travel_time_standard_deviation	deprecated
link_queue_length	deprecated
link_travel_delay	Average delay experienced by travelers on the link in the time slice	seconds
link_travel_delay_standard_deviation	deprecated
link_speed	Average speed of vehicles traversing the link in the time slice	meters/second
link_density	The number of vehicles per mile per lane	veh/mi/lane
link_in_flow_rate	The number of vehicles per interval entering the link	veh/hr/lane
link_out_flow_rate	The number of vehicles per interval exiting the link	veh/hr/lane
link_in_volume	The number of vehicles entering the link in the time slice	# of vehicles
link_out_volume	The number of vehicles exiting the link in the time slice	# of vehicles
link_speed_ratio	Ratio of link_speed to free flow speed of link	No unit
link_in_flow_ratio	Ratio of current in-flow to in-flow capacity	No unit
link_out_flow_ratio	Ratio of current out-flow to out-flow capacity	No unit
link_density_ratio	Ratio of link_density to jam density	No unit
link_travel_time_ratio	Ratio of average travel time to free-flow travel time	No unit
num_vehicles_in_link	The number of vehicles presently traversing the link in the time slice	# of vehicles
volume_cum_BPLATE	Cumulative volume of BPlate vehicles on the link in the time slice	# of vehicles
volume_cum_LDT	Cumulative volume of Light Duty vehicles on the link in the time slice	# of vehicles
volume_cum_MDT	Cumulative volume of Medium Duty vehicles on the link in the time slice	# of vehicles
volume_cum_HDT	Cumulative volume of Heavy Duty vehicles on the link in the time slice	# of vehicles
entry_queue_length	The number vehicles waiting to enter the link from outside the network in the time slice	# of vehicles
link_length	Length of the link	meters
link_uids	Link unique ID (derived from database ID and direction)	integer

TurnMOE Attributes	Description	Units
turn_penalty	Time delay in executing a turn	seconds
turn_penalty_sd	deprecated
inbound_turn_travel_time	Travel time to reach turn from inbound link	seconds
outbound_turn_travel_time	deprecated
turn_flow_rate	Vehicle count in the time slice that left the turn (**volume and not flow rate)	# of vehicles
turn_flow_rate_cv	CV Vehicle count in the time slice that left the turn (**volume and not flow rate)	veh/hr/turn
turn_penalty_cv	Time delay in executing a turn for a connected vehicle	seconds
total_delay_interval	Total turn delay for vehicles in that interval experienced in the time slice	seconds
total_delay_interval_cv	Total turn delay for connected vehicles in that interval experienced in the time slice	seconds
turn_penalty_by_entry	Similar to turn penalty but based on vehicle step entrance and not when vehicle left the turn (used for DTA convergence)	seconds
turn_uids	Turn unique identifiers	integer

Additional metadata such as the start time and timestep length are also stored as attributes of each group (“link_moe” and “turn_moe”):

num_records
num_timesteps
start_time
timestep

UID Conversion#

Each record in the H5 file is uniquely identified by its UID (link_uid and turn_uid respectively). These UIDs are related back to corresponding unique IDs in the Link and Connection tables in the Supply database by the following identities:

\[T_{id} = T_{uid}\]

$$ L_{id} = floor(L_{uid} / 2) $$

$$ L_{dir} = L_{uid} \mod{2} $$

$$ L_{uid} = 2 . L_{id} + L_{dir} $$

Working with H5 in C++#

Reading and writing into HDF5 files is done with the HighFive API and the H5IO class in MasterType.

// Open a file - there are multiple enum options available for how to open the file
// This will close when out of scope, so just create a local variable for it
HighFive::File file(filename, File::OpenOrCreate);

// Separate functions to create a new matrix/table and append rows to it
MT::H5IO::template Create_Matrix<float>(file, "group_name", "table_name", data_vector, rows, cols);
MT::H5IO::template Append_Matrix<float>(file, "group_name", "table_name", data_vector, row_to_write);

// Write_Matrix will automatically create a new matrix if it does not exist, otherwise writes to the row specified
MT::H5IO::template Write_Matrix<float>(file, "group_name", "table_name", data_vector, row_to_write, rows, cols);

// POLARIS reads every matrix into a 1-dimensional vector rather than actual matrices
// We need to specify the type - be very careful not to mix up 32- and 64-bit types!
std::vector<float> data_array = MT::H5IO::template Read_Matrix_Into_Array<float>(file, "group_name", "table_name");

Working H5 in Python#

HDF5 is supported in Python with the H5Py Library. For most of our uses, after opening a file, its groups and tables can be treated as a dictionary, with the additionals “attrs” value containing another dictionary of attributes.

import h5py

with h5py.File(self.result_hdf5, "r") as h5_result:
    # Get attributes
    timesteps = h5_result["link_moe"].attrs["num_timesteps"]
    records = h5_result["link_moe"].attrs["num_records"]
    
    # Get data
    lengths = h5_result["link_moe"]["link_lengths"]          # This is an array
    travel_times = h5_result["link_moe"]["link_travel_time"] # This is a table

    # Write a new table
    h5_result.create_dataset("new_table", 
        data=some_pandas_dataframe, 
        compression="gzip",             # gzip/deflate compression is our standard
        compression_level=comp_level,   # generally level 3 or 4 is appropriate
        scaleoffset=2, 
        dtype="f4")                     # POLARIS expectes 4-byte floating point (ie. float)

Skims as OMX Outputs#

Highway and transit skim files have been moved from our internal binary format to the industry open standard OMX (OpenMatrix File Format). This was done to introduce compression of the data and to better use open source formats. The introduction of compression reduced the file size of our transit skims (which are quite sparse) by a factor of approximately 20.

OMX is a standardized implementation of HDF5 that stores square tables (i.e. matrices). It inherently knows about zoning system and each table stored in OMX is guaranteed to have the same dimensions (num_zones x num_zones)

Each table is named and given attributes representing the time (as an integer with no decimal values) and the metric (i.e. time, distance, cost, etc) being represented in the matrix. The Transit file tables are also labeled with the transit mode.

OMX files have a lookup functionality which maps our human-friendly Zone identifiers to the indices of our matrix. We use this to store the TAZ mapping for each zone in our model (i.e. column 0 represents zone 1).

Working With OMX in C++#

We use a modified version of the omx-cpp API in POLARIS. According to the license OMX uses, changes need to be made note of, so be sure to do that if anything needs changing.

To use OMX, we need to declare all tables ahead of time:

omxHandler->createFile(number_of_tables, rows, cols, table_name_vector, filename, lookup_table_name);
omxHandler->writeMapping(lookup_table_name, id_container_vector);
omxHandler->closeFile();

Then we can add rows as we go:

OMXMatrix omxHandler;
omxHandler.openFile(filename);
omxHandler.writeRow(table_name, row_number, pointer_to_data);

Working With OMX in Python#

PolarisLib uses the pip installable OMX library “openmatrix” in its skim library (polarislib.skims.highway and polarislib.skims.transit). This library works for both OMX and the deprecated binary format (V3 only) to open files and make the data accessible to Python. Some example code can be found in the notebooks/skim_manipulations.ipynb notebook.

Warning

Package Name There is also an omx pip package which deals with XML parsing. Make sure to use openmatrix.