Comparing Model Ouputs

Comparing Model Ouputs#

In this example we show how to compare outputs across iterations or across models. Here we illustrate using two iterations from the same project, but thanks to the use of standardised KPIs, results can be compared across model years or even across cities.

sphinx_gallery_thumbnail_path = ‘../../examples/result_analysis/pax_in_network.png’

from pathlib import Path

from polaris.analyze.kpi_comparator import KpiComparator
from polaris.analyze.result_kpis import ResultKPIs
from polaris.runs.convergence.convergence_iteration import ConvergenceIteration
from polaris.runs.scenario_compression import ScenarioCompression
from polaris.utils.database.migration_manager import MigrationManager
from polaris.utils.database.standard_database import DatabaseType
from polaris.utils.testing.temp_model import TempModel
project_dir = TempModel("Bloomington")

iteration_3 = ConvergenceIteration.from_dir(project_dir / "Bloomington_iteration_3")
iteration_4 = ConvergenceIteration.from_dir(project_dir / "Bloomington_iteration_4")

# This is generally not needed unless you are analysing results from an older model
MigrationManager.upgrade(
    ScenarioCompression.maybe_extract(iteration_3.files.demand_db), DatabaseType.Demand, redo_triggers=False
)
MigrationManager.upgrade(
    ScenarioCompression.maybe_extract(iteration_4.files.demand_db), DatabaseType.Demand, redo_triggers=False
)

c = KpiComparator()
c.add_run(ResultKPIs.from_iteration(iteration_3), "A label (it3)")
c.add_run(ResultKPIs.from_iteration(iteration_4), "A diff label [it4]")
c.plot_everything()
  • Activity Distributions
  • Activity Start Time distribution
  • Destination Choice Validation
  • Mode Share Validation, Home-Based Work, Home-Based Other, Non Home-Based, Total
  • Timing Choice Validation, WORK, ERRANDS, PICKUP, HEALTHCARE, EAT_OUT, SERVICE, SHOP_MAJOR, RELIGIOUS, PERSONAL, SOCIAL, LEISURE, HOME, SHOP_OTHER, WORK_PART, WORK_HOME, SCHOOL, TOTAL
  • Congestion Pricing Revenue
  • Congestion Removal
  • Runtime and Memory usage
  • Freight In Network
  • Flow-Density Relationship, Freeways, Major Arterials
  • Gaps (styled by gap_type)
  • Mode Share by Activity Type
  • Gaps by Link Type (left) and by Hour (right)
  • Pax In Network
  • Planned Activity Start Time distribution
  • RMSE against calibration targets, Activity Generation, Mode Share, Departure Time
  • Population
  • RMSE against calibration targets, Activity Generation, Mode Share, TTime by Activity, Departure Time
  • Average Skim value by hour
  • Demand, Wait time, IVTT
  • Board/Alightings by Mode & Agency
  • Trip length distribution
  • Trips With vs Without Path
  • In Network
  • Proportion of Connected Vehicles within Fleet
  • plot compare iterations
  • VMT & Speed by Link Type
<class 'tuple'> <class 'tuple'>
<class 'tuple'> <class 'tuple'>
plot_act_dist
plot_activity_start_time_distributions
plot_calibration_destination
plot_calibration_for_activity_generation
plot_calibration_for_activity_generation: failed
plot_calibration_for_boardings
plot_calibration_for_mode_share
plot_calibration_timing
plot_congestion_pricing
plot_congestion_removal
plot_count_validation
plot_count_validation: failed
plot_cpu_mem
plot_freight_in_network
plot_fundamental_diagram
plot_gaps
plot_mode_share
plot_network_gaps
plot_pax_in_network
plot_planned_activity_start_time_distributions
plot_planned_calibration_for_activity_generation_planned
plot_planned_calibration_for_activity_generation_planned: failed
plot_planned_rmse_vs_observed
/home/gitlab-runner/builds/polaris/code/polarislib/polaris/runs/calibrate/mode_choice.py:146: DeprecationWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  return df.groupby("person").apply(set_home_based).reset_index(drop=True)
/home/gitlab-runner/builds/polaris/code/polarislib/polaris/runs/calibrate/mode_choice.py:161: DeprecationWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  return df.groupby("mode").apply(f).reset_index()
/home/gitlab-runner/builds/polaris/code/polarislib/polaris/runs/calibrate/mode_choice.py:146: DeprecationWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  return df.groupby("person").apply(set_home_based).reset_index(drop=True)
/home/gitlab-runner/builds/polaris/code/polarislib/polaris/runs/calibrate/mode_choice.py:161: DeprecationWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
  return df.groupby("mode").apply(f).reset_index()
plot_polaris_exe
<IPython.core.display.HTML object>
<pandas.io.formats.style.Styler object at 0x7760f0b9ad80>
plot_population
plot_population: failed
plot_rmse_vs_observed
plot_skim_stats
plot_tnc
plot_transit
/home/gitlab-runner/builds/polaris/code/polarislib/polaris/analyze/kpi_comparator.py:184: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`). Consider using `matplotlib.pyplot.close()`.
  fig, axes = plt.subplots(2, 1, figsize=(15, 10), sharex=True)
plot_trip_length_distributions
plot_trips_with_path
plot_validation_for_speeds
plot_veh_in_network
plot_vehicle_connectivity
plot_vmt
plot_vmt_by_link_type

Total running time of the script: (0 minutes 43.042 seconds)

Gallery generated by Sphinx-Gallery