Overhaul reference comparison: new error metrics, DataFrames CSV, richer HTML table#28
Draft
AnHeuermann wants to merge 4 commits intomainfrom
Draft
Overhaul reference comparison: new error metrics, DataFrames CSV, richer HTML table#28AnHeuermann wants to merge 4 commits intomainfrom
AnHeuermann wants to merge 4 commits intomainfrom
Conversation
…her HTML table Replace the per-point `_check_point` / `error_fn` dispatch with two vectorised helpers (`_absolute_error`, `_scaled_relative_error`) and a globally-scaled relative tolerance check. `abs_tol` is now optional (`nothing` = disabled, shown as ∞ in the HTML); when set it acts as an independent hard threshold rather than a per-point floor. Switch diff-CSV output from manual text-mode writing to DataFrames.jl + CSV.jl; the new format includes `_abserr` and `_relerr` columns for every failing signal. Improve the `_diff.html` variable-coverage table: - Add "Max Abs Error" (scientific notation) and "Max Rel Error" (percent) columns for both passing and failing signals. - Read per-signal max errors for failing signals directly from the diff CSV. - Hide the abs-tolerance from the meta line when `abs_tol` is `nothing`. Thread `settings::CompareSettings` explicitly through `test_model` and `main`; remove the module-level `_CMP_SETTINGS` global and the `configure_comparison!` / `compare_settings` API. Co-Authored-By: claude-sonnet-4-6 <noreply@anthropic.com>
…tions - Add `cmp_time` to `ModelResult` to track reference comparison wall time; report it in the per-model status line - Read comparison signals in `pipeline.jl` before phase 3 and pass them to both `run_simulate` and `compare_with_reference`, avoiding redundant file reads and enabling observed-variable filtering in the CSV writer - Filter observed variables written to the simulation CSV to only those needed for comparison, keeping large models (e.g. LightningSegmented- TransmissionLine) within the CSV size limit - Write CSV at fixed intervals (from `ode_prob.kwargs[:saveat]` or tspan/500) instead of at every solver step; use dense interpolation for accuracy - Fix `Nothing → Float64` for models where BaseModelica sets `saveat=nothing` by using `something(get(..., nothing), default)` - Accept caller-supplied signal list in `compare_with_reference`; fall back to `comparisonSignals.txt` / all ref-CSV columns when none supplied - Delete versioned results directory before rsync in CI to remove stale files - Use qualified `Dates.now()` / `DifferentialEquations.*` after switching to module-level imports Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Replace the per-point
_check_point/error_fndispatch with two vectorised helpers (_absolute_error,_scaled_relative_error) and a globally-scaled relative tolerance check.abs_tolis now optional (nothing= disabled, shown as ∞ in the HTML); when set it acts as an independent hard threshold rather than a per-point floor.Switch diff-CSV output from manual text-mode writing to DataFrames.jl + CSV.jl; the new format includes
_abserrand_relerrcolumns for every failing signal.Improve the
_diff.htmlvariable-coverage table:abs_tolisnothing.Thread
settings::CompareSettingsexplicitly throughtest_modelandmain; remove the module-level_CMP_SETTINGSglobal and theconfigure_comparison!/compare_settingsAPI.