External Silent Run

Overview

The External Silent Run toolkit automates validation and analysis for medical models, especially when working with new, unlabeled datasets. It helps ensure your data is ready for model scoring and provides tools to estimate model performance.

This documentation guides you through setup, running tests, and interpreting results.

Goals

• Make sure data is OK for running the model
• Estimate performances

Structure

• configs/: Configuration files and environment setup
• Tests/: Shell and Python scripts for each validation step

How to Use

Refer to Creating a New TestKit for Your Model.

To run all tests, execute from your TestKit folder:

./run.sh

To run a specific test:

./run.specific.sh

Review results in your configured output directory.

Available Tests and Tools

• Test 01 - Generate Repository
• Test 02 - Fit Model to Repository
• Test 03 - Create Samples
• Test 05 - Compare Repository with Reference Matrix
• Test 06 - Compare Score Distribution
• Test 07 - Calc Score KLD
• Test 08 - Sex Ratio
• Test 09 - Coverage Special Groups
• Test 10 - Compare Important Feature
• Test 11 - Estimate Performances
• Test 12 - Lab Frequency
• Test 13 - test_but_why
• Test 14 - 14.test_model_explainability
• Test 16 - Sample Dates
• Test 17 - Estimate Performance from Calibration
• Test 18 - Analyze Messages
• Test 19 - Search Missing mappings

Configuration

Set required parameters in env.sh. If a parameter is missing for a test, that test will be skipped.

Data Input Parameters for Scoring

If your data is stored in an AWS S3 bucket, set the following credentials:

• AWS_REGION
• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_INPUT_PATH: Path to input dataset ("file_api" format)

• AWS_OUTPUT_PATH: Path to output scores generated by AlgoAnalyzer

  • Example input format:

  ID	Date	Signal	Value	Unit
  1	20251005	Hemoglobin	14.1	mg/dL

If not using AWS S3, set these parameters directly:

• SILENCE_RUN_INPUT_FILES_PATH: Path to input dataset ("file_api" format)
• SILENCE_RUN_OUTPUT_FILES_PATH: Path to AlgoAnalyzer output scores
  • If not available, set to GENERATE to create sample files based on last Hemoglobin tests per patient
  • Use TAKE_JUST_LAST to select only the last Hemoglobin date per patient, or 0 to analyze all dates

When using AWS parameters, you do not need to set SILENCE_RUN_INPUT_FILES_PATH or SILENCE_RUN_OUTPUT_FILES_PATH. These files will be downloaded to $WORK_DIR/data for inspection.

Model Parameters

• ALGOMARKER_PATH: Path to model
• REFERENCE_MATRIX: Full path to reference matrix for comparison
• CMP_FEATURE_RES: Comma-separated list of important features and their resolutions (e.g., Age:1,MCH.slope.win_0_1000:0.01)
  • Feature_Name: Unique string identifying the feature
  • Resolution: Used for plotting value distributions
• SCORE_MIN_RANGE: Minimum score range for sex ratio test
• SCORE_MAX_RANGE: Maximum score range for sex ratio test
• FILTER_LAST_DATE: Set to 0 to analyze all dates, or use to filter duplicate runs. If you don't know, keep 0

Output Parameters

• WORK_DIR: Output directory for results