Skip to content

Testing with DllAPITester

The DllAPITester is a comprehensive testing tool for AlgoMarkers. It contains several useful features:

  • Testing results on a sample set via the infrastructure and via the AlgoMarker
  • Testing via the AlgoMarker library in the infrastructure or via the .so compiled for it.
  • Generating json examples from data.
  • Testing on json examples.

In the following explanations we will assume one has an AlgoMarker with a model inside, a repository to test on, and a samples file to work with.

DllAPITester Help Command

1
DllAPITester --help

Options:

  • --rep: Repository file name
  • --samples: MedSamples file to use
  • --model: Model file to use
  • --amconfig: AlgoMarker configuration file
  • --amlib: The actual .so to use when calling AM_API calls (if not given will use current recent code compiled with DllAPITester)
  • --direct_csv: (optional) The full feature matrix generated by running through the infrastructure
  • --am_csv: (optional) The full feature matrix generated by running through the AlgoMarker
  • --out_jsons: (optional) if given will generate a file with a list of jsons (in a long array) that contain all the data needed to give a prediction. These can be used for direct tests in the AlgoAnalyzer.
  • --in_jsons: (optional) if given will take input data from the given jsons, instead of the repository and samples.
  • --jreq: input json request (could contain also data of in_jsons)
  • --jresp: get output as a json response to store in file
  • --create_jreq: generate a request for a given samples file (will need also jreq_defs, optionally the --add_data_to_jreq flag) and an output file in --jreq_out)

Application Code and Build

The App is located in ../MR/Libs/Internal/AlgoMarker/Linux/Release/DllAPITester. To compile, simply compile the AlgoMarker directory with Boost Library (Internal/AlgoMarker/full_build.sh).

Testing a Batch of examples and comparing infrastructure to AlgoMarker implementation

Example Run
1
./Linux/Release/DllAPITester --rep /home/Repositories/THIN/thin_jun2017/thin.repository --samples /server/Work/Users/Tal/Temp/test.samples --model /server/Products/Pre2D/QA_Versions/1.0.0.1/pre2d.model --amconfig /server/Products/Pre2D/QA_Versions/1.0.0.1/pre2d.amconfig
The tester will compare the scores given by both methods and will return passed/failed. For example:

Example Output
1
2
3
4
5
6
...
#Res1 :: pid 5000529 time 20060308 pred 0.135475 #Res2 pid 5000529 time 20060308 pred 0.135475
#Res1 :: pid 5000529 time 20060315 pred 0.090177 #Res2 pid 5000529 time 20060315 pred 0.090177
Comparing 99 scores
>>>>>TEST1: test DLL API batch: total 99 : n_similar 81 : n_bad 0 : n_miss 18
PASSED

[!NOTE] This only tests for scores and not other outputs, like explainability, or outliers warnings.

A full usage example of creating json inputs and comparing the results with using directly the infrasturure, can be see in this script: MR_Tools/test_algomarker/scripts_template/run_minimal_score_compare.sh

Testing a Single Example Directly

Another option is running a direct test using self-explanatory test data formats:

Testing a single example directly
1
Linux/Release/DllAPITester --rep /home/Repositories/THIN/thin_jun2017/thin.repository --amconfig /nas1/Products/Pre2D/QA_Versions/dev/pre2d.amconfig --direct_test --test_data "Glucose:120:20171101;GENDER:1;BYEAR:1988" --date 20180520
Single test result
1
2
3
4
5
6
7
8
9
Algomarker Pre2D was loaded with config file /nas1/Products/Pre2D/QA_Versions/dev/pre2d.amconfig
Adding Data: sig Glucose :: vals: 120.000000,  times: 20171101,
Adding Data: sig GENDER :: vals: 1.000000,  times:
...
Got 1 responses
Response Messages: 0
Score 0 Messages: 0
resp_rc = 0
i 0 , pid 1 ts 20180520 scr 0.272233

Generating json requests

  1. To create JSON request from repository and samples files, use: 
    1
    DllAPITester --rep ${REP_PATH} --samples ${SAMPLES_PATH} --model  ${MODEL_PATH} --amconfig ${AM_CONFIG} --create_jreq --single --jreq_defs ${PATH_TO_REQUEST_DEFINITION_FILE} --jreq_out ${OUTPUT_JREP_PATH}

Here is jreq_defs example: 

1
2
3
4
5
6
7
{
    "type" : "request" ,
    "request_id" : "my test",
    "export" : {
        "prediction" : "pred_0"
    }
}
This will create JSON inputs for the AlgoMarker from our repository and samples.

  1. Create json data for AddDataByType request (if we haven't included --add_data_to_jreq in above code, so the data is provided in a different API call): 
    1
    DllAPITester --rep ${REP_PATH} --samples ${SAMPLES_PATH} --model  ${MODEL_PATH} --amconfig ${AM_CONFIG} --out_jsons ${EXP_DIR}/data.json --single --calculator "AlgoMarkerName_To_Be_Documented_In_The_Request" --allow_rep_adjustment 1 --signal_categ_regex ${CURR_DIR}/signal_regex_filter.tsv --rename_signal ${CURR_DIR}/signal_rename.tsv
  • You can use the --rename_signal flag to provide a file that maps repository signal names to official AlgoMarker names. This tab-delimited file allows for standardized naming; for example, converting GENDER to SEX.
  • You can use the --signal_categ_regex to keep only categorical values that matched a specific regex, to avoid uneccerary categorical values that aren't supported by the AlgoMarker.
  1. Get Full response from the AlgoMarker (without repository, samples): 
    1
2
    DllAPITester --amlib $LIB_PATH --amconfig $AM_CONFIG --json_req_test --in_jsons ${PATH_TO_DATA_JSON_FROM_STEP2_IF_NOT_INSIDE_JREQ} --jreq ${PATH_TO_REQ_JSON_FROM_STEP1} --jresp ${OUTPUT_PATH} --single --discovery ${AM_DIR}/discovery.json
fi

You can also provide path to get the discovery API json with --discovery