Skip to content

Cleaners Json Examples

Examples for creating Jsons with default cleaner. An Example of cleaner can be found in git: full_rep_processors.json   full_cleaners is based on:

  • configured simple cleaner with strict boundaries for each signal 
  • sim_val - when signal appears with the same time and different value which value to take? (from my observations in THIN - taking the first value is better)
  • panels calculation and remove mismatches of biological rules. For example wrong calculation of BMI Notes:
  • There is a problem with Cholesterol_over_HDL signal in the loading process,  so using full_cleaner is not recommended for now with the rule of the Cholesterol_over_HDL activated (rules 15,17)
  • You can now use Flow with "pids_sigs_print" mode or Yaron print program to print pids and signals after rep_processors. It may be useful for cleaners, virtual signals and more. I had created a different print (not in Flow) that shows the difference between 2 run modes of rep_processings (or compare run with rep processing to no rep processing at all) and prints the removed rows with "[REMOVED]" in each removed row to see what happened. for more information contact me.   
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
    {
  "model_json_version": "2",
  "serialize_learning_set": "0",
  "model_actions": [
    {
      "action_type": "rp_set",
      "members": [
        {
          "rp_type":"conf_cln",
          "conf_file":"../settings/cleanDictionary.csv",
          "time_channel":"0",
          "clean_method":"confirmed",
          "signal":"file:../settings/all_rules_sigs.list"
          //,"verbose_file":"/tmp/cleaning.log"
        },
        {
          "rp_type":"conf_cln",
          "conf_file":"../settings/cleanDictionary.csv",
          "val_channel":["0", "1"],
          "clean_method":"confirmed",
          "signal": ["BP"]
          //,"verbose_file":"/tmp/cleaning.log"
        }
      ]
    },
    {
      "action_type": "rp_set",
      "members": [
        {
          "rp_type":"sim_val",
          "signal":"file:../settings/all_rules_sigs.list",
          "type":"first",
          "debug":"0"

        }

      ]
    },
    {
      "action_type": "rp_set",
      "members": [
        {
          "rp_type":"rule_cln",
          "addRequiredSignals":"1",
          "time_window":"0",
          "tolerance":"0.1",
          "calc_res":"0.1",
          "rules2Signals":"../settings/ruls2Signals.tsv",
          "consideredRules":[ "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "12", "13", "14", "15", "16", "17", "18", "19", "20", "21", "22" ] 
          //,"verbose_file":"/tmp/panel_cleaning.log"
        }
      ]
    }

  ]
}
      This was tested with Cleaner Program that checks for filtered stats with examples. here are the filtered stats on THIN: Stats of simple cleaner To create this table with full examples: 
    1
    Flow --rep /home/Repositories/THIN/thin_jun2017/thin.repository --rep_processor_print --sigs /server/Work/Users/Alon/UnitTesting/examples/general_config_files/Cleaner/all_rules_sigs.list --max_examples 10 --seed 0 --f_output /tmp/test.log --cleaner_path /server/Work/Users/Alon/UnitTesting/examples/general_config_files/Cleaner/only_configure.json
      Simple Filtering - only Configure Rules for stricted bounderies
Signal TOTAL_CNT TOTAL_CNT_NON_ZERO TOTAL_CLEANED CLEAN_PERCENTAGE CLEAN_NON_ZERO_PERCENTAGE TOTAL_PIDS PIDS_FILTERED PIDS_FILTERED_NON_ZEROS PIDS_FILTER_PERCENTAGE PIDS_FILTER_NON_ZERO_PERCENTAGE comment
eGFR_MDRD 30992945 30976157 30258567 2.37% 2.32% 5328855 427719 414278 8.03% 7.77% Remove filter - not needed
UrineCreatinine 2603228 2602550 2556325 1.80% 1.78% 694781 17213 17210 2.48% 2.48% Maybe problem in units that can be solved - checking
PlasmaViscosity 1258162 1257236 1237837 1.62% 1.54% 459295 8899 8147 1.94% 1.77% Maybe problem in units that can be solved - checking
Transferrin 386039 386034 382521 0.91% 0.91% 232202 2709 2704 1.17% 1.16% OK
HDL_over_nonHDL 14593886 14580124 14459042 0.92% 0.83% 3414705 77777 77335 2.28% 2.26% OK
eGFR_CKD_EPI 30992945 30992374 30763090 0.74% 0.74% 5328855 171719 171715 3.22% 3.22% Remove filter - not needed
Height 18860856 18780829 18714169 0.78% 0.35% 9334026 124126 57892 1.33% 0.62% Maybe problem in units that can be solved - checking
CA125 252667 252550 251745 0.36% 0.32% 193979 599 598 0.31% 0.31% OK
BP 90295429 89497237 89264282 1.14% 0.26% 9410580 525587 178924 5.59% 1.90% bugfix to work on each channel
MCHC-M 25816227 25786164 25719334 0.38% 0.26% 5459883 56481 38450 1.03% 0.70% OK
BMI 35211327 35194623 35118884 0.26% 0.22% 8293631 73201 59596 0.88% 0.72% OK
Phosphore 4571174 4570766 4561223 0.22% 0.21% 1866755 3057 2683 0.16% 0.14%  
Lymphocytes% 24656284 24652334 24610137 0.19% 0.17% 5312408 26240 26119 0.49% 0.49%  
Neutrophils% 24740548 24736537 24695583 0.18% 0.17% 5321777 25181 25046 0.47% 0.47%  
INR 8505951 8496622 8488159 0.21% 0.10% 455102 5643 4012 1.24% 0.88%  
PDW 384442 383910 383555 0.23% 0.09% 109155 772 772 0.71% 0.71%  
PlasmaAnionGap 21530 21530 21513 0.08% 0.08% 5491 15 15 0.27% 0.27%  
WBC 26610249 26593657 26572901 0.14% 0.08% 5554159 25789 12033 0.46% 0.22%  
FreeT3 506119 505466 505091 0.20% 0.07% 205428 877 314 0.43% 0.15%  
RBC 25905761 25876600 25857697 0.19% 0.07% 5471987 28169 27732 0.51% 0.51%  
Ca 7173055 7172113 7167232 0.08% 0.07% 2582464 4826 3942 0.19% 0.15%  
Mg 182248 182218 182094 0.08% 0.07% 107908 134 108 0.12% 0.10%  
T4 475830 473666 473368 0.52% 0.06% 213799 2235 277 1.05% 0.13%  
Hematocrit 25862482 25836512 25822132 0.16% 0.06% 5463443 27793 10406 0.51% 0.19%  
TEMP 1801163 1786361 1785441 0.87% 0.05% 964098 12126 917 1.26% 0.10%  
Digoxin 93861 93617 93584 0.30% 0.04% 43469 215 212 0.49% 0.49%  
SerumAnionGap 59666 59661 59640 0.04% 0.04% 22779 24 24 0.11% 0.11%  
K+ 28662323 28620055 28610424 0.18% 0.03% 5174781 35073 6931 0.68% 0.13%  
MPV 3443266 3442461 3441344 0.06% 0.03% 980181 1361 978 0.14% 0.10%  
Bicarbonate 3175248 3174729 3173852 0.04% 0.03% 741895 1332 844 0.18% 0.11%  
Cholesterol 18909856 18888280 18883203 0.14% 0.03% 3882355 22236 4367 0.57% 0.11%  
Albumin 23700344 23664696 23658553 0.18% 0.03% 4850137 25065 4754 0.52% 0.10%  
Na 29000033 28945701 28940315 0.21% 0.02% 5195665 26686 5191 0.51% 0.10%  
Iron_Fe 613710 613514 613409 0.05% 0.02% 363871 284 102 0.08% 0.03%  
Weight 39402271 39199410 39192705 0.53% 0.02% 9808932 167114 6298 1.70% 0.06%  
NonHDLCholesterol 14588087 14587707 14585866 0.02% 0.01% 3413836 1912 1909 0.06% 0.06%  
RandomGlucose 710956 701691 701603 1.32% 0.01% 423583 6726 87 1.59% 0.02%  
Platelets_Hematocrit 3400998 3399872 3399528 0.04% 0.01% 972680 1193 1191 0.12% 0.12%  
Hemoglobin 27748929 27701627 27698933 0.18% 0.01% 5689777 24331 2523 0.43% 0.04%  
LDL 12668050 12639848 12638813 0.23% 0.01% 3124704 20626 965 0.66% 0.03%  
MCV 26246642 26230251 26228319 0.07% 0.01% 5510085 12313 1833 0.22% 0.03%  
CO2 262289 261884 261866 0.16% 0.01% 44949 216 18 0.48% 0.04%  
Glucose 16484078 16472703 16471686 0.08% 0.01% 4466133 9937 928 0.22% 0.02%  
Amylase 355012 354296 354275 0.21% 0.01% 275843 685 20 0.25% 0.01%  
CorrectedCa 6448360 6447552 6447174 0.02% 0.01% 2381369 1087 357 0.05% 0.01%  
Triglycerides 14035783 14004268 14003515 0.23% 0.01% 3295899 19785 707 0.60% 0.02%  
LDH 223522 218968 218958 2.04% 0.00% 102148 3316 10 3.25% 0.01%  
ALKP 24501543 24491562 24490462 0.05% 0.00% 4975550 8773 844 0.18% 0.02%  
PULSE 5607620 5597890 5597659 0.18% 0.00% 2221767 7933 224 0.36% 0.01%  
Cl 6074637 6068936 6068789 0.10% 0.00% 1205272 4050 146 0.34% 0.01%  
Protein_Total 15053134 15051906 15051694 0.01% 0.00% 3350737 1336 188 0.04% 0.01%  
FreeT4 8375885 8347195 8347096 0.34% 0.00% 2572146 19968 99 0.78% 0.00%  
AST 5017954 5016921 5016887 0.02% 0.00% 1391446 972 33 0.07% 0.00%  
Platelets 26572350 26551880 26551731 0.08% 0.00% 5546115 15154 133 0.27% 0.00%  
ALT 20504083 20485612 20485528 0.09% 0.00% 4431602 12872 82 0.29% 0.00%  
MCH 25858658 25846546 25846464 0.05% 0.00% 5463374 7628 82 0.14% 0.00%  
B12 3015306 3013860 3013860 0.05% 0% 1607426 1212 0 0.08% 0%  
RDW 4969565 4969446 4969563 4.02E-07 0.00% 1579652 2 2 0.00% 0.00%  
Urea 22375296 22373145 22373971 0.01% 0.00% 4367623 1291 1289 0.03% 0.03%  
Monocytes% 24411657 24387478 24388588 0.09% 0.00% 5276761 13670 13512 0.26% 0.26%  
VitaminD_25 352787 352738 352775 0.00% -0.01% 228066 12 12 0.01% 0.01%  
Fibrinogen 196551 196504 196550 0.00% -0.02% 143822 1 1 0.00% 0.00%  
HDL_over_LDL 12596626 12590998 12594958 0.01% -0.03% 3118377 1509 1507 0.05% 0.05%  
Urine_Dipstick_pH 136097 136047 136091 0.00% -0.03% 64581 6 6 0.01% 0.01%  
Transferrin_Saturation_Index 252126 251899 251984 0.06% -0.03% 162901 134 134 0.08% 0.08%  
Ferritin 3729644 3728208 3729625 0.00% -0.04% 1809350 18 18 0.00% 0.00%  
Bilirubin 23598709 23588476 23598656 0.00% -0.04% 4904303 50 50 0.00% 0.00%  
Sex_Hormone_Binding_Globulin 133505 133441 133503 0.00% -0.05% 110127 2 2 0.00% 0.00%  
Lymphocytes# 24916931 24903489 24916729 0.00% -0.05% 5337938 159 159 0.00% 0.00%  
HbA1C 7597400 7592635 7597179 0.00% -0.06% 1520278 163 160 0.01% 0.01%  
TIBC 151135 151012 151114 0.01% -0.07% 97981 21 21 0.02% 0.02%  
Prolactin 407790 407328 407663 0.03% -0.08% 297447 91 91 0.03% 0.03%  
Cholesterol_over_HDL 15035230 15018683 15031577 0.02% -0.09% 3421173 3150 2923 0.09% 0.09%  
Follic_Acid 2703458 2701120 2703458 0% -0.09% 1472362 0 0 0% 0%  
HDL_over_Cholesterol 14670753 14656530 14669390 0.01% -0.09% 3420769 1342 1331 0.04% 0.04%  
Neutrophils# 24970648 24946553 24970234 0.00% -0.09% 5346509 383 378 0.01% 0.01%  
FSH 1045438 1044297 1045438 0% -0.11% 688285 0 0 0% 0%  
Testosterone 388036 387595 388027 0.00% -0.11% 292007 9 9 0.00% 0.00%  
LDL_over_HDL 12606025 12591423 12605655 0.00% -0.11% 3118883 357 349 0.01% 0.01%  
Globulin 8009111 7999593 8008795 0.00% -0.12% 1896822 254 253 0.01% 0.01%  
Creatinine 31070691 31033058 31069711 0.00% -0.12% 5331326 753 732 0.01% 0.01%  
eGFR 16442849 16422354 16442849 0% -0.12% 3426403 0 0 0% 0%  
Monocytes# 24612290 24579520 24612015 0.00% -0.13% 5298350 234 219 0.00% 0.00%  
CK 942771 938846 940828 0.21% -0.21% 481006 1367 1364 0.28% 0.28%  
Uric_Acid 1151516 1147621 1150405 0.10% -0.24% 655507 1073 1068 0.16% 0.16%  
Erythrocyte 7239836 7222198 7239830 8.29E-07 -0.24% 2647537 6 5 0.00% 0.00%  
Lithium 274155 273378 274113 0.02% -0.27% 21344 26 26 0.12% 0.12%  
GGT 9354116 9324434 9354066 0.00% -0.32% 2369564 49 49 0.00% 0.00%  
UrineAlbumin 1366696 1361871 1366696 0% -0.35% 391018 0 0 0% 0%  
GFR 3225746 3213478 3225746 0% -0.38% 882940 0 0 0% 0%  
LUC 1330513 1324624 1330144 0.03% -0.42% 324604 250 215 0.08% 0.07%  
HDL 14818204 14749529 14813285 0.03% -0.43% 3431910 3956 2632 0.12% 0.08%  
UrineTotalProtein 386928 384724 386787 0.04% -0.54% 179141 115 115 0.06% 0.06%  
Reticulocyte 69816 69418 69814 0.00% -0.57% 49484 2 2 0.00% 0.00%  
UrineAlbumin_over_Creatinine 2451211 2431361 2451204 0.00% -0.82% 678233 7 7 0.00% 0.00%  
PSA 1926651 1906549 1926420 0.01% -1.04% 660296 159 157 0.02% 0.02%  
Serum_Oestradiol 394083 389241 394083 0% -1.24% 274758 0 0 0% 0%  
PFR 5264955 5164019 5264358 0.01% -1.94% 1355995 569 535 0.04% 0.04%  
TSH 16173524 15848163 16172909 0.00% -2.05% 4501457 528 450 0.01% 0.01%  
LuteinisingHormone 847445 830210 847445 0% -2.08% 585366 0 0 0% 0%  
Rheumatoid_Factor 461205 450087 460648 0.12% -2.35% 364740 494 492 0.14% 0.13%  
Progesterone 233686 227831 233686 0% -2.57% 156019 0 0 0% 0%  
CRP 5980408 5820488 5980374 0.00% -2.75% 2320376 30 30 0.00% 0.00%  
Urine_Protein_Creatinine 324747 314833 324741 0.00% -3.15% 151145 6 6 0.00% 0.00%  
Urine_Epithelial_Cell 299537 287827 299537 0% -4.07% 145624 0 0 0% 0%  
Eosinophils% 24273861 22801564 24270972 0.01% -6.44% 5260185 2185 1951 0.04% 0.04%  
Eosinophils# 24445262 22942339 24445176 0.00% -6.55% 5281030 70 59 0.00% 0.00%  
Urine_Microalbumin 1386074 1269197 1386072 0.00% -9.21% 375904 2 0 0.00% 0%  
Basophils# 23695244 14449541 23692629 0.01% -63.97% 5194395 2007 1665 0.04% 0.03%  
Basophils% 23660542 9994176 23648885 0.05% -136.63% 5193897 9473 3998 0.18% 0.08%  
NRBC 2113870 3939 2113869 4.73E-07 -53565.10% 733525 1 0 0.00% 0%  

Examples of filtered row log from program: EXAMPLE pid     13055975        Signal  UrineCreatinine Time    20090210        Value   12225   [REMOVED] EXAMPLE pid     13055975        Signal  UrineCreatinine Time    20100204        Value   14970   [REMOVED] EXAMPLE pid     13055975        Signal  UrineCreatinine Time    20110304        Value   15219   [REMOVED] EXAMPLE pid     13055975        Signal  UrineCreatinine Time    20120319        Value   15607   [REMOVED] EXAMPLE pid     13055975        Signal  UrineCreatinine Time    20130221        Value   14.8 EXAMPLE pid     13055975        Signal  UrineCreatinine Time    20131115        Value   13 EXAMPLE pid     13055975        Signal  UrineCreatinine Time    20141124        Value   9 EXAMPLE pid     13055975        Signal  UrineCreatinine Time    20160122        Value   6.1 EXAMPLE pid     11134171        Signal  UrineCreatinine Time    20091215        Value   6694    [REMOVED] EXAMPLE pid     11134171        Signal  UrineCreatinine Time    20111115        Value   6.4 EXAMPLE pid     6835336 Signal  UrineCreatinine Time    20080424        Value   3686    [REMOVED] EXAMPLE pid     6835336 Signal  UrineCreatinine Time    20090918        Value   4630    [REMOVED] EXAMPLE pid     6835336 Signal  UrineCreatinine Time    20100611        Value   13229   [REMOVED] EXAMPLE pid     6835336 Signal  UrineCreatinine Time    20110106        Value   13349   [REMOVED] EXAMPLE pid     6835336 Signal  UrineCreatinine Time    20110404        Value   3275    [REMOVED] EXAMPLE pid     6835336 Signal  UrineCreatinine Time    20111202        Value   5063    [REMOVED] EXAMPLE pid     6835336 Signal  UrineCreatinine Time    20120315        Value   3616    [REMOVED] EXAMPLE pid     6835336 Signal  UrineCreatinine Time    20120628        Value   5262    [REMOVED] EXAMPLE pid     6835336 Signal  UrineCreatinine Time    20130306        Value   2.9 EXAMPLE pid     6835336 Signal  UrineCreatinine Time    20140221        Value   4.3 EXAMPLE pid     6835336 Signal  UrineCreatinine Time    20150527        Value   4 EXAMPLE pid     6835336 Signal  UrineCreatinine Time    20160822        Value   3.6 EXAMPLE pid     8685466 Signal  UrineCreatinine Time    20091113        Value   11823   [REMOVED] EXAMPLE pid     14408958        Signal  UrineCreatinine Time    20081124        Value   5452    [REMOVED] EXAMPLE pid     14408958        Signal  UrineCreatinine Time    20100125        Value   6526    [REMOVED] EXAMPLE pid     14408958        Signal  UrineCreatinine Time    20100624        Value   4616    [REMOVED] EXAMPLE pid     14408958        Signal  UrineCreatinine Time    20110715        Value   3074    [REMOVED] EXAMPLE pid     14408958        Signal  UrineCreatinine Time    20120615        Value   3674    [REMOVED] EXAMPLE pid     14408958        Signal  UrineCreatinine Time    20130611        Value   4.2 EXAMPLE pid     14408958        Signal  UrineCreatinine Time    20140607        Value   3.1 EXAMPLE pid     14408958        Signal  UrineCreatinine Time    20140722        Value   3 STATS   UrineCreatinine TOTAL_CNT       2603228 TOTAL_CNT_NON_ZERO      2602550 TOTAL_CLEANED   2556325 CLEAN_PERCENTAGE        1.80172%        CLEAN_NON_ZERO_PERCENTAGE       1.77614%        TOTAL_PIDS    694781   PIDS_FILTERED   17213   PIDS_FILTERED_NON_ZEROS 17210   PIDS_FILTER_PERCENTAGE  2.47747%        PIDS_FILTER_PERCENTAGE  2.47704%  1. Height- looks like there is factor 100 sometimes EXAMPLE pid     17008937        Signal  Height  Time    20041111        Value   165 EXAMPLE pid     17008937        Signal  Height  Time    20050302        Value   16500   [REMOVED] EXAMPLE pid     17008937        Signal  Height  Time    20060522        Value   16500   [REMOVED] EXAMPLE pid     17008937        Signal  Height  Time    20060607        Value   165 EXAMPLE pid     17008937        Signal  Height  Time    20060607        Value   16500   [REMOVED] EXAMPLE pid     17008937        Signal  Height  Time    20071109        Value   165 EXAMPLE pid     17008937        Signal  Height  Time    20090617        Value   165 EXAMPLE pid     5044235 Signal  Height  Time    19980408        Value   17      [REMOVED] EXAMPLE pid     5044235 Signal  Height  Time    20040209        Value   173 EXAMPLE pid     5044235 Signal  Height  Time    20061212        Value   173 EXAMPLE pid     5044235 Signal  Height  Time    20091203        Value   173 EXAMPLE pid     5044235 Signal  Height  Time    20111122        Value   173 EXAMPLE pid     5044235 Signal  Height  Time    20121113        Value   173 EXAMPLE pid     11310017        Signal  Height  Time    19930902        Value   150 EXAMPLE pid     11310017        Signal  Height  Time    19990422        Value   12      [REMOVED] EXAMPLE pid     5188073 Signal  Height  Time    20031118        Value   165 EXAMPLE pid     5188073 Signal  Height  Time    20031118        Value   16500   [REMOVED] EXAMPLE pid     5188073 Signal  Height  Time    20040809        Value   158 EXAMPLE pid     5188073 Signal  Height  Time    20041120        Value   165 EXAMPLE pid     5188073 Signal  Height  Time    20060901        Value   158 EXAMPLE pid     5188073 Signal  Height  Time    20071010        Value   159 EXAMPLE pid     10759614        Signal  Height  Time    19930602        Value   168 EXAMPLE pid     10759614        Signal  Height  Time    19980225        Value   2       [REMOVED] STATS   Height  TOTAL_CNT       18860856        TOTAL_CNT_NON_ZERO      18780829        TOTAL_CLEANED   18714169        CLEAN_PERCENTAGE        0.777732%       CLEAN_NON_ZERO_PERCENTAGE       0.354936%     TOTAL_PIDS       9334026 PIDS_FILTERED   124126  PIDS_FILTERED_NON_ZEROS 57892   PIDS_FILTER_PERCENTAGE  1.32982%        PIDS_FILTER_PERCENTAGE  0.620225% 1. MCHC-M – looks like factor 10 problem: EXAMPLE pid     17224844        Signal  MCHC-M  Time    20010622        Value   357     [*] EXAMPLE pid     17224844        Signal  MCHC-M  Time    20060711        Value   34.4 EXAMPLE pid     17224844        Signal  MCHC-M  Time    20090505        Value   32 EXAMPLE pid     16311924        Signal  MCHC-M  Time    19950502        Value   330     [] EXAMPLE pid     16311924        Signal  MCHC-M  Time    19950605        Value   326     [] EXAMPLE pid     16311924        Signal  MCHC-M  Time    20021114        Value   33.1 EXAMPLE pid     16311924        Signal  MCHC-M  Time    20050328        Value   32.9 EXAMPLE pid     16311924        Signal  MCHC-M  Time    20070111        Value   32.5 EXAMPLE pid     16311924        Signal  MCHC-M  Time    20070201        Value   33.4 EXAMPLE pid     16311924        Signal  MCHC-M  Time    20071030        Value   33.4 EXAMPLE pid     16311924        Signal  MCHC-M  Time    20071112        Value   33.1 EXAMPLE pid     16311924        Signal  MCHC-M  Time    20080422        Value   31.7 EXAMPLE pid     16311924        Signal  MCHC-M  Time    20081030        Value   34.8 EXAMPLE pid     16311924        Signal  MCHC-M  Time    20090611        Value   31.1 EXAMPLE pid     16311924        Signal  MCHC-M  Time    20090715        Value   33.5 EXAMPLE pid     16311924        Signal  MCHC-M  Time    20091230        Value   31.6 EXAMPLE pid     16311924        Signal  MCHC-M  Time    20100106        Value   30 EXAMPLE pid     16311924        Signal  MCHC-M  Time    20100126        Value   32.1 EXAMPLE pid     16311924        Signal  MCHC-M  Time    20100204        Value   31.6 EXAMPLE pid     16311924        Signal  MCHC-M  Time    20100225        Value   31.2 EXAMPLE pid     16311924        Signal  MCHC-M  Time    20100301        Value   31.7 EXAMPLE pid     16311924        Signal  MCHC-M  Time    20100308        Value   31.5 EXAMPLE pid     16311924        Signal  MCHC-M  Time    20100315        Value   29.5 EXAMPLE pid     15517308        Signal  MCHC-M  Time    20040511        Value   313     [] EXAMPLE pid     15517308        Signal  MCHC-M  Time    20071022        Value   32.7 EXAMPLE pid     17224984        Signal  MCHC-M  Time    19991116        Value   326     [] EXAMPLE pid     17224984        Signal  MCHC-M  Time    20050808        Value   34.1 EXAMPLE pid     17224984        Signal  MCHC-M  Time    20051026        Value   34.4 EXAMPLE pid     17224984        Signal  MCHC-M  Time    20051222        Value   34.8 EXAMPLE pid     17141004        Signal  MCHC-M  Time    20040728        Value   33.9 EXAMPLE pid     17141004        Signal  MCHC-M  Time    20090522        Value   396     [] EXAMPLE pid     17141004        Signal  MCHC-M  Time    20090623        Value   35 EXAMPLE pid     17141004        Signal  MCHC-M  Time    20090918        Value   35.3 EXAMPLE pid     17141004        Signal  MCHC-M  Time    20091123        Value   34.9 EXAMPLE pid     17141004        Signal  MCHC-M  Time    20100319        Value   370     [***] STATS   MCHC-M  TOTAL_CNT       25816227        TOTAL_CNT_NON_ZERO      25786164        TOTAL_CLEANED   25719334        CLEAN_PERCENTAGE        0.375318%       CLEAN_NON_ZERO_PERCENTAGE       0.25917%      TOTAL_PIDS       5459883 PIDS_FILTERED   56481   PIDS_FILTERED_NON_ZEROS 38450   PIDS_FILTER_PERCENTAGE  1.03447%        PIDS_FILTER_PERCENTAGE  0.704228%   Stats of Full (with panels check) 

1
Flow --rep /home/Repositories/THIN/thin_jun2017/thin.repository --rep_processor_print --sigs "Cholesterol_over_HDL,HDL_over_Cholesterol,UrineAlbumin,HDL,UrineAlbumin_over_Creatinine,UrineCreatinine,LDL,Cholesterol,LDL_over_HDL,HDL_over_LDL,Platelets_Hematocrit,MPV,NonHDLCholesterol,HDL_over_nonHDL,MCV,RBC,Hematocrit,Platelets,MCH,MCHC-M,Protein_Total,Hemoglobin,Height,Albumin,Basophils#,Eosinophils#,Monocytes#,Lymphocytes#,Neutrophils#,NRBC,WBC,BMI,Weight,UrineTotalProtein,ALKP,ALT,Amylase,AST,B12,Basophils%,Bicarbonate,Bilirubin,Ca,CA125,CK,Cl,CO2,CorrectedCa,Creatinine,CRP,Digoxin,eGFR,Eosinophils%,Erythrocyte,Ferritin,Fibrinogen,Follic_Acid,FreeT3,FreeT4,FSH,GFR,GGT,Globulin,Glucose,HbA1C,INR,Iron_Fe,K+,LDH,Lithium,LUC,LuteinisingHormone,Lymphocytes%,Mg,Monocytes%,Na,Neutrophils%,PDW,PFR,Phosphore,PlasmaAnionGap,PlasmaViscosity,Progesterone,Prolactin,PSA,PULSE,RandomGlucose,RDW,Reticulocyte,Rheumatoid_Factor,Serum_Oestradiol,SerumAnionGap,Sex_Hormone_Binding_Globulin,T4,Testosterone,TIBC,Transferrin,Transferrin_Saturation_Index,Triglycerides,TSH,Urea,Uric_Acid,Urine_Dipstick_pH,Urine_Epithelial_Cell,Urine_Microalbumin,Urine_Protein_Creatinine,VitaminD_25,TEMP" --max_examples 10 --seed 0 --f_output /tmp/test.log --cleaner_path_before /server/Work/Users/Alon/UnitTesting/examples/general_config_files/Cleaner/configure_sim_val.json --cleaner_path /server/Work/Users/Alon/UnitTesting/examples/general_config_files/Cleaner/full_cleaners.json

Signal TOTAL_CNT TOTAL_CNT_NON_ZERO TOTAL_CLEANED CLEAN_PERCENTAGE TOTAL_PIDS PIDS_FILTERED PIDS_FILTERED_NON_ZEROS PIDS_FILTER_PERCENTAGE PIDS_FILTER_NON_ZERO_PERCENTAGE Comment
UrineAlbumin 1364175 1359350 1301695 4.58% 391018 33583 33206 8.59% 8.49%  Urine_panel, lot of real errors
UrineAlbumin_over_Creatinine 2444786 2424936 2382427 2.55% 678233 33467 30981 4.93% 4.57%  Urine_panel, lot of real errors
UrineCreatinine 2550757 2550079 2488398 2.44% 694781 33467 33438 4.82% 4.81%  Urine_panel, lot of real errors
Cholesterol_over_HDL 14653976 14653976 14375111 1.90% 3421173 115517 115517 3.38% 3.38% there is problem in load of wrong source
HDL_over_LDL 12594284 12588656 12368827 1.79% 3118377 129667 128275 4.16% 4.11%  
Platelets_Hematocrit 3398224 3398224 3337846 1.78% 972680 19763 19763 2.03% 2.03%  
MPV 3438309 3438309 3377931 1.76% 980181 19763 19763 2.02% 2.02%  
HDL 14751464 14682789 14546126 1.39% 3431910 114884 98162 3.35% 2.86%  
HDL_over_Cholesterol 14665993 14651770 14467583 1.35% 3420769 86917 86067 2.54% 2.52% there is problem in load of wrong source
Cholesterol 18855175 18855175 18649837 1.09% 3882355 114884 114884 2.96% 2.96%  
NonHDLCholesterol 14585464 14585084 14428102 1.08% 3413836 83548 83450 2.45% 2.44%  
LDL_over_HDL 12605642 12591040 12550215 0.44% 3118883 27352 26925 0.88% 0.86%  
LDL 12633401 12633401 12577976 0.44% 3124704 37293 37293 1.19% 1.19%  
MCV 26166367 26166367 26084584 0.31% 5510085 44007 44007 0.80% 0.80%  
HDL_over_nonHDL 14459035 14445273 14414026 0.31% 3414705 19387 18580 0.57% 0.54%  
RBC 25832962 25803801 25758846 0.29% 5471987 38602 34538 0.71% 0.63%  
Hematocrit 25784644 25784644 25725376 0.23% 5463443 29769 29769 0.54% 0.54%  
Platelets 26511344 26511344 26450966 0.23% 5546115 19763 19763 0.36% 0.36%  
MCH 25791202 25791202 25735104 0.22% 5463374 36564 36564 0.67% 0.67%  
MCHC-M 25697538 25697538 25655308 0.16% 5459883 25802 25802 0.47% 0.47%  
Protein_Total 15030698 15030698 15014686 0.11% 3350737 10857 10857 0.32% 0.32%  
Hemoglobin 27672196 27672196 27648170 0.09% 5689777 16426 16426 0.29% 0.29%  
Albumin 23630785 23630785 23614773 0.07% 4850137 10857 10857 0.22% 0.22%  
Basophils# 23647655 14401952 23636707 0.05% 5194395 9378 3885 0.18% 0.07%  
Eosinophils# 24411167 22908244 24400219 0.04% 5281030 9378 7428 0.18% 0.14%  
Monocytes# 24561221 24528451 24550273 0.04% 5298350 9378 9178 0.18% 0.17%  
Lymphocytes# 24824132 24810690 24813184 0.04% 5337938 9378 9326 0.18% 0.17%  
Neutrophils# 24907730 24883635 24896782 0.04% 5346509 9378 9272 0.18% 0.17%  
NRBC 2113731 3800 2112819 0.04% 733525 646 503 0.09% 0.07%  
WBC 26543214 26543214 26532266 0.04% 5554159 9378 9378 0.17% 0.17%  
BMI 35030317 35030317 35017055 0.04% 8293631 12869 12869 0.16% 0.16%  
UrineTotalProtein 385952 383748 385824 0.03% 179141 125 91 0.07% 0.05%  
Height 18664187 18664187 18664187 0% 9334026 0 0 0% 0%  
Weight 39068717 39068717 39068717 0% 9808932 0 0 0% 0%  
ALKP 24458720 24458720 24458720 0% 4975550 0 0 0% 0%  
ALT 20471720 20471720 20471720 0% 4431602 0 0 0% 0%  
Amylase 353637 353637 353637 0% 275843 0 0 0% 0%  
AST 5013392 5013392 5013392 0% 1391446 0 0 0% 0%  
B12 3011237 3011237 3011237 0% 1607426 0 0 0% 0%  
Basophils% 23647760 9981394 23647760 0% 5193897 0 0 0% 0%  
Bicarbonate 3171108 3171108 3171108 0% 741895 0 0 0% 0%  
Bilirubin 23542527 23532294 23542527 0% 4904303 0 0 0% 0%  
Ca 7095160 7095160 7095160 0% 2582464 0 0 0% 0%  
CA125 252115 251998 252115 0% 193979 0 0 0% 0%  
CK 939752 935827 939752 0% 481006 0 0 0% 0%  
Cl 6064395 6064395 6064395 0% 1205272 0 0 0% 0%  
CO2 261631 261631 261631 0% 44949 0 0 0% 0%  
CorrectedCa 6440715 6440715 6440715 0% 2381369 0 0 0% 0%  
Creatinine 31019168 30981535 31019168 0% 5331326 0 0 0% 0%  
CRP 5976368 5816448 5976368 0% 2320376 0 0 0% 0%  
Digoxin 92754 92510 92754 0% 43469 0 0 0% 0%  
eGFR 16433805 16413310 16433805 0% 3426403 0 0 0% 0%  
Eosinophils% 24269187 22796890 24269187 0% 5260185 0 0 0% 0%  
Erythrocyte 7233824 7216186 7233824 0% 2647537 0 0 0% 0%  
Ferritin 3722680 3721244 3722680 0% 1809350 0 0 0% 0%  
Fibrinogen 196069 196022 196069 0% 143822 0 0 0% 0%  
Follic_Acid 2700849 2698511 2700849 0% 1472362 0 0 0% 0%  
FreeT3 504917 504917 504917 0% 205428 0 0 0% 0%  
FreeT4 8340984 8340984 8340984 0% 2572146 0 0 0% 0%  
FSH 1044260 1043119 1044260 0% 688285 0 0 0% 0%  
GFR 3223523 3211255 3223523 0% 882940 0 0 0% 0%  
GGT 9337150 9307468 9337150 0% 2369564 0 0 0% 0%  
Globulin 7992922 7983404 7992922 0% 1896822 0 0 0% 0%  
Glucose 16291642 16291642 16291642 0% 4466133 0 0 0% 0%  
HbA1C 7510232 7505467 7510232 0% 1520278 0 0 0% 0%  
INR 8402552 8402552 8402552 0% 455102 0 0 0% 0%  
Iron_Fe 610270 610270 610270 0% 363871 0 0 0% 0%  
K+ 28586751 28586751 28586751 0% 5174781 0 0 0% 0%  
LDH 218679 218679 218679 0% 102148 0 0 0% 0%  
Lithium 273406 272629 273406 0% 21344 0 0 0% 0%  
LUC 1328863 1322974 1328863 0% 324604 0 0 0% 0%  
LuteinisingHormone 844893 827658 844893 0% 585366 0 0 0% 0%  
Lymphocytes% 24605312 24601362 24605312 0% 5312408 0 0 0% 0%  
Mg 181832 181832 181832 0% 107908 0 0 0% 0%  
Monocytes% 24385874 24361695 24385874 0% 5276761 0 0 0% 0%  
Na 28917933 28917933 28917933 0% 5195665 0 0 0% 0%  
Neutrophils% 24692458 24688447 24692458 0% 5321777 0 0 0% 0%  
PDW 383011 382479 383011 0% 109155 0 0 0% 0%  
PFR 5127271 5026335 5127271 0% 1355995 0 0 0% 0%  
Phosphore 4557642 4557642 4557642 0% 1866755 0 0 0% 0%  
PlasmaAnionGap 21500 21500 21500 0% 5491 0 0 0% 0%  
PlasmaViscosity 1237193 1237193 1237193 0% 459295 0 0 0% 0%  
Progesterone 233310 227455 233310 0% 156019 0 0 0% 0%  
Prolactin 406527 406065 406527 0% 297447 0 0 0% 0%  
PSA 1924332 1904230 1924332 0% 660296 0 0 0% 0%  
PULSE 5527058 5527058 5527058 0% 2221767 0 0 0% 0%  
RandomGlucose 700394 700394 700394 0% 423583 0 0 0% 0%  
RDW 4960073 4959954 4960073 0% 1579652 0 0 0% 0%  
Reticulocyte 69336 68938 69336 0% 49484 0 0 0% 0%  
Rheumatoid_Factor 460276 449158 460276 0% 364740 0 0 0% 0%  
Serum_Oestradiol 393299 388457 393299 0% 274758 0 0 0% 0%  
SerumAnionGap 59530 59525 59530 0% 22779 0 0 0% 0%  
Sex_Hormone_Binding_Globulin 132646 132582 132646 0% 110127 0 0 0% 0%  
T4 473023 473023 473023 0% 213799 0 0 0% 0%  
Testosterone 386501 386060 386501 0% 292007 0 0 0% 0%  
TIBC 149863 149740 149863 0% 97981 0 0 0% 0%  
Transferrin 382006 382006 382006 0% 232202 0 0 0% 0%  
Transferrin_Saturation_Index 251706 251479 251706 0% 162901 0 0 0% 0%  
Triglycerides 13994934 13994934 13994934 0% 3295899 0 0 0% 0%  
TSH 16164536 15839175 16164536 0% 4501457 0 0 0% 0%  
Urea 22350867 22348716 22350867 0% 4367623 0 0 0% 0%  
Uric_Acid 1149741 1145846 1149741 0% 655507 0 0 0% 0%  
Urine_Dipstick_pH 136044 135994 136044 0% 64581 0 0 0% 0%  
Urine_Epithelial_Cell 299102 287392 299102 0% 145624 0 0 0% 0%  
Urine_Microalbumin 1379741 1262864 1379741 0% 375904 0 0 0% 0%  
Urine_Protein_Creatinine 321517 311603 321517 0% 151145 0 0 0% 0%  
VitaminD_25 349534 349485 349534 0% 228066 0 0 0% 0%  
TEMP 1779120 1779120 1779120 0% 964098 0 0 0% 0%