MADIS RT RUN-TIME STATUS for Sat Aug 29 10:32:01 GMT 2026 on vm-cprk-madis-ops-proc1
     INFO RT VM STATUS for Node  vm-cprk-madis-ops-proc1 Time Sat Aug 29 10:32:01 GMT 2026

     PASS METAR processing tbl                     file:  179772 Aug 25 18:25 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
     PASS Station update flag                      file:  0 Aug 27  2025 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
     WARN SNOTEL files late:       Last updates:  hourly<120min  108  daily<1800min  84  monthly<45000min  0 
     PASS Metar station table                      file:  1121683 Aug 27  2025 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
     PASS Accept/reject|autorej files              Cnts: AO: 3 IQC: 3 NON-SFC: 20
     PASS Persistent processes                     Cnts: pqact (1+): 1 ldmd (1+): 25 DECODE (5): 5
     PASS Dcmetr files                             CNT-LDAD.raw: 547
     PASS RSAS values range check                  Total chk: 317840 Outside of range (<100): 0
     WARN Aircraft data late        Check: REC-CNT>20   #TAMDAR 20260829_0900:20 20260829_0800:58  #PUBLIC-DP 20260829_0900:665 20260829_0800:228  #PUBLIC-DP-profile 20260829_0900:4 20260829_0800:1      
     WARN Aircraft data late        Check: REC-CNT>20   #TAMDAR 20260829_0900:20 20260829_0800:58  #PUBLIC-DP 20260829_0900:665 20260829_0800:228  #PUBLIC-DP-profile 20260829_0900:4 20260829_0800:1      
     PASS Load resource                 m15 < 9 cores 74% (goal > 60%)   -- % of the time M15 < 9  Samples: 633    #10=+ 155 #9 6 #8 6 #7 38
     PASS Open file resource                       check: open file < 5000 Max Files: -1 #<1000: 633 #<10000 633
     PASS Process count                            check: processes < 1000 Max Processes: 461
     PASS AO Throughput                            0 -- check: #"conflicts" <20 delayed %Delayed: 0% #Sample: 252 #Delayed: 0
     INFO AO runtime avg sfcanl                     67  seconds   2818/42
     INFO AO runtime avg sfcncdf                     1  seconds   32/32
     INFO AO runtime avg sfcver                     75  seconds   2638/35
     INFO AO runtime avg sfcqcmsg                   21  seconds   888/42
     INFO AO runtime avg sfcqcout meso              61  seconds   7137/116
     INFO AO runtime avg sfcqcout_late meso        170  seconds   19381/114
     INFO AO runtime avg FSL_qcout5m_               76  seconds   9670/126
     INFO AO runtime avg FSL_analrun15m_           158  seconds   6664/42
     PASS IQC Throughput                           2  -- check: #"conflicts" <30 delayed %Delayed: 2% #Sample: 630 #Delayed: 15 Competed timely:  555
     INFO IQC runtime avg qcstg1_2                   6  seconds   76542/11969
     INFO IQC runtime avg qcstg1_2 meso             30  seconds   37330/1230
     INFO IQC runtime avg qcstg1_2_late meso        29  seconds   27384/930
     INFO IQC runtime avg sfcing                    40  seconds   24951/615
     INFO IQC runtime avg sfchqc                    35  seconds   21633/615
     PASS QC applied max value                     Max applied check should be more than 107, 123 is good:      123,   // temperatureQCA(9999)
     PASS RT Throughput                            153 secs -- check: Avg time < 180 sec. #Hrs: 10 Max secs: 165
     PASS '0' length data Station table            check: cnt '0' length files  #: 0
     PASS '0' length master Station table          check: cnt '0' length files  #: 0
     PASS netcdf-4.1 lib                           NetCDF lib link not in routerStoreNetCDF
     WARN Shefdecoder issues.  Missing stations in either hydro and mesonet:   check 20260829_0900: #CPYRMA: 0 #NJWXNET: 636
     PASS QC DEW POINT DEPRESSION (K)              4.45 % OBS FAILED  CHECK (  1899 OF  42719) AVGTHR 5.2917 RATIO 0.303 AVGEPS 0.1572E-01 FRMSE 18.417 TRMSE 1.6009
     PASS QC POTENTIAL TEMPERATURE (K)             5.30 % OBS FAILED  CHECK (  2396 OF  45236) AVGTHR 3.6434 RATIO 0.315 AVGEPS 0.2422E-01 FRMSE 13.854 TRMSE 1.1463
     PASS QC DEW POINT TEMPERATURE (K)             4.41 % OBS FAILED  CHECK (  1884 OF  42730) AVGTHR 3.9323 RATIO 0.276 AVGEPS 0.1564E-01 FRMSE 20.175 TRMSE 1.0855
     PASS QC U COMPONENT OF SFC WIND (M/S)         0.67 % OBS FAILED  CHECK (   288 OF  43252) AVGTHR 4.4125 RATIO 0.266 AVGEPS 0.1372E-01 FRMSE 7.1754 TRMSE 1.1721
     PASS QC V COMPONENT OF SFC WIND (M/S)         0.49 % OBS FAILED  CHECK (   214 OF  43252) AVGTHR 4.5353 RATIO 0.265 AVGEPS 0.1368E-01 FRMSE 7.0577 TRMSE 1.2036
     PASS QC 3H ALTIMETER CHANGE (MB)              0.32 % OBS FAILED  CHECK (    59 OF  18282) AVGTHR 1.2987 RATIO 0.224 AVGEPS 0.1802E-01 FRMSE 3.6567 TRMSE 0.29045
     PASS QC NWS SEA-LEVEL PRESSURE (MB)           6.03 % OBS FAILED  CHECK (    64 OF   1061) AVGTHR 1.4867 RATIO 0.335 AVGEPS 0.4372E-01 FRMSE 4.1224 TRMSE 0.49831
     PASS QC ALTIMETER (MB)                       21.56 % OBS FAILED  CHECK (  4261 OF  19759) AVGTHR 1.9351 RATIO 0.100E+06 AVGEPS 0.2583E-01 FRMSE 38.006 TRMSE 99999.
     PASS MXST_P limit                             283389    MXST_P: 500000
     PASS AO Possible Seg fault                    0  #logs checked: 2278  
     PASS IQC Possible Seg fault                   0  #logs checked: 5146  
     PASS NONSFC Possible Seg fault                0  #logs checked: 239  
     PASS MADIS DB Connection ok                    madis/proc/storage/log/MadisDbUpload/MadisDataUpload_2026.08.29.10.05.35_00000.log
     INFO MADIS DB Connection cnts                 [2026-08-29 10:21:49.271473]: LastUploadStats:mode:upload_sfc,pid:1322807,aborted:0,fileCount:1,wholeFileErrors:0,lines:12877,inserts:11168,updates:1124,duplicates:0,errors:1216,stationsAdded:0,stationsUpdated:36,publishLines:0,publishErrors:0,notes:./202608291021_receipt.sfcd2.txt,run ms:20245
     PASS AFWS files current                       # in last hour: 20
     PASS madis.noaa.gov.stations current             # updated in last day: 2
     PASS madis.noaa.gov.trans current             # updated in last day: 147
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Today Day: 26241
                                     madis/proc/sfc/ao/output/sfcncdf.out.262410000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262410100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262410200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262410300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262410400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262410500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262410600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262410700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262410800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262410900_4:
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Prior Day: 26240
                                     madis/proc/sfc/ao/output/sfcncdf.out.262391200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262391300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262391400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262391500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262391600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262391700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262391800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262391900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262392000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262392100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262392200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262392300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262400000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262400100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262400200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262400300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262400400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262400500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262400600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262400700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262400800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262400900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262401000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262401100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262401200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262401300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262401400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262401500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262401600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262401700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262401800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262401900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262402000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262402100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262402200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262402300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262410000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262410100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262410200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262410300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262410400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262410500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262410600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262410700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262410800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262410900_4:
     PASS SFCANL MATRIX FAILURE                     # Today Day: 26241
     WARN SFCANL MATRIX FAILURE                     # Prior Day: 26240
                                     Binary
                                     madis/proc/sfc/ao/output/sfcanl.out.262400400_1:
                                     madis/proc/sfc/ao/output/sfcanl.out.262400700_4:
                                     madis/proc/sfc/ao/output/sfcanl.out.262401500_4:

  COMPARE Mesonet data providers cnt                43     for 20260829_0800:
     PASS AO SEG FAULTS sfc/ao/logs
     PASS IQC SEG FAULTS sfc/iqc*/logs
     PASS NONSFC SEG FAULTS nonsfc/logs
     PASS AIRCRAFT WEB GOV DATA FILE EXISTS /common/data/apps/madis/current/private/aircraft/gov/datacube/DataCubeReadOnly/Aircraft/2026/08/29/Aircraft.2026.08.29_10.mv.db
     PASS AIRCRAFT WEB GOV DATA FILE EXISTS /common/data/apps/madis/current/private/aircraft/public/datacube/DataCubeReadOnly/Aircraft/2026/08/29/Aircraft.2026.08.29_10.mv.db
  COMPARE DECODE STN TBLS                           149        Newest:   234921 Aug 29 01:50 LDAD/data/RAWSStation.txt  Oldest:     8848 Oct  3  2025 LDAD/data/AWS-GANetStation.txt
  COMPARE PRE-DECODE STN TBLS                       149        Newest:   234921 Aug 29 01:50 LDAD/data/master/RAWSStation.txt  Oldest:     1568 Oct  3  2025 LDAD/data/master/AWS-KSNetStation.txt
  COMPARE pqsend entries                            6           (Only on primary, 1 for each feedtype/distr)
  COMPARE Surface QC text products                  7320         hour (09): Newest: 2 Aug 29 10:27 madis/proc/sfc/ao/qc122/qcsumry10.dat Oldest:
  COMPARE sda IO Writes 2624108                     Local Disk MB/s: 2 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3445+
  COMPARE sda IO Reads(2624108                      Local Disk MB/s: 0 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3445+
  COMPARE NetCDF REC Writes 2624108                 Avg DECODE rec/s: 86 Avg AO rec/s: 90 Avg NON-SFC rec/s: 1 Tot secs: 3445
     INFO Network  2624108                          RX 0 MB/s TX    0 MB/s RX     Tot secs: 3445  
  COMPARE opt Disk 2624108                          0 WRITE MB/s | 0 READ MB/s | 0 Avg WRT TPS | 0 Max 1-min WRT TPS

  COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size.  Note: Max #recNum has to be < MXST_P 500000
AO-20260829_08  sfc/ao/qcms_data/buoy/20260829_0800                               5089    5968444
AO-20260829_08  sfc/ao/qcms_data/crn/20260829_0800                                 118     250216
AO-20260829_08  sfc/ao/qcms_data/hcn/20260829_0800                                  15      88832
AO-20260829_08  sfc/ao/qcms_data/hfmetar/20260829_0800                           12047   18958404
AO-20260829_08  sfc/ao/qcms_data/hydro/20260829_0800                             17404   22280672
AO-20260829_08  sfc/ao/qcms_data/mesonet/20260829_0800                          259899  575301040
AO-20260829_08  sfc/ao/qcms_data/metar/20260829_0800                              7894   11815780
AO-20260829_08  sfc/ao/qcms_data/nepp/20260829_0800                                216     386560
AO-20260829_08  sfc/ao/qcms_data/rwis/20260829_0800                               5838   24007668
AO-20260829_08  sfc/ao/qcms_data/urbanet/20260829_0800                            3606    9949552

NS-20260829_08  nonsfc/madis_data/LDAD/profiler/netCDF/20260829_0800                37    1904512
NS-20260829_08  nonsfc/madis_data/LDAD/profiler/netCDF/20260829_0830                30    1552104
NS-20260829_08  nonsfc/madis_data/LDAD/snow/netCDF/20260829_0800                  3759    4054392
NS-20260829_08  nonsfc/madis_data/point/raob/netcdf/20260829_0800                    1     180508

AO-20260829_07  sfc/ao/qcms_data/buoy/20260829_0700                               5229    6128604
AO-20260829_07  sfc/ao/qcms_data/crn/20260829_0700                                 118     250216
AO-20260829_07  sfc/ao/qcms_data/hcn/20260829_0700                                  15      88832
AO-20260829_07  sfc/ao/qcms_data/hfmetar/20260829_0700                           10998   17326160
AO-20260829_07  sfc/ao/qcms_data/hydro/20260829_0700                             15008   19395888
AO-20260829_07  sfc/ao/qcms_data/mesonet/20260829_0700                          260082  575704372
AO-20260829_07  sfc/ao/qcms_data/metar/20260829_0700                              7845   11743652
AO-20260829_07  sfc/ao/qcms_data/nepp/20260829_0700                                216     386560
AO-20260829_07  sfc/ao/qcms_data/rwis/20260829_0700                               5845   24035024
AO-20260829_07  sfc/ao/qcms_data/urbanet/20260829_0700                            3611    9962852

NS-20260829_07  nonsfc/madis_data/LDAD/profiler/netCDF/20260829_0700                37    1904512
NS-20260829_07  nonsfc/madis_data/LDAD/profiler/netCDF/20260829_0730                30    1552104
NS-20260829_07  nonsfc/madis_data/LDAD/snow/netCDF/20260829_0700                  3749    4044352

DECODE          decode/data/fxa/LDAD/crn/netCDF/20260829_0800                      118     184864
DECODE          decode/data/fxa/LDAD/hcn/netCDF/20260829_0800                       15      66244
DECODE          decode/data/fxa/LDAD/hfmetar/netCDF/20260829_0800                12047   13346524
DECODE          decode/data/fxa/LDAD/hydro/netCDF/20260829_0800                  17405   14634964
DECODE          decode/data/fxa/LDAD/mesonet/netCDF/20260829_0800               260932  322937664
DECODE          decode/data/fxa/LDAD/nepp/netCDF/20260829_0800                     216     229908
DECODE          decode/data/fxa/LDAD/rwis/netCDF/20260829_0800                    5838   18681128
DECODE          decode/data/fxa/LDAD/snow/netCDF/20260829_0800                    3759    2901384
DECODE          decode/data/fxa/LDAD/urbanet/netCDF/20260829_0800                 3606    3358224