MADIS RT RUN-TIME STATUS for Thu Jul 23 09:32:02 GMT 2026 on vm-cprk-madis-ops-proc1
     INFO RT VM STATUS for Node  vm-cprk-madis-ops-proc1 Time Thu Jul 23 09:32:02 GMT 2026

     PASS METAR processing tbl                     file:  179772 Jul 21 17:30 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
     PASS Station update flag                      file:  0 Aug 27  2025 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
     WARN SNOTEL files late:       Last updates:  hourly<120min  108  daily<1800min  84  monthly<45000min  0 
     PASS Metar station table                      file:  1121683 Aug 27  2025 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
     PASS Accept/reject|autorej files              Cnts: AO: 3 IQC: 3 NON-SFC: 20
     PASS Persistent processes                     Cnts: pqact (1+): 1 ldmd (1+): 25 DECODE (5): 5
     PASS Dcmetr files                             CNT-LDAD.raw: 543
     PASS RSAS values range check                  Total chk: 317840 Outside of range (<100): 0
     WARN Aircraft data late        Check: REC-CNT>20   #TAMDAR 20260723_0800:25 20260723_0700:38  #PUBLIC-DP 20260723_0800:861 20260723_0700:1047  #PUBLIC-DP-profile 20260723_0800:7 20260723_0700:11      
     WARN Aircraft data late        Check: REC-CNT>20   #TAMDAR 20260723_0800:25 20260723_0700:38  #PUBLIC-DP 20260723_0800:861 20260723_0700:1047  #PUBLIC-DP-profile 20260723_0800:7 20260723_0700:11      
     PASS Load resource                 m15 < 9 cores 94% (goal > 60%)   -- % of the time M15 < 9  Samples: 572    #10=+ 22 #9 7 #8 3 #7 26
     PASS Open file resource                       check: open file < 5000 Max Files: -1 #<1000: 572 #<10000 572
     PASS Process count                            check: processes < 1000 Max Processes: 531
     PASS AO Throughput                            0 -- check: #"conflicts" <20 delayed %Delayed: 0% #Sample: 228 #Delayed: 0
     INFO AO runtime avg sfcanl                     66  seconds   2535/38
     INFO AO runtime avg sfcncdf                     1  seconds   23/23
     INFO AO runtime avg sfcver                     86  seconds   2334/27
     INFO AO runtime avg sfcqcmsg                   17  seconds   670/38
     INFO AO runtime avg sfcqcout meso              61  seconds   6416/104
     INFO AO runtime avg sfcqcout_late meso        169  seconds   17331/102
     INFO AO runtime avg FSL_qcout5m_               74  seconds   8494/114
     INFO AO runtime avg FSL_analrun15m_           152  seconds   5780/38
     PASS IQC Throughput                           1  -- check: #"conflicts" <30 delayed %Delayed: 1% #Sample: 570 #Delayed: 9 Competed timely:  506
     INFO IQC runtime avg qcstg1_2                   7  seconds   75602/10730
     INFO IQC runtime avg qcstg1_2 meso             33  seconds   37798/1122
     INFO IQC runtime avg qcstg1_2_late meso        32  seconds   27425/847
     INFO IQC runtime avg sfcing                    44  seconds   25203/561
     INFO IQC runtime avg sfchqc                    37  seconds   21313/561
     PASS QC applied max value                     Max applied check should be more than 107, 123 is good:      123,   // temperatureQCA(99999)
     PASS RT Throughput                            156 secs -- check: Avg time < 180 sec. #Hrs: 9 Max secs: 165
     PASS '0' length data Station table            check: cnt '0' length files  #: 0
     PASS '0' length master Station table          check: cnt '0' length files  #: 0
     PASS netcdf-4.1 lib                           NetCDF lib link not in routerStoreNetCDF
     WARN Shefdecoder issues.  Missing stations in either hydro and mesonet:   check 20260723_0800: #CPYRMA: 0 #NJWXNET: 648
     PASS QC DEW POINT DEPRESSION (K)              4.42 % OBS FAILED  CHECK (  2247 OF  50808) AVGTHR 5.2824 RATIO 0.300 AVGEPS 0.1542E-01 FRMSE 20.829 TRMSE 1.5856
     PASS QC POTENTIAL TEMPERATURE (K)             4.33 % OBS FAILED  CHECK (  2324 OF  53635) AVGTHR 3.6499 RATIO 0.342 AVGEPS 0.2461E-01 FRMSE 14.014 TRMSE 1.2485
     PASS QC DEW POINT TEMPERATURE (K)             4.82 % OBS FAILED  CHECK (  2450 OF  50819) AVGTHR 3.9319 RATIO 0.243 AVGEPS 0.1571E-01 FRMSE 22.293 TRMSE 0.95688
     PASS QC U COMPONENT OF SFC WIND (M/S)         1.27 % OBS FAILED  CHECK (   654 OF  51692) AVGTHR 4.4202 RATIO 0.275 AVGEPS 0.1419E-01 FRMSE 9.0860 TRMSE 1.2164
     PASS QC V COMPONENT OF SFC WIND (M/S)         1.03 % OBS FAILED  CHECK (   531 OF  51692) AVGTHR 4.5425 RATIO 0.279 AVGEPS 0.1403E-01 FRMSE 9.4607 TRMSE 1.2659
     PASS QC 3H ALTIMETER CHANGE (MB)              0.77 % OBS FAILED  CHECK (   162 OF  21168) AVGTHR 1.3194 RATIO 0.239 AVGEPS 0.1839E-01 FRMSE 38.158 TRMSE 0.31570
     PASS QC NWS SEA-LEVEL PRESSURE (MB)           5.72 % OBS FAILED  CHECK (    69 OF   1206) AVGTHR 1.5286 RATIO 0.401 AVGEPS 0.5618E-01 FRMSE 3.1702 TRMSE 0.61267
     PASS QC ALTIMETER (MB)                       22.36 % OBS FAILED  CHECK (  5129 OF  22942) AVGTHR 1.9454 RATIO 0.100E+06 AVGEPS 0.2745E-01 FRMSE 39.891 TRMSE 99999.
     FAIL MXST_P sizing check dailed to no information in the log file madis/proc//sfc/ao/output/sfcanl.out.262040800_3
     PASS AO Possible Seg fault                    0  #logs checked: 2274  
     PASS IQC Possible Seg fault                   0  #logs checked: 5301  
     PASS NONSFC Possible Seg fault                0  #logs checked: 235  
     PASS MADIS DB Connection ok                    madis/proc/storage/log/MadisDbUpload/MadisDataUpload_2026.07.23.09.06.10_00000.log
     INFO MADIS DB Connection cnts                 [2026-07-23 09:22:24.954327]: LastUploadStats:mode:upload_sfc,pid:2521854,aborted:0,fileCount:1,wholeFileErrors:0,lines:26671,inserts:23361,updates:2618,duplicates:0,errors:1772,stationsAdded:0,stationsUpdated:92,publishLines:0,publishErrors:0,notes:./202607230921_receipt.sfcd2.txt,run ms:34565
     PASS AFWS files current                       # in last hour: 20
     PASS madis.noaa.gov.stations current             # updated in last day: 2
     PASS madis.noaa.gov.trans current             # updated in last day: 147
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Today Day: 26204
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040800_4:
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Prior Day: 26203
                                     madis/proc/sfc/ao/output/sfcncdf.out.262021100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262021200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262021300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262021400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262021500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262021600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262021700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262021800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262021900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262022000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262022100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262022200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262022300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262030000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262030100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262030200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262030300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262030400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262030500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262030600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262030700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262030800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262030900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262031000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262031100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262031200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262031300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262031400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262031500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262031600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262031700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262031800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262031900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262032000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262032100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262032200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262032300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040800_4:
     PASS SFCANL MATRIX FAILURE                     # Today Day: 26204
     PASS SFCANL MATRIX FAILURE                     # Prior Day: 26203

  COMPARE Mesonet data providers cnt                44     for 20260723_0700:
     PASS AO SEG FAULTS sfc/ao/logs
     PASS IQC SEG FAULTS sfc/iqc*/logs
     PASS NONSFC SEG FAULTS nonsfc/logs
     PASS AIRCRAFT WEB GOV DATA FILE EXISTS /common/data/apps/madis/current/private/aircraft/gov/datacube/DataCubeReadOnly/Aircraft/2026/07/23/Aircraft.2026.07.23_09.mv.db
     PASS AIRCRAFT WEB GOV DATA FILE EXISTS /common/data/apps/madis/current/private/aircraft/public/datacube/DataCubeReadOnly/Aircraft/2026/07/23/Aircraft.2026.07.23_09.mv.db
  COMPARE DECODE STN TBLS                           149        Newest:   232900 Jul 23 01:51 LDAD/data/RAWSStation.txt  Oldest:     8848 Oct  3  2025 LDAD/data/AWS-GANetStation.txt
  COMPARE PRE-DECODE STN TBLS                       149        Newest:   232900 Jul 23 01:51 LDAD/data/master/RAWSStation.txt  Oldest:     1568 Oct  3  2025 LDAD/data/master/AWS-KSNetStation.txt
  COMPARE pqsend entries                            6           (Only on primary, 1 for each feedtype/distr)
  COMPARE Surface QC text products                  7320         hour (08): Newest: 5 Jul 23 09:27 madis/proc/sfc/ao/qc122/qcsumry09.dat Oldest:
  COMPARE sda IO Writes 2620407                     Local Disk MB/s: 1 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3469+
  COMPARE sda IO Reads(2620407                      Local Disk MB/s: 0 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3469+
  COMPARE NetCDF REC Writes 2620407                 Avg DECODE rec/s: 96 Avg AO rec/s: 101 Avg NON-SFC rec/s: 0 Tot secs: 3469
     INFO Network  2620407                          RX 0 MB/s TX    0 MB/s RX     Tot secs: 3469  
  COMPARE opt Disk 2620407                          0 WRITE MB/s | 0 READ MB/s | 0 Avg WRT TPS | 0 Max 1-min WRT TPS

  COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size.  Note: Max #recNum has to be < MXST_P -1
AO-20260723_07  sfc/ao/qcms_data/buoy/20260723_0700                               5186    6079412
AO-20260723_07  sfc/ao/qcms_data/crn/20260723_0700                                 117     248764
AO-20260723_07  sfc/ao/qcms_data/hcn/20260723_0700                                  15      88832
AO-20260723_07  sfc/ao/qcms_data/hfmetar/20260723_0700                           11820   18605192
AO-20260723_07  sfc/ao/qcms_data/hydro/20260723_0700                             14236   18466400
AO-20260723_07  sfc/ao/qcms_data/mesonet/20260723_0700                          302468  669123116
AO-20260723_07  sfc/ao/qcms_data/metar/20260723_0700                              7879   11793700
AO-20260723_07  sfc/ao/qcms_data/nepp/20260723_0700                                216     386560
AO-20260723_07  sfc/ao/qcms_data/rwis/20260723_0700                               6426   26305572
AO-20260723_07  sfc/ao/qcms_data/urbanet/20260723_0700                            3449    9531932

NS-20260723_07  nonsfc/madis_data/LDAD/profiler/netCDF/20260723_0700                34    1753480
NS-20260723_07  nonsfc/madis_data/LDAD/profiler/netCDF/20260723_0730                28    1451416
NS-20260723_07  nonsfc/madis_data/LDAD/snow/netCDF/20260723_0700                  2794    3085532

AO-20260723_06  sfc/ao/qcms_data/buoy/20260723_0600                               5300    6209828
AO-20260723_06  sfc/ao/qcms_data/crn/20260723_0600                                  83     199396
AO-20260723_06  sfc/ao/qcms_data/hcn/20260723_0600                                  15      88832
AO-20260723_06  sfc/ao/qcms_data/hfmetar/20260723_0600                           11764   18518056
AO-20260723_06  sfc/ao/qcms_data/hydro/20260723_0600                             16539   21239212
AO-20260723_06  sfc/ao/qcms_data/mesonet/20260723_0600                          301540  667077804
AO-20260723_06  sfc/ao/qcms_data/metar/20260723_0600                              7855   11758372
AO-20260723_06  sfc/ao/qcms_data/nepp/20260723_0600                                216     386560
AO-20260723_06  sfc/ao/qcms_data/rwis/20260723_0600                               6376   26110172
AO-20260723_06  sfc/ao/qcms_data/urbanet/20260723_0600                            3449    9531932

NS-20260723_06  nonsfc/madis_data/LDAD/profiler/netCDF/20260723_0600                34    1753480
NS-20260723_06  nonsfc/madis_data/LDAD/profiler/netCDF/20260723_0630                27    1401072
NS-20260723_06  nonsfc/madis_data/LDAD/snow/netCDF/20260723_0600                  2854    3145772
NS-20260723_06  nonsfc/madis_data/point/raob/netcdf/20260723_0600                   18    1344260

DECODE          decode/data/fxa/LDAD/crn/netCDF/20260723_0700                      117     183836
DECODE          decode/data/fxa/LDAD/hcn/netCDF/20260723_0700                       15      66244
DECODE          decode/data/fxa/LDAD/hfmetar/netCDF/20260723_0700                11820   13098640
DECODE          decode/data/fxa/LDAD/hydro/netCDF/20260723_0700                  14236   12150468
DECODE          decode/data/fxa/LDAD/mesonet/netCDF/20260723_0700               303285  375116560
DECODE          decode/data/fxa/LDAD/nepp/netCDF/20260723_0700                     216     229908
DECODE          decode/data/fxa/LDAD/rwis/netCDF/20260723_0700                    6426   20447480
DECODE          decode/data/fxa/LDAD/snow/netCDF/20260723_0700                    2794    2225884
DECODE          decode/data/fxa/LDAD/urbanet/netCDF/20260723_0700                 3449    3224460