MADIS RT RUN-TIME STATUS for Thu Jul 23 06:32:02 GMT 2026 on vm-cprk-madis-ops-proc1
     INFO RT VM STATUS for Node  vm-cprk-madis-ops-proc1 Time Thu Jul 23 06:32:02 GMT 2026

     PASS METAR processing tbl                     file:  179772 Jul 21 17:30 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
     PASS Station update flag                      file:  0 Aug 27  2025 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
     WARN SNOTEL files late:       Last updates:  hourly<120min  108  daily<1800min  84  monthly<45000min  0 
     PASS Metar station table                      file:  1121683 Aug 27  2025 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
     PASS Accept/reject|autorej files              Cnts: AO: 3 IQC: 3 NON-SFC: 20
     PASS Persistent processes                     Cnts: pqact (1+): 1 ldmd (1+): 25 DECODE (5): 5
     PASS Dcmetr files                             CNT-LDAD.raw: 542
     PASS RSAS values range check                  Total chk: 317840 Outside of range (<100): 0
     WARN Aircraft data late        Check: REC-CNT>20   #TAMDAR 20260723_0500:40 20260723_0400:48  #PUBLIC-DP 20260723_0500:1655 20260723_0400:2314  #PUBLIC-DP-profile 20260723_0500:14 20260723_0400:27      
     WARN Aircraft data late        Check: REC-CNT>20   #TAMDAR 20260723_0500:40 20260723_0400:48  #PUBLIC-DP 20260723_0500:1655 20260723_0400:2314  #PUBLIC-DP-profile 20260723_0500:14 20260723_0400:27      
     PASS Load resource                 m15 < 9 cores 92% (goal > 60%)   -- % of the time M15 < 9  Samples: 391    #10=+ 22 #9 7 #8 3 #7 26
     PASS Open file resource                       check: open file < 5000 Max Files: -1 #<1000: 391 #<10000 391
     PASS Process count                            check: processes < 1000 Max Processes: 531
     PASS AO Throughput                            0 -- check: #"conflicts" <20 delayed %Delayed: 0% #Sample: 156 #Delayed: 0
     INFO AO runtime avg sfcanl                     74  seconds   1926/26
     INFO AO runtime avg sfcncdf                     1  seconds   18/18
     INFO AO runtime avg sfcver                     86  seconds   1817/21
     INFO AO runtime avg sfcqcmsg                   17  seconds   461/26
     INFO AO runtime avg sfcqcout meso              61  seconds   4355/71
     INFO AO runtime avg sfcqcout_late meso        169  seconds   11203/66
     INFO AO runtime avg FSL_qcout5m_               74  seconds   5785/78
     INFO AO runtime avg FSL_analrun15m_           168  seconds   4369/26
     PASS IQC Throughput                           1  -- check: #"conflicts" <30 delayed %Delayed: 1% #Sample: 393 #Delayed: 7 Competed timely:  328
     INFO IQC runtime avg qcstg1_2                   7  seconds   50149/7145
     INFO IQC runtime avg qcstg1_2 meso             33  seconds   25563/766
     INFO IQC runtime avg qcstg1_2_late meso        32  seconds   17714/551
     INFO IQC runtime avg sfcing                    44  seconds   17124/383
     INFO IQC runtime avg sfchqc                    37  seconds   14448/383
     PASS QC applied max value                     Max applied check should be more than 107, 123 is good:      123,   // temperatureQCA(99999)
     PASS RT Throughput                            155 secs -- check: Avg time < 180 sec. #Hrs: 6 Max secs: 165
     PASS '0' length data Station table            check: cnt '0' length files  #: 0
     PASS '0' length master Station table          check: cnt '0' length files  #: 0
     PASS netcdf-4.1 lib                           NetCDF lib link not in routerStoreNetCDF
     WARN Shefdecoder issues.  Missing stations in either hydro and mesonet:   check 20260723_0500: #CPYRMA: 0 #NJWXNET: 648
     PASS QC DEW POINT DEPRESSION (K)              4.51 % OBS FAILED  CHECK (  2289 OF  50733) AVGTHR 5.2846 RATIO 0.330 AVGEPS 0.1556E-01 FRMSE 19.516 TRMSE 1.7433
     PASS QC POTENTIAL TEMPERATURE (K)             3.93 % OBS FAILED  CHECK (  2099 OF  53469) AVGTHR 3.6512 RATIO 0.338 AVGEPS 0.2485E-01 FRMSE 15.778 TRMSE 1.2339
     PASS QC DEW POINT TEMPERATURE (K)             4.64 % OBS FAILED  CHECK (  2353 OF  50744) AVGTHR 3.9366 RATIO 0.270 AVGEPS 0.1614E-01 FRMSE 21.963 TRMSE 1.0612
     PASS QC U COMPONENT OF SFC WIND (M/S)         1.51 % OBS FAILED  CHECK (   784 OF  51798) AVGTHR 4.4226 RATIO 0.300 AVGEPS 0.1436E-01 FRMSE 8.7002 TRMSE 1.3270
     PASS QC V COMPONENT OF SFC WIND (M/S)         1.15 % OBS FAILED  CHECK (   597 OF  51798) AVGTHR 4.5470 RATIO 0.302 AVGEPS 0.1430E-01 FRMSE 9.1327 TRMSE 1.3753
     PASS QC 3H ALTIMETER CHANGE (MB)              1.53 % OBS FAILED  CHECK (   323 OF  21133) AVGTHR 1.3004 RATIO 0.332 AVGEPS 0.1915E-01 FRMSE 22.535 TRMSE 0.43207
     PASS QC NWS SEA-LEVEL PRESSURE (MB)           5.82 % OBS FAILED  CHECK (    68 OF   1169) AVGTHR 1.5495 RATIO 0.366 AVGEPS 0.6105E-01 FRMSE 3.0854 TRMSE 0.56696
     PASS QC ALTIMETER (MB)                       22.40 % OBS FAILED  CHECK (  5206 OF  23245) AVGTHR 1.9492 RATIO 0.100E+06 AVGEPS 0.2808E-01 FRMSE 38.193 TRMSE 99999.
     PASS MXST_P limit                             329469    MXST_P: 500000
     PASS AO Possible Seg fault                    0  #logs checked: 2274  
     PASS IQC Possible Seg fault                   0  #logs checked: 5313  
     PASS NONSFC Possible Seg fault                0  #logs checked: 241  
     PASS MADIS DB Connection ok                    madis/proc/storage/log/MadisDbUpload/MadisDataUpload_2026.07.23.06.06.11_00000.log
     INFO MADIS DB Connection cnts                 [2026-07-23 06:22:47.253128]: LastUploadStats:mode:upload_sfc,pid:2134273,aborted:0,fileCount:1,wholeFileErrors:0,lines:34764,inserts:29631,updates:4371,duplicates:4,errors:1944,stationsAdded:0,stationsUpdated:86,publishLines:0,publishErrors:0,notes:./202607230621_receipt.sfcd2.txt,run ms:48225
     PASS AFWS files current                       # in last hour: 20
     PASS madis.noaa.gov.stations current             # updated in last day: 2
     PASS madis.noaa.gov.trans current             # updated in last day: 147
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Today Day: 26204
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040500_4:
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Prior Day: 26203
                                     madis/proc/sfc/ao/output/sfcncdf.out.262020800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262020900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262021000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262021100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262021200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262021300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262021400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262021500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262021600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262021700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262021800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262021900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262022000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262022100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262022200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262022300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262030000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262030100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262030200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262030300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262030400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262030500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262030600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262030700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262030800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262030900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262031000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262031100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262031200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262031300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262031400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262031500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262031600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262031700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262031800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262031900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262032000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262032100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262032200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262032300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262040500_4:
     PASS SFCANL MATRIX FAILURE                     # Today Day: 26204
     PASS SFCANL MATRIX FAILURE                     # Prior Day: 26203

  COMPARE Mesonet data providers cnt                45     for 20260723_0400:
     PASS AO SEG FAULTS sfc/ao/logs
     PASS IQC SEG FAULTS sfc/iqc*/logs
     PASS NONSFC SEG FAULTS nonsfc/logs
     PASS AIRCRAFT WEB GOV DATA FILE EXISTS /common/data/apps/madis/current/private/aircraft/gov/datacube/DataCubeReadOnly/Aircraft/2026/07/23/Aircraft.2026.07.23_06.mv.db
     PASS AIRCRAFT WEB GOV DATA FILE EXISTS /common/data/apps/madis/current/private/aircraft/public/datacube/DataCubeReadOnly/Aircraft/2026/07/23/Aircraft.2026.07.23_06.mv.db
  COMPARE DECODE STN TBLS                           149        Newest:   232900 Jul 23 01:51 LDAD/data/RAWSStation.txt  Oldest:     8848 Oct  3  2025 LDAD/data/AWS-GANetStation.txt
  COMPARE PRE-DECODE STN TBLS                       149        Newest:   232900 Jul 23 01:51 LDAD/data/master/RAWSStation.txt  Oldest:     1568 Oct  3  2025 LDAD/data/master/AWS-KSNetStation.txt
  COMPARE pqsend entries                            6           (Only on primary, 1 for each feedtype/distr)
  COMPARE Surface QC text products                  7320         hour (05): Newest: 3 Jul 23 06:27 madis/proc/sfc/ao/qc122/qcsumry06.dat Oldest:
  COMPARE sda IO Writes 2620404                     Local Disk MB/s: 1 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3481+
  COMPARE sda IO Reads(2620404                      Local Disk MB/s: 0 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3481+
  COMPARE NetCDF REC Writes 2620404                 Avg DECODE rec/s: 96 Avg AO rec/s: 100 Avg NON-SFC rec/s: 0 Tot secs: 3481
     INFO Network  2620404                          RX 0 MB/s TX    0 MB/s RX     Tot secs: 3481  
  COMPARE opt Disk 2620404                          0 WRITE MB/s | 0 READ MB/s | 0 Avg WRT TPS | 0 Max 1-min WRT TPS

  COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size.  Note: Max #recNum has to be < MXST_P 500000
AO-20260723_04  sfc/ao/qcms_data/buoy/20260723_0400                               5183    6075980
AO-20260723_04  sfc/ao/qcms_data/crn/20260723_0400                                 117     248764
AO-20260723_04  sfc/ao/qcms_data/hcn/20260723_0400                                  15      88832
AO-20260723_04  sfc/ao/qcms_data/hfmetar/20260723_0400                           11769   18525836
AO-20260723_04  sfc/ao/qcms_data/hydro/20260723_0400                             17469   22358932
AO-20260723_04  sfc/ao/qcms_data/mesonet/20260723_0400                          303208  670754076
AO-20260723_04  sfc/ao/qcms_data/metar/20260723_0400                              7655   11463972
AO-20260723_04  sfc/ao/qcms_data/nepp/20260723_0400                                216     386560
AO-20260723_04  sfc/ao/qcms_data/rwis/20260723_0400                               6395   26184424
AO-20260723_04  sfc/ao/qcms_data/urbanet/20260723_0400                            3453    9542572

NS-20260723_04  nonsfc/madis_data/LDAD/profiler/netCDF/20260723_0400                34    1753480
NS-20260723_04  nonsfc/madis_data/LDAD/profiler/netCDF/20260723_0430                27    1401072
NS-20260723_04  nonsfc/madis_data/LDAD/snow/netCDF/20260723_0400                  2758    3049388

AO-20260723_03  sfc/ao/qcms_data/buoy/20260723_0300                               4627    5439916
AO-20260723_03  sfc/ao/qcms_data/crn/20260723_0300                                 117     248764
AO-20260723_03  sfc/ao/qcms_data/hcn/20260723_0300                                  15      88832
AO-20260723_03  sfc/ao/qcms_data/hfmetar/20260723_0300                           11935   18784132
AO-20260723_03  sfc/ao/qcms_data/hydro/20260723_0300                             17003   21797868
AO-20260723_03  sfc/ao/qcms_data/mesonet/20260723_0300                          293910  650261284
AO-20260723_03  sfc/ao/qcms_data/metar/20260723_0300                              7664   11477220
AO-20260723_03  sfc/ao/qcms_data/nepp/20260723_0300                                216     386560
AO-20260723_03  sfc/ao/qcms_data/rwis/20260723_0300                               6393   26176608
AO-20260723_03  sfc/ao/qcms_data/urbanet/20260723_0300                            2406    6757552

NS-20260723_03  nonsfc/madis_data/LDAD/profiler/netCDF/20260723_0300                33    1703136
NS-20260723_03  nonsfc/madis_data/LDAD/profiler/netCDF/20260723_0330                27    1401072
NS-20260723_03  nonsfc/madis_data/LDAD/snow/netCDF/20260723_0300                  2838    3129708

DECODE          decode/data/fxa/LDAD/crn/netCDF/20260723_0400                      117     183836
DECODE          decode/data/fxa/LDAD/hcn/netCDF/20260723_0400                       15      66244
DECODE          decode/data/fxa/LDAD/hfmetar/netCDF/20260723_0400                11769   13042948
DECODE          decode/data/fxa/LDAD/hydro/netCDF/20260723_0400                  17469   14685140
DECODE          decode/data/fxa/LDAD/mesonet/netCDF/20260723_0400               304085  376102160
DECODE          decode/data/fxa/LDAD/nepp/netCDF/20260723_0400                     216     229908
DECODE          decode/data/fxa/LDAD/rwis/netCDF/20260723_0400                    6395   20354356
DECODE          decode/data/fxa/LDAD/snow/netCDF/20260723_0400                    2758    2200684
DECODE          decode/data/fxa/LDAD/urbanet/netCDF/20260723_0400                 3453    3227868