MADIS RT RUN-TIME STATUS for Thu Aug 13 12:32:02 GMT 2026 on vm-bldr-madis-ops-proc1
     INFO RT VM STATUS for Node  vm-bldr-madis-ops-proc1 Time Thu Aug 13 12:32:02 GMT 2026

     PASS METAR processing tbl                     file:  179772 Aug 11 17:30 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
     PASS Station update flag                      file:  0 Aug 27  2025 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
     WARN SNOTEL files late:       Last updates:  hourly<120min  108  daily<1800min  84  monthly<45000min  0 
     PASS Metar station table                      file:  1121683 Aug 27  2025 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
     PASS Accept/reject|autorej files              Cnts: AO: 3 IQC: 3 NON-SFC: 20
     PASS Persistent processes                     Cnts: pqact (1+): 1 ldmd (1+): 27 DECODE (5): 5
     PASS Dcmetr files                             CNT-LDAD.raw: 557
     PASS RSAS values range check                  Total chk: 317840 Outside of range (<100): 0
     WARN Aircraft data late        Check: REC-CNT>20   #TAMDAR 20260813_1100:37 20260813_1000:19  #PUBLIC-DP 20260813_1100:3098 20260813_1000:2720  #PUBLIC-DP-profile 20260813_1100:36 20260813_1000:21      
     WARN Aircraft data late        Check: REC-CNT>20   #TAMDAR 20260813_1100:37 20260813_1000:19  #PUBLIC-DP 20260813_1100:3098 20260813_1000:2720  #PUBLIC-DP-profile 20260813_1100:36 20260813_1000:21      
     PASS Load resource                 m15 < 9 cores 100% (goal > 60%)   -- % of the time M15 < 9  Samples: 752    #10=+ 0 #9 0 #8 0 #7 13
     PASS Open file resource                       check: open file < 5000 Max Files: -1 #<1000: 752 #<10000 752
     PASS Process count                            check: processes < 1000 Max Processes: 530
     PASS AO Throughput                            0 -- check: #"conflicts" <20 delayed %Delayed: 0% #Sample: 300 #Delayed: 0
     INFO AO runtime avg sfcanl                     69  seconds   3494/50
     INFO AO runtime avg sfcncdf                     1  seconds   32/32
     INFO AO runtime avg sfcver                     91  seconds   3571/39
     INFO AO runtime avg sfcqcmsg                   17  seconds   863/50
     INFO AO runtime avg sfcqcout meso              59  seconds   8427/141
     INFO AO runtime avg sfcqcout_late meso        167  seconds   23157/138
     INFO AO runtime avg FSL_qcout5m_               74  seconds   11131/150
     INFO AO runtime avg FSL_analrun15m_           164  seconds   8242/50
     PASS IQC Throughput                           1  -- check: #"conflicts" <30 delayed %Delayed: 1% #Sample: 751 #Delayed: 12 Competed timely:  680
     INFO IQC runtime avg qcstg1_2                   7  seconds   102329/14321
     INFO IQC runtime avg qcstg1_2 meso             34  seconds   50236/1476
     INFO IQC runtime avg qcstg1_2_late meso        33  seconds   38022/1135
     INFO IQC runtime avg sfcing                    45  seconds   33317/738
     INFO IQC runtime avg sfchqc                    37  seconds   28030/738
     PASS QC applied max value                     Max applied check should be more than 107, 123 is good:      123,   // temperatureQCA(99999)
     PASS RT Throughput                            157 secs -- check: Avg time < 180 sec. #Hrs: 12 Max secs: 164
     PASS '0' length data Station table            check: cnt '0' length files  #: 0
     PASS '0' length master Station table          check: cnt '0' length files  #: 0
     PASS netcdf-4.1 lib                           NetCDF lib link not in routerStoreNetCDF
     WARN Shefdecoder issues.  Missing stations in either hydro and mesonet:   check 20260813_1100: #CPYRMA: 0 #NJWXNET: 659
     PASS QC DEW POINT DEPRESSION (K)              4.84 % OBS FAILED  CHECK (  2453 OF  50685) AVGTHR 5.2669 RATIO 0.287 AVGEPS 0.1463E-01 FRMSE 21.686 TRMSE 1.5098
     PASS QC POTENTIAL TEMPERATURE (K)             5.05 % OBS FAILED  CHECK (  2701 OF  53492) AVGTHR 3.6501 RATIO 0.317 AVGEPS 0.2460E-01 FRMSE 13.223 TRMSE 1.1580
     PASS QC DEW POINT TEMPERATURE (K)             5.40 % OBS FAILED  CHECK (  2737 OF  50696) AVGTHR 3.9264 RATIO 0.228 AVGEPS 0.1534E-01 FRMSE 22.616 TRMSE 0.89353
     PASS QC U COMPONENT OF SFC WIND (M/S)         0.69 % OBS FAILED  CHECK (   354 OF  51666) AVGTHR 4.4171 RATIO 0.251 AVGEPS 0.1400E-01 FRMSE 7.4889 TRMSE 1.1104
     PASS QC V COMPONENT OF SFC WIND (M/S)         0.61 % OBS FAILED  CHECK (   315 OF  51666) AVGTHR 4.5451 RATIO 0.262 AVGEPS 0.1418E-01 FRMSE 7.7196 TRMSE 1.1908
     PASS QC 3H ALTIMETER CHANGE (MB)              0.36 % OBS FAILED  CHECK (    76 OF  21161) AVGTHR 1.3173 RATIO 0.197 AVGEPS 0.1822E-01 FRMSE 7.0928 TRMSE 0.25937
     PASS QC NWS SEA-LEVEL PRESSURE (MB)           3.86 % OBS FAILED  CHECK (    45 OF   1167) AVGTHR 1.5206 RATIO 0.347 AVGEPS 0.5368E-01 FRMSE 3.2183 TRMSE 0.52803
     PASS QC ALTIMETER (MB)                       21.11 % OBS FAILED  CHECK (  4833 OF  22896) AVGTHR 1.9370 RATIO 0.100E+06 AVGEPS 0.2624E-01 FRMSE 38.142 TRMSE 99999.
     PASS MXST_P limit                             328440    MXST_P: 500000
     PASS AO Possible Seg fault                    0  #logs checked: 2277  
     PASS IQC Possible Seg fault                   0  #logs checked: 5162  
     PASS NONSFC Possible Seg fault                0  #logs checked: 241  
     PASS MADIS DB Connection ok                    madis/proc/storage/log/MadisDbUpload/MadisDataUpload_2026.08.13.12.06.19_00000.log
     INFO MADIS DB Connection cnts                 [2026-08-13 12:22:32.745773]: LastUploadStats:mode:upload_sfc,pid:1445232,aborted:0,fileCount:1,wholeFileErrors:0,lines:26320,inserts:23463,updates:2088,duplicates:1,errors:1748,stationsAdded:0,stationsUpdated:66,publishLines:0,publishErrors:0,notes:./202608131221_receipt.sfcd2.txt,run ms:37064
     PASS AFWS files current                       # in last hour: 20
     PASS madis.noaa.gov.stations current             # updated in last day: 2
     PASS madis.noaa.gov.trans current             # updated in last day: 147
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Today Day: 26225
                                     madis/proc/sfc/ao/output/sfcncdf.out.262250000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262250100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262250200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262250300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262250400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262250500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262250600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262250700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262250800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262250900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262251000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262251100_4:
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Prior Day: 26224
                                     madis/proc/sfc/ao/output/sfcncdf.out.262231400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262231500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262231600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262231700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262231800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262231900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262232000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262232100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262232200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262232300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262240000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262240100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262240200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262240300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262240400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262240500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262240600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262240700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262240800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262240900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262241000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262241100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262241200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262241300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262241400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262241500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262241600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262241700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262241800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262241900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262242000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262242100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262242200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262242300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262250000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262250100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262250200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262250300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262250400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262250500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262250600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262250700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262250800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262250900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262251000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.262251100_4:
     PASS SFCANL MATRIX FAILURE                     # Today Day: 26225
     WARN SFCANL MATRIX FAILURE                     # Prior Day: 26224
                                     Binary
                                     madis/proc/sfc/ao/output/sfcanl.out.262240200_4:

  COMPARE Mesonet data providers cnt                45     for 20260813_1000:
     PASS AO SEG FAULTS sfc/ao/logs
     PASS IQC SEG FAULTS sfc/iqc*/logs
     PASS NONSFC SEG FAULTS nonsfc/logs
     PASS AIRCRAFT WEB GOV DATA FILE EXISTS /common/data/apps/madis/current/private/aircraft/gov/datacube/DataCubeReadOnly/Aircraft/2026/08/13/Aircraft.2026.08.13_12.mv.db
     PASS AIRCRAFT WEB GOV DATA FILE EXISTS /common/data/apps/madis/current/private/aircraft/public/datacube/DataCubeReadOnly/Aircraft/2026/08/13/Aircraft.2026.08.13_12.mv.db
  COMPARE DECODE STN TBLS                           149        Newest:   234216 Aug 13 01:50 LDAD/data/RAWSStation.txt  Oldest:     1568 Oct  2  2025 LDAD/data/AWS-KSNetStation.txt
  COMPARE PRE-DECODE STN TBLS                       149        Newest:   234216 Aug 13 01:50 LDAD/data/master/RAWSStation.txt  Oldest:     1568 Oct  2  2025 LDAD/data/master/AWS-KSNetStation.txt
  COMPARE pqsend entries                            6           (Only on primary, 1 for each feedtype/distr)
  COMPARE Surface QC text products                  7320         hour (11): Newest: 4 Aug 13 12:27 madis/proc/sfc/ao/qc122/qcsumry12.dat Oldest:
  COMPARE sda IO Writes 2622510                     Local Disk MB/s: 4 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3445+
  COMPARE sda IO Reads(2622510                      Local Disk MB/s: 0 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3445+
  COMPARE NetCDF REC Writes 2622510                 Avg DECODE rec/s: 97 Avg AO rec/s: 102 Avg NON-SFC rec/s: 1 Tot secs: 3445
     INFO Network  2622510                          RX 0 MB/s TX    0 MB/s RX     Tot secs: 3445  
  COMPARE opt Disk 2622510                          0 WRITE MB/s | 0 READ MB/s | 0 Avg WRT TPS | 0 Max 1-min WRT TPS

  COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size.  Note: Max #recNum has to be < MXST_P 500000
AO-20260813_10  sfc/ao/qcms_data/buoy/20260813_1000                               5141    6027932
AO-20260813_10  sfc/ao/qcms_data/crn/20260813_1000                                 116     247312
AO-20260813_10  sfc/ao/qcms_data/hcn/20260813_1000                                  16      90108
AO-20260813_10  sfc/ao/qcms_data/hfmetar/20260813_1000                           11921   18762348
AO-20260813_10  sfc/ao/qcms_data/hydro/20260813_1000                             17307   22163884
AO-20260813_10  sfc/ao/qcms_data/mesonet/20260813_1000                          299163  661838896
AO-20260813_10  sfc/ao/qcms_data/metar/20260813_1000                              8221   12297124
AO-20260813_10  sfc/ao/qcms_data/nepp/20260813_1000                                216     386560
AO-20260813_10  sfc/ao/qcms_data/rwis/20260813_1000                               6338   25961668
AO-20260813_10  sfc/ao/qcms_data/urbanet/20260813_1000                            3541    9776652

NS-20260813_10  nonsfc/madis_data/LDAD/profiler/netCDF/20260813_1000                36    1854168
NS-20260813_10  nonsfc/madis_data/LDAD/profiler/netCDF/20260813_1030                29    1501760
NS-20260813_10  nonsfc/madis_data/LDAD/snow/netCDF/20260813_1000                  3630    3924876

AO-20260813_09  sfc/ao/qcms_data/buoy/20260813_0900                               4770    5603508
AO-20260813_09  sfc/ao/qcms_data/crn/20260813_0900                                 116     247312
AO-20260813_09  sfc/ao/qcms_data/hcn/20260813_0900                                  16      90108
AO-20260813_09  sfc/ao/qcms_data/hfmetar/20260813_0900                           11721   18451148
AO-20260813_09  sfc/ao/qcms_data/hydro/20260813_0900                             16727   21465564
AO-20260813_09  sfc/ao/qcms_data/mesonet/20260813_0900                          300467  664712912
AO-20260813_09  sfc/ao/qcms_data/metar/20260813_0900                              8218   12292708
AO-20260813_09  sfc/ao/qcms_data/nepp/20260813_0900                                216     386560
AO-20260813_09  sfc/ao/qcms_data/rwis/20260813_0900                               6263   25668568
AO-20260813_09  sfc/ao/qcms_data/urbanet/20260813_0900                            3540    9773992

NS-20260813_09  nonsfc/madis_data/LDAD/profiler/netCDF/20260813_0900                36    1854168
NS-20260813_09  nonsfc/madis_data/LDAD/profiler/netCDF/20260813_0930                29    1501760
NS-20260813_09  nonsfc/madis_data/LDAD/snow/netCDF/20260813_0900                  3836    4131700

DECODE          decode/data/fxa/LDAD/crn/netCDF/20260813_1000                      116     182808
DECODE          decode/data/fxa/LDAD/hcn/netCDF/20260813_1000                       16      67088
DECODE          decode/data/fxa/LDAD/hfmetar/netCDF/20260813_1000                11921   13208932
DECODE          decode/data/fxa/LDAD/hydro/netCDF/20260813_1000                  17329   14575380
DECODE          decode/data/fxa/LDAD/mesonet/netCDF/20260813_1000               301407  372802864
DECODE          decode/data/fxa/LDAD/nepp/netCDF/20260813_1000                     216     229908
DECODE          decode/data/fxa/LDAD/rwis/netCDF/20260813_1000                    6338   20183128
DECODE          decode/data/fxa/LDAD/snow/netCDF/20260813_1000                    3633    2813184
DECODE          decode/data/fxa/LDAD/urbanet/netCDF/20260813_1000                 3541    3302844