MADIS RT RUN-TIME STATUS for Fri Jan 17 12:32:01 GMT 2025 on vm-bldr-madis-ops-proc1
     INFO RT VM STATUS for Node  vm-bldr-madis-ops-proc1 Time Fri Jan 17 12:32:02 GMT 2025

     PASS METAR processing tbl                     file:  179538 Dec 31 17:30 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
     PASS Station update flag                      file:  0 May 20  2024 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
     FAIL SNOTEL files missing:    Last updates:  hourly<120min  0  daily<1800min  108  monthly<45000min  52 
     PASS Metar station table                      file:  1121683 May 20  2024 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
     PASS Accept/reject|autorej files              Cnts: AO: 3 IQC: 3 NON-SFC: 20
     PASS Persistent processes                     Cnts: pqact (1+): 1 ldmd (1+): 21 DECODE (5): 5
     PASS Dcmetr files                             CNT-LDAD.raw: 770
     PASS RSAS values range check                  Total chk: 317840 Outside of range (<100): 0
     WARN Aircraft data late        Check: REC-CNT>20   #TAMDAR 20250117_1100:65 20250117_1000:74  #PUBLIC-DP 20250117_1100:1548 20250117_1000:1631  #PUBLIC-DP-profile 20250117_1100:14 20250117_1000:16      
     WARN Aircraft data late        Check: REC-CNT>20   #TAMDAR 20250117_1100:65 20250117_1000:74  #PUBLIC-DP 20250117_1100:1548 20250117_1000:1631  #PUBLIC-DP-profile 20250117_1100:14 20250117_1000:16      
     PASS Load resource                 m15 < 9 cores 100% (goal > 60%)   -- % of the time M15 < 9  Samples: 752    #10=+ 0 #9 0 #8 10 #7 90
     PASS Open file resource                       check: open file < 5000 Max Files: -1 #<1000: 752 #<10000 752
     PASS Process count                            check: processes < 1000 Max Processes: 466
     PASS AO Throughput                            0 -- check: #"conflicts" <20 delayed %Delayed: 0% #Sample: 300 #Delayed: 0
     INFO AO runtime avg sfcanl                     69  seconds   3471/50
     INFO AO runtime avg sfcncdf                     1  seconds   25/25
     INFO AO runtime avg sfcver                    108  seconds   4002/37
     INFO AO runtime avg sfcqcmsg                   10  seconds   522/50
     INFO AO runtime avg sfcqcout meso              57  seconds   8107/142
     INFO AO runtime avg sfcqcout_late meso        155  seconds   21527/138
     INFO AO runtime avg FSL_qcout5m_               66  seconds   9956/150
     INFO AO runtime avg FSL_analrun15m_           163  seconds   8191/50
     PASS IQC Throughput                           0  -- check: #"conflicts" <30 delayed %Delayed: 0% #Sample: 750 #Delayed: 0 Competed timely:  690
     INFO IQC runtime avg qcstg1_2                   6  seconds   82978/13781
     INFO IQC runtime avg qcstg1_2 meso             21  seconds   32996/1500
     INFO IQC runtime avg qcstg1_2_late meso        32  seconds   37210/1155
     INFO IQC runtime avg sfcing                    45  seconds   33966/750
     INFO IQC runtime avg sfchqc                    35  seconds   26418/750
     PASS QC applied max value                     Max applied check should be more than 107, 123 is good:      123,   // temperatureQCA(69154)
     PASS RT Throughput                            144 secs -- check: Avg time < 180 sec. #Hrs: 12 Max secs: 155
     PASS '0' length data Station table            check: cnt '0' length files  #: 0
     PASS '0' length master Station table          check: cnt '0' length files  #: 0
     PASS netcdf-4.1 lib                           NetCDF lib link not in routerStoreNetCDF
     WARN Shefdecoder issues.  Missing stations in either hydro and mesonet:   check 20250117_1100: #CPYRMA: 0 #NJWXNET: 658
     PASS QC DEW POINT DEPRESSION (K)              5.03 % OBS FAILED  CHECK (  2413 OF  48017) AVGTHR 5.3054 RATIO 0.369 AVGEPS 0.1640E-01 FRMSE 18.872 TRMSE 1.9573
     PASS QC POTENTIAL TEMPERATURE (K)             9.40 % OBS FAILED  CHECK (  4843 OF  51517) AVGTHR 3.6466 RATIO 0.00 AVGEPS 0.2510E-01 FRMSE 1060.0 TRMSE 0.0000
     PASS QC DEW POINT TEMPERATURE (K)             5.55 % OBS FAILED  CHECK (  2665 OF  48017) AVGTHR 3.9601 RATIO 0.324 AVGEPS 0.1748E-01 FRMSE 19.571 TRMSE 1.2823
     PASS QC U COMPONENT OF SFC WIND (M/S)         0.62 % OBS FAILED  CHECK (   286 OF  46374) AVGTHR 4.4143 RATIO 0.248 AVGEPS 0.1390E-01 FRMSE 10.692 TRMSE 1.0963
     PASS QC V COMPONENT OF SFC WIND (M/S)         0.68 % OBS FAILED  CHECK (   314 OF  46374) AVGTHR 4.5385 RATIO 0.239 AVGEPS 0.1392E-01 FRMSE 10.580 TRMSE 1.0862
     PASS QC 3H ALTIMETER CHANGE (MB)              0.92 % OBS FAILED  CHECK (   184 OF  20016) AVGTHR 1.2565 RATIO 0.235 AVGEPS 0.1802E-01 FRMSE 21.620 TRMSE 0.29480
     PASS QC NWS SEA-LEVEL PRESSURE (MB)           9.03 % OBS FAILED  CHECK (    67 OF    742) AVGTHR 1.5614 RATIO 0.364 AVGEPS 0.5904E-01 FRMSE 3.3829 TRMSE 0.56783
     PASS QC ALTIMETER (MB)                       19.17 % OBS FAILED  CHECK (  4199 OF  21900) AVGTHR 1.9546 RATIO 0.100E+06 AVGEPS 0.2975E-01 FRMSE 39.306 TRMSE 99999.
     WARN MXST_P limit reach. Increase next RFC Using: 330000   MXST_P: 330000 --   TOTAL SITES LOADED:       330000  MXST_P:       330000
     PASS AO Possible Seg fault                    0  #logs checked: 2268  
     PASS IQC Possible Seg fault                   0  #logs checked: 5220  
     PASS NONSFC Possible Seg fault                0  #logs checked: 203  
     PASS MADIS DB Connection ok                    madis/proc/storage/log/MadisDbUpload/MadisDataUpload_2025.01.17.12.18.22_00000.log
     INFO MADIS DB Connection cnts                 [2025-01-17 12:21:10.922557]: LastUploadStats:mode:upload_sfc,pid:2339231,aborted:0,fileCount:1,wholeFileErrors:0,lines:17366,inserts:14533,updates:2083,duplicates:1,errors:1724,stationsAdded:0,stationsUpdated:28,publishLines:0,publishErrors:0,notes:./202501171220_receipt.sfcd2.txt,run ms:19836
     PASS AFWS files current                       # in last hour: 20
     PASS madis.noaa.gov.stations current             # updated in last day: 2
     PASS madis.noaa.gov.trans current             # updated in last day: 147
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Today Day: 25017
                                     madis/proc/sfc/ao/output/sfcncdf.out.250170000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250170100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250170200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250170300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250170400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250170500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250170600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250170700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250170800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250170900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250171000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250171100_4:
     WARN MSAS ANAL PRODUCT -- SFCNCDF              # Prior Day: 25016
                                     madis/proc/sfc/ao/output/sfcncdf.out.250151400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250151500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250151600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250151700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250151800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250151900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250152000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250152100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250152200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250152300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250160000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250160100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250160200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250160300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250160400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250160500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250160600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250160700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250160800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250160900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250161000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250161100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250161200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250161300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250161400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250161500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250161600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250161700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250161800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250161900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250162000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250162100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250162200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250162300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250170000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250170100_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250170200_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250170300_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250170400_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250170500_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250170600_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250170700_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250170800_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250170900_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250171000_4:
                                     madis/proc/sfc/ao/output/sfcncdf.out.250171100_4:
     WARN SFCANL MATRIX FAILURE                     # Today Day: 25017
                                     Binary
     WARN SFCANL MATRIX FAILURE                     # Prior Day: 25016
                                     Binary

  COMPARE Mesonet data providers cnt                49     for 20250117_1000:
     PASS AO SEG FAULTS sfc/ao/logs
     PASS IQC SEG FAULTS sfc/iqc*/logs
     PASS NONSFC SEG FAULTS nonsfc/logs
     PASS AIRCRAFT WEB GOV DATA FILE EXISTS /common/data/apps/madis/current/private/aircraft/gov/datacube/DataCubeReadOnly/Aircraft/2025/01/17/Aircraft.2025.01.17_12.mv.db
     PASS AIRCRAFT WEB GOV DATA FILE EXISTS /common/data/apps/madis/current/private/aircraft/public/datacube/DataCubeReadOnly/Aircraft/2025/01/17/Aircraft.2025.01.17_12.mv.db
  COMPARE DECODE STN TBLS                           149        Newest:   219114 Jan 17 01:51 LDAD/data/RAWSStation.txt  Oldest:     8848 Jul 18  2024 LDAD/data/AWS-GANetStation.txt
  COMPARE PRE-DECODE STN TBLS                       149        Newest:   219114 Jan 17 01:51 LDAD/data/master/RAWSStation.txt  Oldest:     1568 Jul 18  2024 LDAD/data/master/AWS-KSNetStation.txt
  COMPARE pqsend entries                            6           (Only on primary, 1 for each feedtype/distr)
  COMPARE Surface QC text products                  7320         hour (11): Newest: 1 Jan 17 12:27 madis/proc/sfc/ao/qc122/qcsumry12.dat Oldest:
  COMPARE sda IO Writes 2501710                     Local Disk MB/s: 0 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3460+
  COMPARE sda IO Reads(2501710                      Local Disk MB/s: 0 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3460+
  COMPARE NetCDF REC Writes 2501710                 Avg DECODE rec/s: 95 Avg AO rec/s: 101 Avg NON-SFC rec/s: 0 Tot secs: 3460
     INFO Network  2501710                          RX 0 MB/s TX    0 MB/s RX     Tot secs: 3460  
  COMPARE opt Disk 2501710                          0 WRITE MB/s | 0 READ MB/s | 0 Avg WRT TPS | 0 Max 1-min WRT TPS

  COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size.  Note: Max #recNum has to be < MXST_P 330000
AO-20250117_10  sfc/ao/qcms_data/buoy/20250117_1000                               4488    5280900
AO-20250117_10  sfc/ao/qcms_data/crn/20250117_1000                                 247     437524
AO-20250117_10  sfc/ao/qcms_data/hcn/20250117_1000                                  27     104144
AO-20250117_10  sfc/ao/qcms_data/hfmetar/20250117_1000                           11752   18499384
AO-20250117_10  sfc/ao/qcms_data/hydro/20250117_1000                             15593   20100228
AO-20250117_10  sfc/ao/qcms_data/mesonet/20250117_1000                          300225  664179544
AO-20250117_10  sfc/ao/qcms_data/metar/20250117_1000                              8073   12079268
AO-20250117_10  sfc/ao/qcms_data/nepp/20250117_1000                                 48     142624
AO-20250117_10  sfc/ao/qcms_data/rwis/20250117_1000                               6109   25066736
AO-20250117_10  sfc/ao/qcms_data/urbanet/20250117_1000                            4502   12332912

NS-20250117_10  nonsfc/madis_data/LDAD/profiler/netCDF/20250117_1000                31    1602448
NS-20250117_10  nonsfc/madis_data/LDAD/profiler/netCDF/20250117_1030                21    1099008
NS-20250117_10  nonsfc/madis_data/LDAD/snow/netCDF/20250117_1000                  2908    3199988

AO-20250117_09  sfc/ao/qcms_data/buoy/20250117_0900                               4256    5015492
AO-20250117_09  sfc/ao/qcms_data/crn/20250117_0900                                 247     437524
AO-20250117_09  sfc/ao/qcms_data/hcn/20250117_0900                                  27     104144
AO-20250117_09  sfc/ao/qcms_data/hfmetar/20250117_0900                           11852   18654984
AO-20250117_09  sfc/ao/qcms_data/hydro/20250117_0900                             15697   20225444
AO-20250117_09  sfc/ao/qcms_data/mesonet/20250117_0900                          303206  670749668
AO-20250117_09  sfc/ao/qcms_data/metar/20250117_0900                              8065   12067492
AO-20250117_09  sfc/ao/qcms_data/nepp/20250117_0900                                 48     142624
AO-20250117_09  sfc/ao/qcms_data/rwis/20250117_0900                               6292   25781900
AO-20250117_09  sfc/ao/qcms_data/urbanet/20250117_0900                            4479   12271732

NS-20250117_09  nonsfc/madis_data/LDAD/profiler/netCDF/20250117_0900                31    1602448
NS-20250117_09  nonsfc/madis_data/LDAD/profiler/netCDF/20250117_0930                21    1099008
NS-20250117_09  nonsfc/madis_data/LDAD/snow/netCDF/20250117_0900                  2891    3182920
NS-20250117_09  nonsfc/madis_data/point/raob/netcdf/20250117_0900                    1     180508

DECODE          decode/data/fxa/LDAD/crn/netCDF/20250117_1000                      247     317476
DECODE          decode/data/fxa/LDAD/hcn/netCDF/20250117_1000                       27      76372
DECODE          decode/data/fxa/LDAD/hfmetar/netCDF/20250117_1000                11752   13024384
DECODE          decode/data/fxa/LDAD/hydro/netCDF/20250117_1000                  15604   13222980
DECODE          decode/data/fxa/LDAD/mesonet/netCDF/20250117_1000               301703  373167536
DECODE          decode/data/fxa/LDAD/nepp/netCDF/20250117_1000                      48      92148
DECODE          decode/data/fxa/LDAD/rwis/netCDF/20250117_1000                    6157   19639404
DECODE          decode/data/fxa/LDAD/snow/netCDF/20250117_1000                    2910    2307084
DECODE          decode/data/fxa/LDAD/urbanet/netCDF/20250117_1000                 4550    4162512