MADIS RT RUN-TIME STATUS for Wed Mar 29 03:32:02 GMT 2023 on vm-bldr-madis-process1a
INFO RT VM STATUS for Node vm-bldr-madis-process1a Time Wed Mar 29 03:32:02 GMT 2023
PASS METAR processing tbl file: 128320 Mar 21 21:03 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
PASS Station update flag file: 0 May 3 2019 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
PASS SNOTEL files Last updates: hourly<120min 102 daily<1800min 102 monthly<45000min 52
PASS Metar station table file: 1121683 May 3 2019 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
PASS Accept/reject|autorej files Cnts: AO: 3 IQC: 3 NON-SFC: 20
PASS Persistent processes Cnts: pqact (1+): 1 ldmd (1+): 14 DECODE (5): 5
PASS Dcmetr files CNT-LDAD.raw: 807
PASS RSAS values range check Total chk: 317840 Outside of range (<100): 0
PASS Aircraft data Check: REC-CNT>300 #TAMDAR 20230329_0200:840 20230329_0100:898 #PUBLIC-DP 20230329_0200:2057 20230329_0100:2178 #PUBLIC-DP-profile 20230329_0200:50 20230329_0100:45
PASS Load resource m15 < 9 cores 98% (goal > 60%) -- % of the time M15 < 9 Samples: 212 #10=+ 0 #9 3 #8 32 #7 92
PASS Open file resource check: open file < 5000 Max Files: 2037 #<1000: 73 #<10000 212
PASS Process count check: processes < 1000 Max Processes: 466
PASS AO Throughput 0 -- check: #"conflicts" <20 delayed %Delayed: 0% #Sample: 42 #Delayed: 0
INFO AO runtime avg sfcanl 14 seconds 203/14
INFO AO runtime avg sfcncdf 1 seconds 13/12
INFO AO runtime avg sfcver 74 seconds 1042/14
INFO AO runtime avg sfcqcmsg 20 seconds 292/14
INFO AO runtime avg sfcqcout meso 32 seconds 1249/39
INFO AO runtime avg sfcqcout_late meso 82 seconds 3047/37
INFO AO runtime avg FSL_qcout5m_ 42 seconds 1842/43
INFO AO runtime avg FSL_analrun15m_ 114 seconds 1601/14
PASS IQC Throughput 0 -- check: #"conflicts" <30 delayed %Delayed: 0% #Sample: 215 #Delayed: 2 Competed timely: 149
INFO IQC runtime avg qcstg1_2 4 seconds 16985/3398
INFO IQC runtime avg qcstg1_2 meso 19 seconds 8309/420
INFO IQC runtime avg qcstg1_2_late meso 21 seconds 4937/229
INFO IQC runtime avg sfcing 29 seconds 6137/208
INFO IQC runtime avg sfchqc 62 seconds 13006/208
PASS QC applied max value Max applied check should be more than 107, 123 is good: 123, // temperatureQCA(99999)
FAIL RT Throughput time too slow 685761 secs -- check: Avg time < 180 sec. #Hrs: 3 Max secs: 2514408
PASS '0' length data Station table check: cnt '0' length files #: 0
PASS '0' length master Station table check: cnt '0' length files #: 0
PASS netcdf-4.1 lib NetCDF lib link not in routerStoreNetCDF
WARN Shefdecoder issues. Missing stations in either hydro and mesonet: check 20230329_0200: #CPYRMA: 0 #NJWXNET: 636
WARN QC MAPS SEA-LEVEL PRESSURE (MB) 25.14 % OBS FAILED CHECK ( 8157 OF 32445) AVGTHR 1.7924 RATIO 0.100E+06 AVGEPS 0.2498E-01 FRMSE 42.015 TRMSE 99999.
PASS QC DEW POINT DEPRESSION (K) 4.96 % OBS FAILED CHECK ( 3226 OF 65072) AVGTHR 5.2688 RATIO 0.332 AVGEPS 0.1462E-01 FRMSE 17.159 TRMSE 1.7503
PASS QC POTENTIAL TEMPERATURE (K) 3.32 % OBS FAILED CHECK ( 2287 OF 68854) AVGTHR 3.6364 RATIO 0.100E+06 AVGEPS 0.2551E-01 FRMSE 1634.6 TRMSE 99999.
PASS QC DEW POINT TEMPERATURE (K) 5.39 % OBS FAILED CHECK ( 3505 OF 65072) AVGTHR 3.9308 RATIO 0.298 AVGEPS 0.1534E-01 FRMSE 20.266 TRMSE 1.1694
PASS QC U COMPONENT OF SFC WIND (M/S) 1.47 % OBS FAILED CHECK ( 960 OF 65472) AVGTHR 4.4162 RATIO 0.338 AVGEPS 0.1413E-01 FRMSE 12.691 TRMSE 1.4918
PASS QC V COMPONENT OF SFC WIND (M/S) 1.42 % OBS FAILED CHECK ( 927 OF 65472) AVGTHR 4.5317 RATIO 0.333 AVGEPS 0.1392E-01 FRMSE 18.823 TRMSE 1.5081
PASS QC 3H ALTIMETER CHANGE (MB) 1.23 % OBS FAILED CHECK ( 357 OF 28926) AVGTHR 1.2117 RATIO 0.288 AVGEPS 0.1637E-01 FRMSE 17.765 TRMSE 0.34872
PASS QC NWS SEA-LEVEL PRESSURE (MB) 6.04 % OBS FAILED CHECK ( 41 OF 679) AVGTHR 1.5713 RATIO 0.392 AVGEPS 0.5969E-01 FRMSE 3.5776 TRMSE 0.61649
PASS QC ALTIMETER (MB) 24.25 % OBS FAILED CHECK ( 7867 OF 32445) AVGTHR 1.9293 RATIO 0.100E+06 AVGEPS 0.2503E-01 FRMSE 42.742 TRMSE 99999.
PASS MXST_P limit 270552 MXST_P: 280000
PASS AO Possible Seg fault 0 #logs checked: 2265
PASS IQC Possible Seg fault 0 #logs checked: 4582
PASS NONSFC Possible Seg fault 0 #logs checked: 759
PASS MADIS DB Connection ok madis/proc/storage/log/MadisDbUpload/MadisDataUpload_20230329_398639.log
INFO MADIS DB Connection cnts Info:486 [2023-Mar-29 03:31:20.548229] - LastUploadStats:mode:upload_sfc,pid:30771,aborted:0,fileCount:1,wholeFileErrors:0,lines:39864,inserts:31662,updates:8202,duplicates:0,errors:2,stationsAdded:0,stationsUpdated:2,publishLines:0,publishErrors:0,notes:./202303290330_receipt.sfcd2.txt,run ms:64467
PASS AFWS files current # in last hour: 20
PASS madis.noaa.gov.stations current # updated in last day: 12
PASS madis.noaa.gov.trans current # updated in last day: 24
WARN MSAS ANAL PRODUCT -- SFCNCDF # Today Day: 23088
madis/proc/sfc/ao/output/sfcncdf.out.230880000_4:
WARN MSAS ANAL PRODUCT -- SFCNCDF # Prior Day: 23087
madis/proc/sfc/ao/output/sfcncdf.out.230861200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861300_2:
madis/proc/sfc/ao/output/sfcncdf.out.230861400_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861700_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861800_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861900_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862000_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862100_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862200_2:
madis/proc/sfc/ao/output/sfcncdf.out.230862200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870500_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870900_1:
madis/proc/sfc/ao/output/sfcncdf.out.230871100_1:
madis/proc/sfc/ao/output/sfcncdf.out.230871200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871300_2:
madis/proc/sfc/ao/output/sfcncdf.out.230871300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871400_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871800_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871900_4:
madis/proc/sfc/ao/output/sfcncdf.out.230872000_4:
madis/proc/sfc/ao/output/sfcncdf.out.230872100_4:
madis/proc/sfc/ao/output/sfcncdf.out.230872200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230880000_4:
WARN SFCANL MATRIX FAILURE # Today Day: 23088
madis/proc/sfc/ao/output/sfcanl.out.230880000_4:
WARN SFCANL MATRIX FAILURE # Prior Day: 23087
madis/proc/sfc/ao/output/sfcanl.out.230870900_1:
madis/proc/sfc/ao/output/sfcanl.out.230871100_1:
madis/proc/sfc/ao/output/sfcanl.out.230871300_4:
COMPARE Mesonet data providers cnt 48 for 20230329_0100:
COMPARE DECODE STN TBLS 149 Newest: 214481 Mar 29 02:44 LDAD/data/RAWSStation.txt Oldest: 8848 Sep 12 2019 LDAD/data/AWS-GANetStation.txt
COMPARE PRE-DECODE STN TBLS 149 Newest: 214481 Mar 29 02:44 LDAD/data/master/RAWSStation.txt Oldest: 1568 Sep 12 2019 LDAD/data/master/AWS-KSNetStation.txt
COMPARE pqsend entries 6 (Only on primary, 1 for each feedtype/distr)
COMPARE Surface QC text products 7320 hour (02): Newest: Mar 29 03:27 madis/proc/sfc/ao/qc122/qcsumry03.dat Oldest:
COMPARE sda IO Writes 2308801 Local Disk MB/s: 25 RS MB/s: 0 PQ MB/s: 28+ Tot secs: 3455+
COMPARE sda IO Reads(2308801 Local Disk MB/s: 1 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3455+
COMPARE NetCDF REC Writes 2308801 Avg DECODE rec/s: 85 Avg AO rec/s: 83 Avg NON-SFC rec/s: 13 Tot secs: 3455
INFO Network 2308801 eth0 15 MB/s TX 3 MB/s RX Tot secs: 3455
COMPARE usr1 Disk 2308801 25 WRITE MB/s | 1 READ MB/s | 6113 Avg WRT TPS | 15525 Max 1-min WRT TPS
COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size. Note: Max #recNum has to be < MXST_P 280000
AO-20230329_01 sfc/ao/qcms_data/buoy/20230329_0100 839 1099348
AO-20230329_01 sfc/ao/qcms_data/crn/20230329_0100 132 270544
AO-20230329_01 sfc/ao/qcms_data/hcn/20230329_0100 96 192188
AO-20230329_01 sfc/ao/qcms_data/hfmetar/20230329_0100 12010 18900832
AO-20230329_01 sfc/ao/qcms_data/hydro/20230329_0100 9353 11778444
AO-20230329_01 sfc/ao/qcms_data/mesonet/20230329_0100 249170 548700600
AO-20230329_01 sfc/ao/qcms_data/metar/20230329_0100 7561 11264756
AO-20230329_01 sfc/ao/qcms_data/nepp/20230329_0100 72 177472
AO-20230329_01 sfc/ao/qcms_data/rwis/20230329_0100 5778 20659008
AO-20230329_01 sfc/ao/qcms_data/urbanet/20230329_0100 5955 16197892
NS-20230329_01 nonsfc/madis_data/LDAD/profiler/netCDF/20230329_0100 44 2256920
NS-20230329_01 nonsfc/madis_data/LDAD/profiler/netCDF/20230329_0130 33 1703136
NS-20230329_01 nonsfc/madis_data/LDAD/snow/netCDF/20230329_0100 2285 2574496
NS-20230329_01 nonsfc/madis_data/point/acars/netcdf/20230329_0100 42158 48487768
NS-20230329_01 nonsfc/madis_data/point/acarsProfiles/netcdf/20230329_0100 329 47924536
NS-20230329_01 nonsfc/madis_data/point/raob/netcdf/20230329_0100 3 317420
AO-20230329_00 sfc/ao/qcms_data/buoy/20230329_0000 998 1279972
AO-20230329_00 sfc/ao/qcms_data/crn/20230329_0000 132 270544
AO-20230329_00 sfc/ao/qcms_data/hcn/20230329_0000 96 192188
AO-20230329_00 sfc/ao/qcms_data/hfmetar/20230329_0000 12000 18885272
AO-20230329_00 sfc/ao/qcms_data/hydro/20230329_0000 9390 11821068
AO-20230329_00 sfc/ao/qcms_data/mesonet/20230329_0000 248301 546792276
AO-20230329_00 sfc/ao/qcms_data/metar/20230329_0000 7622 11354060
AO-20230329_00 sfc/ao/qcms_data/nepp/20230329_0000 72 177472
AO-20230329_00 sfc/ao/qcms_data/rwis/20230329_0000 5775 20648868
AO-20230329_00 sfc/ao/qcms_data/urbanet/20230329_0000 5933 16139372
NS-20230329_00 nonsfc/madis_data/LDAD/profiler/netCDF/20230329_0000 45 2307264
NS-20230329_00 nonsfc/madis_data/LDAD/profiler/netCDF/20230329_0030 36 1854168
NS-20230329_00 nonsfc/madis_data/LDAD/snow/netCDF/20230329_0000 2250 2539356
NS-20230329_00 nonsfc/madis_data/point/acars/netcdf/20230329_0000 44395 51046896
NS-20230329_00 nonsfc/madis_data/point/acarsProfiles/netcdf/20230329_0000 367 53455056
NS-20230329_00 nonsfc/madis_data/point/HDW/netcdf/20230329_0000 181 1289000
NS-20230329_00 nonsfc/madis_data/point/raob/netcdf/20230329_0000 418 28726660
DECODE decode/data/fxa/LDAD/crn/netCDF/20230329_0100 245 315420
DECODE decode/data/fxa/LDAD/hcn/netCDF/20230329_0100 102 139672
DECODE decode/data/fxa/LDAD/hfmetar/netCDF/20230329_0100 12010 13306120
DECODE decode/data/fxa/LDAD/hydro/netCDF/20230329_0100 9356 7835568
DECODE decode/data/fxa/LDAD/mesonet/netCDF/20230329_0100 251112 310839424
DECODE decode/data/fxa/LDAD/nepp/netCDF/20230329_0100 72 111828
DECODE decode/data/fxa/LDAD/rwis/netCDF/20230329_0100 5809 15463288
DECODE decode/data/fxa/LDAD/snow/netCDF/20230329_0100 2285 1869584
DECODE decode/data/fxa/LDAD/urbanet/netCDF/20230329_0100 5955 5359572
IQC sfc/iqc1/qcms_data/buoy/20230329_0100 613 842612
IQC sfc/iqc1/qcms_data/crn/20230329_0100 120 253120
IQC sfc/iqc1/qcms_data/hcn/20230329_0100 0 69692
IQC sfc/iqc1/qcms_data/hfmetar/20230329_0100 3534 5712176
IQC sfc/iqc1/qcms_data/hydro/20230329_0100 3529 5069196
IQC sfc/iqc1/qcms_data/mesonet/20230329_0100 87335 193310940
IQC sfc/iqc1/qcms_data/metar/20230329_0100 3576 5430716
IQC sfc/iqc1/qcms_data/nepp/20230329_0100 0 72928
IQC sfc/iqc1/qcms_data/rwis/20230329_0100 1151 5019748
IQC sfc/iqc1/qcms_data/urbanet/20230329_0100 1240 3655992
IQC sfc/iqc2/qcms_data/buoy/20230329_0100 4 150788
IQC sfc/iqc2/qcms_data/crn/20230329_0100 0 78880
IQC sfc/iqc2/qcms_data/hcn/20230329_0100 0 69692
IQC sfc/iqc2/qcms_data/hfmetar/20230329_0100 2842 4635424
IQC sfc/iqc2/qcms_data/hydro/20230329_0100 1832 3114252
IQC sfc/iqc2/qcms_data/mesonet/20230329_0100 62475 138718380
IQC sfc/iqc2/qcms_data/metar/20230329_0100 1460 2332892
IQC sfc/iqc2/qcms_data/nepp/20230329_0100 0 72928
IQC sfc/iqc2/qcms_data/rwis/20230329_0100 2639 10049188
IQC sfc/iqc2/qcms_data/urbanet/20230329_0100 507 1706212
IQC sfc/iqc3/qcms_data/buoy/20230329_0100 52 205316
IQC sfc/iqc3/qcms_data/crn/20230329_0100 0 78880
IQC sfc/iqc3/qcms_data/hcn/20230329_0100 72 161564
IQC sfc/iqc3/qcms_data/hfmetar/20230329_0100 2571 4213748
IQC sfc/iqc3/qcms_data/hydro/20230329_0100 1630 2881548
IQC sfc/iqc3/qcms_data/mesonet/20230329_0100 36153 80915268
IQC sfc/iqc3/qcms_data/metar/20230329_0100 1286 2078156
IQC sfc/iqc3/qcms_data/nepp/20230329_0100 0 72928
IQC sfc/iqc3/qcms_data/rwis/20230329_0100 1078 4773008
IQC sfc/iqc3/qcms_data/urbanet/20230329_0100 2209 6233532
IQC sfc/iqc4/qcms_data/buoy/20230329_0100 146 312100
IQC sfc/iqc4/qcms_data/crn/20230329_0100 12 96304
IQC sfc/iqc4/qcms_data/hcn/20230329_0100 84 176876
IQC sfc/iqc4/qcms_data/hfmetar/20230329_0100 2535 4157732
IQC sfc/iqc4/qcms_data/hydro/20230329_0100 2501 3884940
IQC sfc/iqc4/qcms_data/mesonet/20230329_0100 46108 102776448
IQC sfc/iqc4/qcms_data/metar/20230329_0100 1121 1836596
IQC sfc/iqc4/qcms_data/nepp/20230329_0100 0 72928
IQC sfc/iqc4/qcms_data/rwis/20230329_0100 239 1937188
IQC sfc/iqc4/qcms_data/urbanet/20230329_0100 2608 7294872
IQC sfc/iqc5/qcms_data/buoy/20230329_0100 106 266660
IQC sfc/iqc5/qcms_data/crn/20230329_0100 12 96304
IQC sfc/iqc5/qcms_data/hcn/20230329_0100 0 69692
IQC sfc/iqc5/qcms_data/hfmetar/20230329_0100 2329 3837196
IQC sfc/iqc5/qcms_data/hydro/20230329_0100 2130 3457548
IQC sfc/iqc5/qcms_data/mesonet/20230329_0100 47613 106081428
IQC sfc/iqc5/qcms_data/metar/20230329_0100 970 1615532
IQC sfc/iqc5/qcms_data/nepp/20230329_0100 72 177472
IQC sfc/iqc5/qcms_data/rwis/20230329_0100 1375 5776868
IQC sfc/iqc5/qcms_data/urbanet/20230329_0100 1213 3584172