MADIS RT RUN-TIME STATUS for Wed Mar 29 00:32:01 GMT 2023 on vm-bldr-madis-process1a
INFO RT VM STATUS for Node vm-bldr-madis-process1a Time Wed Mar 29 00:32:01 GMT 2023
PASS METAR processing tbl file: 128320 Mar 21 21:03 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
PASS Station update flag file: 0 May 3 2019 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
PASS SNOTEL files Last updates: hourly<120min 102 daily<1800min 102 monthly<45000min 52
PASS Metar station table file: 1121683 May 3 2019 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
PASS Accept/reject|autorej files Cnts: AO: 3 IQC: 3 NON-SFC: 20
PASS Persistent processes Cnts: pqact (1+): 1 ldmd (1+): 16 DECODE (5): 5
PASS Dcmetr files CNT-LDAD.raw: 809
PASS RSAS values range check Total chk: 317840 Outside of range (<100): 0
PASS Aircraft data Check: REC-CNT>300 #TAMDAR 20230328_2300:806 20230328_2200:639 #PUBLIC-DP 20230328_2300:2035 20230328_2200:2028 #PUBLIC-DP-profile 20230328_2300:44 20230328_2200:39
PASS Load resource m15 < 9 cores 90% (goal > 60%) -- % of the time M15 < 9 Samples: 32 #10=+ 0 #9 3 #8 15 #7 14
PASS Open file resource check: open file < 5000 Max Files: 1690 #<1000: 9 #<10000 32
PASS Process count check: processes < 1000 Max Processes: 443
PASS AO Throughput 0 -- check: #"conflicts" <20 delayed %Delayed: 0% #Sample: 6 #Delayed: 0
INFO AO runtime avg sfcanl 14 seconds 28/2
INFO AO runtime avg sfcncdf 1 seconds 2/2
INFO AO runtime avg sfcver 74 seconds 149/2
INFO AO runtime avg sfcqcmsg 21 seconds 42/2
INFO AO runtime avg sfcqcout meso 17 seconds 104/6
INFO AO runtime avg sfcqcout_late meso -1 seconds 0/0
INFO AO runtime avg FSL_qcout5m_ 25 seconds 180/7
INFO AO runtime avg FSL_analrun15m_ 114 seconds 228/2
PASS IQC Throughput 2 -- check: #"conflicts" <30 delayed %Delayed: 2% #Sample: 35 #Delayed: 1 Competed timely: 0
INFO IQC runtime avg qcstg1_2 5 seconds 1581/309
INFO IQC runtime avg qcstg1_2 meso 19 seconds 1245/63
INFO IQC runtime avg qcstg1_2_late meso -1 seconds 0/0
INFO IQC runtime avg sfcing 28 seconds 857/30
INFO IQC runtime avg sfchqc 61 seconds 1784/29
PASS QC applied max value Max applied check should be more than 107, 123 is good: 123, // temperatureQCA(9998)
FAIL RT Throughput missing logs entries. time < 180 sec. No hours to check:
PASS '0' length data Station table check: cnt '0' length files #: 0
PASS '0' length master Station table check: cnt '0' length files #: 0
PASS netcdf-4.1 lib NetCDF lib link not in routerStoreNetCDF
WARN Shefdecoder issues. Missing stations in either hydro and mesonet: check 20230328_2300: #CPYRMA: 0 #NJWXNET: 636
WARN QC MAPS SEA-LEVEL PRESSURE (MB) 25.33 % OBS FAILED CHECK ( 8127 OF 32090) AVGTHR 1.7923 RATIO 0.100E+06 AVGEPS 0.2515E-01 FRMSE 42.881 TRMSE 99999.
PASS QC DEW POINT DEPRESSION (K) 4.75 % OBS FAILED CHECK ( 3084 OF 64991) AVGTHR 5.2736 RATIO 0.334 AVGEPS 0.1488E-01 FRMSE 17.186 TRMSE 1.7597
PASS QC POTENTIAL TEMPERATURE (K) 3.16 % OBS FAILED CHECK ( 2163 OF 68500) AVGTHR 3.6347 RATIO 0.100E+06 AVGEPS 0.2542E-01 FRMSE 1614.1 TRMSE 99999.
PASS QC DEW POINT TEMPERATURE (K) 5.47 % OBS FAILED CHECK ( 3553 OF 64993) AVGTHR 3.9284 RATIO 0.317 AVGEPS 0.1518E-01 FRMSE 19.441 TRMSE 1.2435
PASS QC U COMPONENT OF SFC WIND (M/S) 2.26 % OBS FAILED CHECK ( 1489 OF 66008) AVGTHR 4.4254 RATIO 0.384 AVGEPS 0.1474E-01 FRMSE 8.9170 TRMSE 1.7012
PASS QC V COMPONENT OF SFC WIND (M/S) 2.40 % OBS FAILED CHECK ( 1587 OF 66008) AVGTHR 4.5394 RATIO 0.376 AVGEPS 0.1439E-01 FRMSE 7.7563 TRMSE 1.7061
PASS QC 3H ALTIMETER CHANGE (MB) 1.67 % OBS FAILED CHECK ( 478 OF 28555) AVGTHR 1.2132 RATIO 0.317 AVGEPS 0.1694E-01 FRMSE 29.621 TRMSE 0.38487
PASS QC NWS SEA-LEVEL PRESSURE (MB) 6.07 % OBS FAILED CHECK ( 42 OF 692) AVGTHR 1.5670 RATIO 0.405 AVGEPS 0.5870E-01 FRMSE 3.8594 TRMSE 0.63497
PASS QC ALTIMETER (MB) 24.29 % OBS FAILED CHECK ( 7794 OF 32090) AVGTHR 1.9272 RATIO 0.100E+06 AVGEPS 0.2480E-01 FRMSE 43.739 TRMSE 99999.
PASS MXST_P limit 269997 MXST_P: 280000
PASS AO Possible Seg fault 0 #logs checked: 2390
PASS IQC Possible Seg fault 0 #logs checked: 4529
PASS NONSFC Possible Seg fault 0 #logs checked: 817
PASS MADIS DB Connection ok madis/proc/storage/log/MadisDbUpload/MadisDataUpload_20230329_398600.log
INFO MADIS DB Connection cnts Info:485 [2023-Mar-29 00:30:58.795381] - LastUploadStats:mode:upload_sfc,pid:64607,aborted:0,fileCount:1,wholeFileErrors:0,lines:39517,inserts:28707,updates:10810,duplicates:0,errors:0,stationsAdded:0,stationsUpdated:5,publishLines:0,publishErrors:0,notes:./202303290029_receipt.sfcd2.txt,run ms:53646
PASS AFWS files current # in last hour: 20
PASS madis.noaa.gov.stations current # updated in last day: 12
PASS madis.noaa.gov.trans current # updated in last day: 24
PASS MSAS ANAL PRODUCT -- SFCNCDF # Today Day: 23088
WARN MSAS ANAL PRODUCT -- SFCNCDF # Prior Day: 23087
madis/proc/sfc/ao/output/sfcncdf.out.230860000_3:
madis/proc/sfc/ao/output/sfcncdf.out.230861200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861300_2:
madis/proc/sfc/ao/output/sfcncdf.out.230861400_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861700_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861800_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861900_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862000_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862100_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862200_2:
madis/proc/sfc/ao/output/sfcncdf.out.230862200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870500_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870900_1:
madis/proc/sfc/ao/output/sfcncdf.out.230871100_1:
madis/proc/sfc/ao/output/sfcncdf.out.230871200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871300_2:
madis/proc/sfc/ao/output/sfcncdf.out.230871300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871400_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871800_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871900_4:
madis/proc/sfc/ao/output/sfcncdf.out.230872000_4:
madis/proc/sfc/ao/output/sfcncdf.out.230872100_4:
madis/proc/sfc/ao/output/sfcncdf.out.230872200_4:
PASS SFCANL MATRIX FAILURE # Today Day: 23088
WARN SFCANL MATRIX FAILURE # Prior Day: 23087
madis/proc/sfc/ao/output/sfcanl.out.230870900_1:
madis/proc/sfc/ao/output/sfcanl.out.230871100_1:
madis/proc/sfc/ao/output/sfcanl.out.230871300_4:
COMPARE Mesonet data providers cnt 48 for 20230328_2200:
COMPARE DECODE STN TBLS 149 Newest: 214481 Mar 28 21:30 LDAD/data/RAWSStation.txt Oldest: 8848 Sep 12 2019 LDAD/data/AWS-GANetStation.txt
COMPARE PRE-DECODE STN TBLS 149 Newest: 214481 Mar 28 21:30 LDAD/data/master/RAWSStation.txt Oldest: 1568 Sep 12 2019 LDAD/data/master/AWS-KSNetStation.txt
COMPARE pqsend entries 6 (Only on primary, 1 for each feedtype/distr)
COMPARE Surface QC text products 7320 hour (23): Newest: Mar 29 00:27 madis/proc/sfc/ao/qc122/qcsumry00.dat Oldest:
COMPARE sda IO Writes 2308722 Local Disk MB/s: 25 RS MB/s: 0 PQ MB/s: 25+ Tot secs: 3488+
COMPARE sda IO Reads(2308722 Local Disk MB/s: 1 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3488+
COMPARE NetCDF REC Writes 2308722 Avg DECODE rec/s: 85 Avg AO rec/s: 80 Avg NON-SFC rec/s: 1 Tot secs: 3488
INFO Network 2308722 eth0 15 MB/s TX 3 MB/s RX Tot secs: 3488
COMPARE usr1 Disk 2308722 24 WRITE MB/s | 1 READ MB/s | 6067 Avg WRT TPS | 14151 Max 1-min WRT TPS
COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size. Note: Max #recNum has to be < MXST_P 280000
AO-20230328_22 sfc/ao/qcms_data/buoy/20230328_2200 947 1222036
AO-20230328_22 sfc/ao/qcms_data/crn/20230328_2200 246 436072
AO-20230328_22 sfc/ao/qcms_data/hcn/20230328_2200 102 199844
AO-20230328_22 sfc/ao/qcms_data/hfmetar/20230328_2200 11998 18882160
AO-20230328_22 sfc/ao/qcms_data/hydro/20230328_2200 9649 12119436
AO-20230328_22 sfc/ao/qcms_data/mesonet/20230328_2200 249774 550026984
AO-20230328_22 sfc/ao/qcms_data/metar/20230328_2200 7693 11458004
AO-20230328_22 sfc/ao/qcms_data/nepp/20230328_2200 72 177472
AO-20230328_22 sfc/ao/qcms_data/rwis/20230328_2200 5943 21216708
AO-20230328_22 sfc/ao/qcms_data/urbanet/20230328_2200 5930 16131392
NS-20230328_22 nonsfc/madis_data/LDAD/profiler/netCDF/20230328_2200 45 2307264
NS-20230328_22 nonsfc/madis_data/LDAD/profiler/netCDF/20230328_2230 35 1803824
NS-20230328_22 nonsfc/madis_data/LDAD/snow/netCDF/20230328_2200 2293 2582528
NS-20230328_22 nonsfc/madis_data/point/acars/netcdf/20230328_2200 43634 50176312
NS-20230328_22 nonsfc/madis_data/point/acarsProfiles/netcdf/20230328_2200 418 60877596
NS-20230328_22 nonsfc/madis_data/point/raob/netcdf/20230328_2200 4 385876
AO-20230328_21 sfc/ao/qcms_data/buoy/20230328_2100 1042 1329956
AO-20230328_21 sfc/ao/qcms_data/crn/20230328_2100 245 434620
AO-20230328_21 sfc/ao/qcms_data/hcn/20230328_2100 102 199844
AO-20230328_21 sfc/ao/qcms_data/hfmetar/20230328_2100 11971 18840148
AO-20230328_21 sfc/ao/qcms_data/hydro/20230328_2100 9871 12375180
AO-20230328_21 sfc/ao/qcms_data/mesonet/20230328_2100 249643 549739308
AO-20230328_21 sfc/ao/qcms_data/metar/20230328_2100 7745 11534132
AO-20230328_21 sfc/ao/qcms_data/nepp/20230328_2100 72 177472
AO-20230328_21 sfc/ao/qcms_data/rwis/20230328_2100 5947 21230228
AO-20230328_21 sfc/ao/qcms_data/urbanet/20230328_2100 5940 16157992
NS-20230328_21 nonsfc/madis_data/LDAD/profiler/netCDF/20230328_2100 43 2206576
NS-20230328_21 nonsfc/madis_data/LDAD/profiler/netCDF/20230328_2130 34 1753480
NS-20230328_21 nonsfc/madis_data/LDAD/snow/netCDF/20230328_2100 2318 2607628
NS-20230328_21 nonsfc/madis_data/point/acars/netcdf/20230328_2100 38950 44817816
NS-20230328_21 nonsfc/madis_data/point/acarsProfiles/netcdf/20230328_2100 372 54182756
NS-20230328_21 nonsfc/madis_data/point/HDW/netcdf/20230328_2100 11 1273360
NS-20230328_21 nonsfc/madis_data/point/raob/netcdf/20230328_2100 1 180508
DECODE decode/data/fxa/LDAD/crn/netCDF/20230328_2200 246 316448
DECODE decode/data/fxa/LDAD/hcn/netCDF/20230328_2200 102 139672
DECODE decode/data/fxa/LDAD/hfmetar/netCDF/20230328_2200 11998 13293016
DECODE decode/data/fxa/LDAD/hydro/netCDF/20230328_2200 9772 8140080
DECODE decode/data/fxa/LDAD/mesonet/netCDF/20230328_2200 251538 311364256
DECODE decode/data/fxa/LDAD/nepp/netCDF/20230328_2200 72 111828
DECODE decode/data/fxa/LDAD/rwis/netCDF/20230328_2200 5974 15871828
DECODE decode/data/fxa/LDAD/snow/netCDF/20230328_2200 2293 1875184
DECODE decode/data/fxa/LDAD/urbanet/netCDF/20230328_2200 5930 5338272
IQC sfc/iqc1/qcms_data/buoy/20230328_2200 644 877828
IQC sfc/iqc1/qcms_data/crn/20230328_2200 153 301036
IQC sfc/iqc1/qcms_data/hcn/20230328_2200 0 69692
IQC sfc/iqc1/qcms_data/hfmetar/20230328_2200 3544 5727736
IQC sfc/iqc1/qcms_data/hydro/20230328_2200 3628 5183244
IQC sfc/iqc1/qcms_data/mesonet/20230328_2200 87629 193956564
IQC sfc/iqc1/qcms_data/metar/20230328_2200 3639 5522948
IQC sfc/iqc1/qcms_data/nepp/20230328_2200 0 72928
IQC sfc/iqc1/qcms_data/rwis/20230328_2200 1153 5026508
IQC sfc/iqc1/qcms_data/urbanet/20230328_2200 1236 3645352
IQC sfc/iqc2/qcms_data/buoy/20230328_2200 7 154196
IQC sfc/iqc2/qcms_data/crn/20230328_2200 40 136960
IQC sfc/iqc2/qcms_data/hcn/20230328_2200 0 69692
IQC sfc/iqc2/qcms_data/hfmetar/20230328_2200 2835 4624532
IQC sfc/iqc2/qcms_data/hydro/20230328_2200 1957 3258252
IQC sfc/iqc2/qcms_data/mesonet/20230328_2200 62614 139023624
IQC sfc/iqc2/qcms_data/metar/20230328_2200 1463 2337284
IQC sfc/iqc2/qcms_data/nepp/20230328_2200 0 72928
IQC sfc/iqc2/qcms_data/rwis/20230328_2200 2817 10650828
IQC sfc/iqc2/qcms_data/urbanet/20230328_2200 505 1700892
IQC sfc/iqc3/qcms_data/buoy/20230328_2200 55 208724
IQC sfc/iqc3/qcms_data/crn/20230328_2200 19 106468
IQC sfc/iqc3/qcms_data/hcn/20230328_2200 78 169220
IQC sfc/iqc3/qcms_data/hfmetar/20230328_2200 2555 4188852
IQC sfc/iqc3/qcms_data/hydro/20230328_2200 1412 2630412
IQC sfc/iqc3/qcms_data/mesonet/20230328_2200 36223 81068988
IQC sfc/iqc3/qcms_data/metar/20230328_2200 1301 2100116
IQC sfc/iqc3/qcms_data/nepp/20230328_2200 0 72928
IQC sfc/iqc3/qcms_data/rwis/20230328_2200 1091 4816948
IQC sfc/iqc3/qcms_data/urbanet/20230328_2200 2197 6201612
IQC sfc/iqc4/qcms_data/buoy/20230328_2200 151 317780
IQC sfc/iqc4/qcms_data/crn/20230328_2200 33 126796
IQC sfc/iqc4/qcms_data/hcn/20230328_2200 90 184532
IQC sfc/iqc4/qcms_data/hfmetar/20230328_2200 2528 4146840
IQC sfc/iqc4/qcms_data/hydro/20230328_2200 2536 3925260
IQC sfc/iqc4/qcms_data/mesonet/20230328_2200 46153 102875268
IQC sfc/iqc4/qcms_data/metar/20230328_2200 1166 1902476
IQC sfc/iqc4/qcms_data/nepp/20230328_2200 0 72928
IQC sfc/iqc4/qcms_data/rwis/20230328_2200 239 1937188
IQC sfc/iqc4/qcms_data/urbanet/20230328_2200 2595 7260292
IQC sfc/iqc5/qcms_data/buoy/20230328_2200 174 343908
IQC sfc/iqc5/qcms_data/crn/20230328_2200 27 118084
IQC sfc/iqc5/qcms_data/hcn/20230328_2200 0 69692
IQC sfc/iqc5/qcms_data/hfmetar/20230328_2200 2319 3821636
IQC sfc/iqc5/qcms_data/hydro/20230328_2200 2152 3482892
IQC sfc/iqc5/qcms_data/mesonet/20230328_2200 47701 106274676
IQC sfc/iqc5/qcms_data/metar/20230328_2200 983 1634564
IQC sfc/iqc5/qcms_data/nepp/20230328_2200 72 177472
IQC sfc/iqc5/qcms_data/rwis/20230328_2200 1363 5736308
IQC sfc/iqc5/qcms_data/urbanet/20230328_2200 1210 3576192