MADIS RT RUN-TIME STATUS for Tue Mar 28 21:32:01 GMT 2023 on vm-bldr-madis-process1a
INFO RT VM STATUS for Node vm-bldr-madis-process1a Time Tue Mar 28 21:32:01 GMT 2023
PASS METAR processing tbl file: 128320 Mar 21 21:03 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
PASS Station update flag file: 0 May 3 2019 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
PASS SNOTEL files Last updates: hourly<120min 102 daily<1800min 102 monthly<45000min 52
PASS Metar station table file: 1121683 May 3 2019 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
PASS Accept/reject|autorej files Cnts: AO: 3 IQC: 3 NON-SFC: 20
PASS Persistent processes Cnts: pqact (1+): 1 ldmd (1+): 16 DECODE (5): 5
PASS Dcmetr files CNT-LDAD.raw: 807
PASS RSAS values range check Total chk: 317840 Outside of range (<100): 0
PASS Aircraft data Check: REC-CNT>300 #TAMDAR 20230328_2000:244 20230328_1900:169 #PUBLIC-DP 20230328_2000:2005 20230328_1900:1658 #PUBLIC-DP-profile 20230328_2000:32 20230328_1900:29
PASS Load resource m15 < 9 cores 98% (goal > 60%) -- % of the time M15 < 9 Samples: 1292 #10=+ 0 #9 16 #8 175 #7 655
PASS Open file resource check: open file < 5000 Max Files: 2833 #<1000: 455 #<10000 1292
PASS Process count check: processes < 1000 Max Processes: 594
PASS AO Throughput 0 -- check: #"conflicts" <20 delayed %Delayed: 0% #Sample: 258 #Delayed: 0
INFO AO runtime avg sfcanl 15 seconds 1291/86
INFO AO runtime avg sfcncdf 1 seconds 85/66
INFO AO runtime avg sfcver 75 seconds 5841/77
INFO AO runtime avg sfcqcmsg 22 seconds 1898/86
INFO AO runtime avg sfcqcout meso 33 seconds 8053/240
INFO AO runtime avg sfcqcout_late meso 83 seconds 27214/325
INFO AO runtime avg FSL_qcout5m_ 45 seconds 11705/259
INFO AO runtime avg FSL_analrun15m_ 109 seconds 9439/86
PASS IQC Throughput 0 -- check: #"conflicts" <30 delayed %Delayed: 0% #Sample: 1295 #Delayed: 9 Competed timely: 1222
INFO IQC runtime avg qcstg1_2 5 seconds 115766/22899
INFO IQC runtime avg qcstg1_2 meso 20 seconds 51643/2567
INFO IQC runtime avg qcstg1_2_late meso 21 seconds 38198/1793
INFO IQC runtime avg sfcing 30 seconds 38565/1282
INFO IQC runtime avg sfchqc 63 seconds 80979/1281
PASS QC applied max value Max applied check should be more than 107, 123 is good: 123, // temperatureQCA(99998)
PASS RT Throughput -162083 secs -- check: Avg time < 180 sec. #Hrs: 21 Max secs: 3666437
PASS '0' length data Station table check: cnt '0' length files #: 0
PASS '0' length master Station table check: cnt '0' length files #: 0
PASS netcdf-4.1 lib NetCDF lib link not in routerStoreNetCDF
WARN Shefdecoder issues. Missing stations in either hydro and mesonet: check 20230328_2000: #CPYRMA: 0 #NJWXNET: 636
PASS QC MAPS SEA-LEVEL PRESSURE (MB) 24.81 % OBS FAILED CHECK ( 8339 OF 33617) AVGTHR 1.7943 RATIO 0.100E+06 AVGEPS 0.2520E-01 FRMSE 41.954 TRMSE 99999.
PASS QC DEW POINT DEPRESSION (K) 4.78 % OBS FAILED CHECK ( 3096 OF 64818) AVGTHR 5.2654 RATIO 0.338 AVGEPS 0.1449E-01 FRMSE 16.836 TRMSE 1.7818
PASS QC POTENTIAL TEMPERATURE (K) 3.83 % OBS FAILED CHECK ( 2683 OF 69971) AVGTHR 3.6408 RATIO 0.100E+06 AVGEPS 0.2576E-01 FRMSE 1312.5 TRMSE 99999.
PASS QC DEW POINT TEMPERATURE (K) 5.56 % OBS FAILED CHECK ( 3602 OF 64831) AVGTHR 3.9294 RATIO 0.319 AVGEPS 0.1524E-01 FRMSE 19.201 TRMSE 1.2515
PASS QC U COMPONENT OF SFC WIND (M/S) 2.67 % OBS FAILED CHECK ( 1760 OF 65909) AVGTHR 4.4230 RATIO 0.376 AVGEPS 0.1455E-01 FRMSE 8.0764 TRMSE 1.6637
PASS QC V COMPONENT OF SFC WIND (M/S) 3.00 % OBS FAILED CHECK ( 1975 OF 65909) AVGTHR 4.5441 RATIO 0.369 AVGEPS 0.1466E-01 FRMSE 8.1180 TRMSE 1.6745
PASS QC 3H ALTIMETER CHANGE (MB) 2.60 % OBS FAILED CHECK ( 782 OF 30038) AVGTHR 1.2193 RATIO 0.326 AVGEPS 0.1807E-01 FRMSE 9.8694 TRMSE 0.39717
PASS QC NWS SEA-LEVEL PRESSURE (MB) 5.92 % OBS FAILED CHECK ( 42 OF 710) AVGTHR 1.5526 RATIO 0.414 AVGEPS 0.5532E-01 FRMSE 3.8637 TRMSE 0.64267
PASS QC ALTIMETER (MB) 23.93 % OBS FAILED CHECK ( 8045 OF 33617) AVGTHR 1.9275 RATIO 0.100E+06 AVGEPS 0.2470E-01 FRMSE 42.560 TRMSE 99999.
PASS MXST_P limit 270888 MXST_P: 280000
PASS AO Possible Seg fault 0 #logs checked: 2265
PASS IQC Possible Seg fault 0 #logs checked: 4628
PASS NONSFC Possible Seg fault 0 #logs checked: 807
PASS MADIS DB Connection ok madis/proc/storage/log/MadisDbUpload/MadisDataUpload_20230328_398561.log
INFO MADIS DB Connection cnts Info:495 [2023-Mar-28 21:31:16.794788] - LastUploadStats:mode:upload_sfc,pid:55665,aborted:0,fileCount:1,wholeFileErrors:0,lines:42632,inserts:31068,updates:11562,duplicates:0,errors:4,stationsAdded:0,stationsUpdated:7,publishLines:0,publishErrors:0,notes:./202303282130_receipt.sfcd2.txt,run ms:61893
PASS AFWS files current # in last hour: 20
PASS madis.noaa.gov.stations current # updated in last day: 12
PASS madis.noaa.gov.trans current # updated in last day: 24
WARN MSAS ANAL PRODUCT -- SFCNCDF # Today Day: 23087
madis/proc/sfc/ao/output/sfcncdf.out.230870200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870500_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870900_1:
madis/proc/sfc/ao/output/sfcncdf.out.230871100_1:
madis/proc/sfc/ao/output/sfcncdf.out.230871200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871300_2:
madis/proc/sfc/ao/output/sfcncdf.out.230871300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871400_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871800_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871900_4:
madis/proc/sfc/ao/output/sfcncdf.out.230872000_4:
WARN MSAS ANAL PRODUCT -- SFCNCDF # Prior Day: 23086
madis/proc/sfc/ao/output/sfcncdf.out.230852100_4:
madis/proc/sfc/ao/output/sfcncdf.out.230852300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230860000_2:
madis/proc/sfc/ao/output/sfcncdf.out.230860000_3:
madis/proc/sfc/ao/output/sfcncdf.out.230861200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861300_2:
madis/proc/sfc/ao/output/sfcncdf.out.230861400_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861700_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861800_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861900_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862000_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862100_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862200_2:
madis/proc/sfc/ao/output/sfcncdf.out.230862200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870500_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870900_1:
madis/proc/sfc/ao/output/sfcncdf.out.230871100_1:
madis/proc/sfc/ao/output/sfcncdf.out.230871200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871300_2:
madis/proc/sfc/ao/output/sfcncdf.out.230871300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871400_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871800_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871900_4:
madis/proc/sfc/ao/output/sfcncdf.out.230872000_4:
WARN SFCANL MATRIX FAILURE # Today Day: 23087
madis/proc/sfc/ao/output/sfcanl.out.230870900_1:
madis/proc/sfc/ao/output/sfcanl.out.230871100_1:
madis/proc/sfc/ao/output/sfcanl.out.230871300_4:
WARN SFCANL MATRIX FAILURE # Prior Day: 23086
madis/proc/sfc/ao/output/sfcanl.out.230860000_2:
madis/proc/sfc/ao/output/sfcanl.out.230860000_3:
madis/proc/sfc/ao/output/sfcanl.out.230862200_2:
COMPARE Mesonet data providers cnt 48 for 20230328_1900:
COMPARE DECODE STN TBLS 149 Newest: 214481 Mar 28 21:30 LDAD/data/RAWSStation.txt Oldest: 8848 Sep 12 2019 LDAD/data/AWS-GANetStation.txt
COMPARE PRE-DECODE STN TBLS 149 Newest: 214481 Mar 28 21:30 LDAD/data/master/RAWSStation.txt Oldest: 1568 Sep 12 2019 LDAD/data/master/AWS-KSNetStation.txt
COMPARE pqsend entries 6 (Only on primary, 1 for each feedtype/distr)
COMPARE Surface QC text products 7320 hour (20): Newest: Mar 28 21:27 madis/proc/sfc/ao/qc122/qcsumry21.dat Oldest:
COMPARE sda IO Writes 2308719 Local Disk MB/s: 24 RS MB/s: 0 PQ MB/s: 25+ Tot secs: 3453+
COMPARE sda IO Reads(2308719 Local Disk MB/s: 1 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3453+
COMPARE NetCDF REC Writes 2308719 Avg DECODE rec/s: 86 Avg AO rec/s: 89 Avg NON-SFC rec/s: 11 Tot secs: 3453
INFO Network 2308719 eth0 15 MB/s TX 3 MB/s RX Tot secs: 3453
COMPARE usr1 Disk 2308719 24 WRITE MB/s | 1 READ MB/s | 5907 Avg WRT TPS | 19796 Max 1-min WRT TPS
COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size. Note: Max #recNum has to be < MXST_P 280000
AO-20230328_19 sfc/ao/qcms_data/buoy/20230328_1900 972 1250436
AO-20230328_19 sfc/ao/qcms_data/crn/20230328_1900 246 436072
AO-20230328_19 sfc/ao/qcms_data/hcn/20230328_1900 102 199844
AO-20230328_19 sfc/ao/qcms_data/hfmetar/20230328_1900 11971 18840148
AO-20230328_19 sfc/ao/qcms_data/hydro/20230328_1900 9802 12295692
AO-20230328_19 sfc/ao/qcms_data/mesonet/20230328_1900 249647 549748092
AO-20230328_19 sfc/ao/qcms_data/metar/20230328_1900 7748 11538524
AO-20230328_19 sfc/ao/qcms_data/nepp/20230328_1900 72 177472
AO-20230328_19 sfc/ao/qcms_data/rwis/20230328_1900 5916 21125448
AO-20230328_19 sfc/ao/qcms_data/urbanet/20230328_1900 5959 16208532
NS-20230328_19 nonsfc/madis_data/LDAD/profiler/netCDF/20230328_1900 44 2256920
NS-20230328_19 nonsfc/madis_data/LDAD/profiler/netCDF/20230328_1930 34 1753480
NS-20230328_19 nonsfc/madis_data/LDAD/snow/netCDF/20230328_1900 2260 2549396
NS-20230328_19 nonsfc/madis_data/point/acars/netcdf/20230328_1900 38157 43910624
NS-20230328_19 nonsfc/madis_data/point/acarsProfiles/netcdf/20230328_1900 384 55929236
NS-20230328_19 nonsfc/madis_data/point/raob/netcdf/20230328_1900 1 180508
AO-20230328_18 sfc/ao/qcms_data/buoy/20230328_1800 972 1250436
AO-20230328_18 sfc/ao/qcms_data/crn/20230328_1800 245 434620
AO-20230328_18 sfc/ao/qcms_data/hcn/20230328_1800 102 199844
AO-20230328_18 sfc/ao/qcms_data/hfmetar/20230328_1800 11987 18865044
AO-20230328_18 sfc/ao/qcms_data/hydro/20230328_1800 9796 12288780
AO-20230328_18 sfc/ao/qcms_data/mesonet/20230328_1800 250192 550944912
AO-20230328_18 sfc/ao/qcms_data/metar/20230328_1800 7923 11794724
AO-20230328_18 sfc/ao/qcms_data/nepp/20230328_1800 72 177472
AO-20230328_18 sfc/ao/qcms_data/rwis/20230328_1800 5863 20946308
AO-20230328_18 sfc/ao/qcms_data/urbanet/20230328_1800 5917 16096812
NS-20230328_18 nonsfc/madis_data/LDAD/profiler/netCDF/20230328_1800 46 2357608
NS-20230328_18 nonsfc/madis_data/LDAD/profiler/netCDF/20230328_1830 36 1854168
NS-20230328_18 nonsfc/madis_data/LDAD/snow/netCDF/20230328_1800 2261 2550400
NS-20230328_18 nonsfc/madis_data/point/acars/netcdf/20230328_1800 36175 41643216
NS-20230328_18 nonsfc/madis_data/point/acarsProfiles/netcdf/20230328_1800 331 48215616
NS-20230328_18 nonsfc/madis_data/point/HDW/netcdf/20230328_1800 28 1274924
NS-20230328_18 nonsfc/madis_data/point/raob/netcdf/20230328_1800 19 1412716
DECODE decode/data/fxa/LDAD/crn/netCDF/20230328_1900 246 316448
DECODE decode/data/fxa/LDAD/hcn/netCDF/20230328_1900 102 139672
DECODE decode/data/fxa/LDAD/hfmetar/netCDF/20230328_1900 11971 13263532
DECODE decode/data/fxa/LDAD/hydro/netCDF/20230328_1900 9925 8252076
DECODE decode/data/fxa/LDAD/mesonet/netCDF/20230328_1900 251474 311285408
DECODE decode/data/fxa/LDAD/nepp/netCDF/20230328_1900 72 111828
DECODE decode/data/fxa/LDAD/rwis/netCDF/20230328_1900 5947 15804976
DECODE decode/data/fxa/LDAD/snow/netCDF/20230328_1900 2260 1852084
DECODE decode/data/fxa/LDAD/urbanet/netCDF/20230328_1900 5959 5362980
IQC sfc/iqc1/qcms_data/buoy/20230328_1900 693 933492
IQC sfc/iqc1/qcms_data/crn/20230328_1900 153 301036
IQC sfc/iqc1/qcms_data/hcn/20230328_1900 0 69692
IQC sfc/iqc1/qcms_data/hfmetar/20230328_1900 3534 5712176
IQC sfc/iqc1/qcms_data/hydro/20230328_1900 3725 5294988
IQC sfc/iqc1/qcms_data/mesonet/20230328_1900 87571 193829196
IQC sfc/iqc1/qcms_data/metar/20230328_1900 3696 5606396
IQC sfc/iqc1/qcms_data/nepp/20230328_1900 0 72928
IQC sfc/iqc1/qcms_data/rwis/20230328_1900 1139 4979188
IQC sfc/iqc1/qcms_data/urbanet/20230328_1900 1238 3650672
IQC sfc/iqc2/qcms_data/buoy/20230328_1900 2 148516
IQC sfc/iqc2/qcms_data/crn/20230328_1900 40 136960
IQC sfc/iqc2/qcms_data/hcn/20230328_1900 0 69692
IQC sfc/iqc2/qcms_data/hfmetar/20230328_1900 2844 4638536
IQC sfc/iqc2/qcms_data/hydro/20230328_1900 1986 3291660
IQC sfc/iqc2/qcms_data/mesonet/20230328_1900 62543 138867708
IQC sfc/iqc2/qcms_data/metar/20230328_1900 1436 2297756
IQC sfc/iqc2/qcms_data/nepp/20230328_1900 0 72928
IQC sfc/iqc2/qcms_data/rwis/20230328_1900 2805 10610268
IQC sfc/iqc2/qcms_data/urbanet/20230328_1900 499 1684932
IQC sfc/iqc3/qcms_data/buoy/20230328_1900 52 205316
IQC sfc/iqc3/qcms_data/crn/20230328_1900 19 106468
IQC sfc/iqc3/qcms_data/hcn/20230328_1900 78 169220
IQC sfc/iqc3/qcms_data/hfmetar/20230328_1900 2569 4210636
IQC sfc/iqc3/qcms_data/hydro/20230328_1900 1455 2679948
IQC sfc/iqc3/qcms_data/mesonet/20230328_1900 36088 80772528
IQC sfc/iqc3/qcms_data/metar/20230328_1900 1293 2088404
IQC sfc/iqc3/qcms_data/nepp/20230328_1900 0 72928
IQC sfc/iqc3/qcms_data/rwis/20230328_1900 1081 4783148
IQC sfc/iqc3/qcms_data/urbanet/20230328_1900 2210 6236192
IQC sfc/iqc4/qcms_data/buoy/20230328_1900 141 306420
IQC sfc/iqc4/qcms_data/crn/20230328_1900 33 126796
IQC sfc/iqc4/qcms_data/hcn/20230328_1900 90 184532
IQC sfc/iqc4/qcms_data/hfmetar/20230328_1900 2514 4125056
IQC sfc/iqc4/qcms_data/hydro/20230328_1900 2547 3937932
IQC sfc/iqc4/qcms_data/mesonet/20230328_1900 46210 103000440
IQC sfc/iqc4/qcms_data/metar/20230328_1900 1163 1898084
IQC sfc/iqc4/qcms_data/nepp/20230328_1900 0 72928
IQC sfc/iqc4/qcms_data/rwis/20230328_1900 240 1940568
IQC sfc/iqc4/qcms_data/urbanet/20230328_1900 2619 7324132
IQC sfc/iqc5/qcms_data/buoy/20230328_1900 159 326868
IQC sfc/iqc5/qcms_data/crn/20230328_1900 27 118084
IQC sfc/iqc5/qcms_data/hcn/20230328_1900 0 69692
IQC sfc/iqc5/qcms_data/hfmetar/20230328_1900 2309 3806076
IQC sfc/iqc5/qcms_data/hydro/20230328_1900 2152 3482892
IQC sfc/iqc5/qcms_data/mesonet/20230328_1900 47816 106527216
IQC sfc/iqc5/qcms_data/metar/20230328_1900 1015 1681412
IQC sfc/iqc5/qcms_data/nepp/20230328_1900 72 177472
IQC sfc/iqc5/qcms_data/rwis/20230328_1900 1369 5756588
IQC sfc/iqc5/qcms_data/urbanet/20230328_1900 1221 3605452