MADIS RT RUN-TIME STATUS for Tue Mar 28 18:32:01 GMT 2023 on vm-bldr-madis-process1a
INFO RT VM STATUS for Node vm-bldr-madis-process1a Time Tue Mar 28 18:32:01 GMT 2023
PASS METAR processing tbl file: 128320 Mar 21 21:03 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
PASS Station update flag file: 0 May 3 2019 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
PASS SNOTEL files Last updates: hourly<120min 102 daily<1800min 102 monthly<45000min 52
PASS Metar station table file: 1121683 May 3 2019 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
PASS Accept/reject|autorej files Cnts: AO: 3 IQC: 3 NON-SFC: 20
PASS Persistent processes Cnts: pqact (1+): 1 ldmd (1+): 19 DECODE (5): 5
PASS Dcmetr files CNT-LDAD.raw: 801
PASS RSAS values range check Total chk: 317840 Outside of range (<100): 0
PASS Aircraft data Check: REC-CNT>300 #TAMDAR 20230328_1700:331 20230328_1600:270 #PUBLIC-DP 20230328_1700:1747 20230328_1600:2103 #PUBLIC-DP-profile 20230328_1700:41 20230328_1600:40
PASS Load resource m15 < 9 cores 98% (goal > 60%) -- % of the time M15 < 9 Samples: 1112 #10=+ 0 #9 16 #8 152 #7 559
PASS Open file resource check: open file < 5000 Max Files: 2833 #<1000: 386 #<10000 1112
PASS Process count check: processes < 1000 Max Processes: 594
PASS AO Throughput 0 -- check: #"conflicts" <20 delayed %Delayed: 0% #Sample: 222 #Delayed: 0
INFO AO runtime avg sfcanl 14 seconds 1106/74
INFO AO runtime avg sfcncdf 1 seconds 72/57
INFO AO runtime avg sfcver 78 seconds 5356/68
INFO AO runtime avg sfcqcmsg 21 seconds 1604/74
INFO AO runtime avg sfcqcout meso 33 seconds 6919/205
INFO AO runtime avg sfcqcout_late meso 83 seconds 23187/277
INFO AO runtime avg FSL_qcout5m_ 45 seconds 10045/223
INFO AO runtime avg FSL_analrun15m_ 113 seconds 8424/74
PASS IQC Throughput 0 -- check: #"conflicts" <30 delayed %Delayed: 0% #Sample: 1115 #Delayed: 6 Competed timely: 1045
INFO IQC runtime avg qcstg1_2 5 seconds 99203/19600
INFO IQC runtime avg qcstg1_2 meso 20 seconds 44445/2213
INFO IQC runtime avg qcstg1_2_late meso 21 seconds 32667/1535
INFO IQC runtime avg sfcing 30 seconds 33222/1106
INFO IQC runtime avg sfchqc 63 seconds 69784/1104
PASS QC applied max value Max applied check should be more than 107, 123 is good: 123, // temperatureQCA(99997)
PASS RT Throughput -8682 secs -- check: Avg time < 180 sec. #Hrs: 18 Max secs: 3666437
PASS '0' length data Station table check: cnt '0' length files #: 0
PASS '0' length master Station table check: cnt '0' length files #: 0
PASS netcdf-4.1 lib NetCDF lib link not in routerStoreNetCDF
WARN Shefdecoder issues. Missing stations in either hydro and mesonet: check 20230328_1700: #CPYRMA: 0 #NJWXNET: 628
PASS QC MAPS SEA-LEVEL PRESSURE (MB) 24.93 % OBS FAILED CHECK ( 8312 OF 33339) AVGTHR 1.7959 RATIO 0.100E+06 AVGEPS 0.2575E-01 FRMSE 41.572 TRMSE 99999.
PASS QC DEW POINT DEPRESSION (K) 4.68 % OBS FAILED CHECK ( 3011 OF 64362) AVGTHR 5.2623 RATIO 0.363 AVGEPS 0.1435E-01 FRMSE 16.071 TRMSE 1.9122
PASS QC POTENTIAL TEMPERATURE (K) 4.23 % OBS FAILED CHECK ( 2943 OF 69587) AVGTHR 3.6384 RATIO 0.100E+06 AVGEPS 0.2569E-01 FRMSE 1433.2 TRMSE 99999.
PASS QC DEW POINT TEMPERATURE (K) 5.44 % OBS FAILED CHECK ( 3504 OF 64383) AVGTHR 3.9396 RATIO 0.338 AVGEPS 0.1595E-01 FRMSE 18.621 TRMSE 1.3307
PASS QC U COMPONENT OF SFC WIND (M/S) 2.30 % OBS FAILED CHECK ( 1495 OF 64951) AVGTHR 4.4159 RATIO 0.341 AVGEPS 0.1415E-01 FRMSE 7.8048 TRMSE 1.5036
PASS QC V COMPONENT OF SFC WIND (M/S) 3.38 % OBS FAILED CHECK ( 2197 OF 64951) AVGTHR 4.5463 RATIO 0.341 AVGEPS 0.1477E-01 FRMSE 8.3031 TRMSE 1.5517
PASS QC 3H ALTIMETER CHANGE (MB) 2.56 % OBS FAILED CHECK ( 759 OF 29661) AVGTHR 1.2142 RATIO 0.173 AVGEPS 0.1706E-01 FRMSE 64.660 TRMSE 0.20999
PASS QC NWS SEA-LEVEL PRESSURE (MB) 6.52 % OBS FAILED CHECK ( 46 OF 705) AVGTHR 1.5489 RATIO 0.422 AVGEPS 0.5455E-01 FRMSE 4.7403 TRMSE 0.65431
PASS QC ALTIMETER (MB) 23.79 % OBS FAILED CHECK ( 7932 OF 33339) AVGTHR 1.9345 RATIO 0.100E+06 AVGEPS 0.2602E-01 FRMSE 42.612 TRMSE 99999.
PASS MXST_P limit 267903 MXST_P: 280000
PASS AO Possible Seg fault 0 #logs checked: 2267
PASS IQC Possible Seg fault 0 #logs checked: 4569
PASS NONSFC Possible Seg fault 0 #logs checked: 803
PASS MADIS DB Connection ok madis/proc/storage/log/MadisDbUpload/MadisDataUpload_20230328_398522.log
INFO MADIS DB Connection cnts Info:490 [2023-Mar-28 18:31:15.096211] - LastUploadStats:mode:upload_sfc,pid:19844,aborted:0,fileCount:1,wholeFileErrors:0,lines:43020,inserts:33368,updates:9651,duplicates:0,errors:2,stationsAdded:0,stationsUpdated:7,publishLines:0,publishErrors:0,notes:./202303281829_receipt.sfcd2.txt,run ms:67263
PASS AFWS files current # in last hour: 20
PASS madis.noaa.gov.stations current # updated in last day: 12
PASS madis.noaa.gov.trans current # updated in last day: 24
WARN MSAS ANAL PRODUCT -- SFCNCDF # Today Day: 23087
madis/proc/sfc/ao/output/sfcncdf.out.230870200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870500_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870900_1:
madis/proc/sfc/ao/output/sfcncdf.out.230871100_1:
madis/proc/sfc/ao/output/sfcncdf.out.230871200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871300_2:
madis/proc/sfc/ao/output/sfcncdf.out.230871300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871400_4:
WARN MSAS ANAL PRODUCT -- SFCNCDF # Prior Day: 23086
madis/proc/sfc/ao/output/sfcncdf.out.230851800_4:
madis/proc/sfc/ao/output/sfcncdf.out.230851900_4:
madis/proc/sfc/ao/output/sfcncdf.out.230852000_4:
madis/proc/sfc/ao/output/sfcncdf.out.230852100_4:
madis/proc/sfc/ao/output/sfcncdf.out.230852300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230860000_2:
madis/proc/sfc/ao/output/sfcncdf.out.230860000_3:
madis/proc/sfc/ao/output/sfcncdf.out.230861200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861300_2:
madis/proc/sfc/ao/output/sfcncdf.out.230861400_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861700_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861800_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861900_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862000_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862100_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862200_2:
madis/proc/sfc/ao/output/sfcncdf.out.230862200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870500_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870900_1:
madis/proc/sfc/ao/output/sfcncdf.out.230871100_1:
madis/proc/sfc/ao/output/sfcncdf.out.230871200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871300_2:
madis/proc/sfc/ao/output/sfcncdf.out.230871300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871400_4:
WARN SFCANL MATRIX FAILURE # Today Day: 23087
madis/proc/sfc/ao/output/sfcanl.out.230870900_1:
madis/proc/sfc/ao/output/sfcanl.out.230871100_1:
madis/proc/sfc/ao/output/sfcanl.out.230871300_4:
WARN SFCANL MATRIX FAILURE # Prior Day: 23086
madis/proc/sfc/ao/output/sfcanl.out.230860000_2:
madis/proc/sfc/ao/output/sfcanl.out.230860000_3:
madis/proc/sfc/ao/output/sfcanl.out.230862200_2:
COMPARE Mesonet data providers cnt 48 for 20230328_1600:
COMPARE DECODE STN TBLS 149 Newest: 214481 Mar 28 16:30 LDAD/data/RAWSStation.txt Oldest: 8848 Sep 12 2019 LDAD/data/AWS-GANetStation.txt
COMPARE PRE-DECODE STN TBLS 149 Newest: 214481 Mar 28 16:30 LDAD/data/master/RAWSStation.txt Oldest: 1568 Sep 12 2019 LDAD/data/master/AWS-KSNetStation.txt
COMPARE pqsend entries 6 (Only on primary, 1 for each feedtype/distr)
COMPARE Surface QC text products 7320 hour (17): Newest: Mar 28 18:27 madis/proc/sfc/ao/qc122/qcsumry18.dat Oldest:
COMPARE sda IO Writes 2308716 Local Disk MB/s: 25 RS MB/s: 0 PQ MB/s: 23+ Tot secs: 3417+
COMPARE sda IO Reads(2308716 Local Disk MB/s: 1 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3417+
COMPARE NetCDF REC Writes 2308716 Avg DECODE rec/s: 85 Avg AO rec/s: 84 Avg NON-SFC rec/s: 12 Tot secs: 3417
INFO Network 2308716 eth0 15 MB/s TX 3 MB/s RX Tot secs: 3417
COMPARE usr1 Disk 2308716 25 WRITE MB/s | 1 READ MB/s | 6140 Avg WRT TPS | 15042 Max 1-min WRT TPS
COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size. Note: Max #recNum has to be < MXST_P 280000
AO-20230328_16 sfc/ao/qcms_data/buoy/20230328_1600 830 1089124
AO-20230328_16 sfc/ao/qcms_data/crn/20230328_1600 246 436072
AO-20230328_16 sfc/ao/qcms_data/hcn/20230328_1600 102 199844
AO-20230328_16 sfc/ao/qcms_data/hfmetar/20230328_1600 11972 18841704
AO-20230328_16 sfc/ao/qcms_data/hydro/20230328_1600 10001 12524940
AO-20230328_16 sfc/ao/qcms_data/mesonet/20230328_1600 249547 549528492
AO-20230328_16 sfc/ao/qcms_data/metar/20230328_1600 8091 12040676
AO-20230328_16 sfc/ao/qcms_data/nepp/20230328_1600 72 177472
AO-20230328_16 sfc/ao/qcms_data/rwis/20230328_1600 5940 21206568
AO-20230328_16 sfc/ao/qcms_data/urbanet/20230328_1600 2881 8021052
NS-20230328_16 nonsfc/madis_data/LDAD/profiler/netCDF/20230328_1600 48 2458296
NS-20230328_16 nonsfc/madis_data/LDAD/profiler/netCDF/20230328_1630 37 1904512
NS-20230328_16 nonsfc/madis_data/LDAD/snow/netCDF/20230328_1600 2268 2557428
NS-20230328_16 nonsfc/madis_data/point/acars/netcdf/20230328_1600 39055 44937936
NS-20230328_16 nonsfc/madis_data/point/acarsProfiles/netcdf/20230328_1600 386 56220316
NS-20230328_16 nonsfc/madis_data/point/raob/netcdf/20230328_1600 2 248964
AO-20230328_15 sfc/ao/qcms_data/buoy/20230328_1500 940 1214084
AO-20230328_15 sfc/ao/qcms_data/crn/20230328_1500 246 436072
AO-20230328_15 sfc/ao/qcms_data/hcn/20230328_1500 101 198568
AO-20230328_15 sfc/ao/qcms_data/hfmetar/20230328_1500 11999 18883716
AO-20230328_15 sfc/ao/qcms_data/hydro/20230328_1500 9888 12394764
AO-20230328_15 sfc/ao/qcms_data/mesonet/20230328_1500 249137 548628132
AO-20230328_15 sfc/ao/qcms_data/metar/20230328_1500 8185 12178292
AO-20230328_15 sfc/ao/qcms_data/nepp/20230328_1500 72 177472
AO-20230328_15 sfc/ao/qcms_data/rwis/20230328_1500 6012 21449928
AO-20230328_15 sfc/ao/qcms_data/urbanet/20230328_1500 5922 16110112
NS-20230328_15 nonsfc/madis_data/LDAD/profiler/netCDF/20230328_1500 49 2508640
NS-20230328_15 nonsfc/madis_data/LDAD/profiler/netCDF/20230328_1530 39 2005200
NS-20230328_15 nonsfc/madis_data/LDAD/snow/netCDF/20230328_1500 2483 2773288
NS-20230328_15 nonsfc/madis_data/point/acars/netcdf/20230328_1500 40842 46982264
NS-20230328_15 nonsfc/madis_data/point/acarsProfiles/netcdf/20230328_1500 383 55783696
NS-20230328_15 nonsfc/madis_data/point/raob/netcdf/20230328_1500 2 248964
DECODE decode/data/fxa/LDAD/crn/netCDF/20230328_1600 246 316448
DECODE decode/data/fxa/LDAD/hcn/netCDF/20230328_1600 102 139672
DECODE decode/data/fxa/LDAD/hfmetar/netCDF/20230328_1600 11972 13264624
DECODE decode/data/fxa/LDAD/hydro/netCDF/20230328_1600 10130 8402136
DECODE decode/data/fxa/LDAD/mesonet/netCDF/20230328_1600 251445 311249680
DECODE decode/data/fxa/LDAD/nepp/netCDF/20230328_1600 72 111828
DECODE decode/data/fxa/LDAD/rwis/netCDF/20230328_1600 5971 15864400
DECODE decode/data/fxa/LDAD/snow/netCDF/20230328_1600 2270 1859084
DECODE decode/data/fxa/LDAD/urbanet/netCDF/20230328_1600 2881 2740524
IQC sfc/iqc1/qcms_data/buoy/20230328_1600 615 844884
IQC sfc/iqc1/qcms_data/crn/20230328_1600 153 301036
IQC sfc/iqc1/qcms_data/hcn/20230328_1600 0 69692
IQC sfc/iqc1/qcms_data/hfmetar/20230328_1600 3517 5685724
IQC sfc/iqc1/qcms_data/hydro/20230328_1600 3730 5300748
IQC sfc/iqc1/qcms_data/mesonet/20230328_1600 87551 193785276
IQC sfc/iqc1/qcms_data/metar/20230328_1600 3762 5703020
IQC sfc/iqc1/qcms_data/nepp/20230328_1600 0 72928
IQC sfc/iqc1/qcms_data/rwis/20230328_1600 1166 5070448
IQC sfc/iqc1/qcms_data/urbanet/20230328_1600 406 1437552
IQC sfc/iqc2/qcms_data/buoy/20230328_1600 7 154196
IQC sfc/iqc2/qcms_data/crn/20230328_1600 40 136960
IQC sfc/iqc2/qcms_data/hcn/20230328_1600 0 69692
IQC sfc/iqc2/qcms_data/hfmetar/20230328_1600 2835 4624532
IQC sfc/iqc2/qcms_data/hydro/20230328_1600 2243 3587724
IQC sfc/iqc2/qcms_data/mesonet/20230328_1600 62523 138823788
IQC sfc/iqc2/qcms_data/metar/20230328_1600 1531 2436836
IQC sfc/iqc2/qcms_data/nepp/20230328_1600 0 72928
IQC sfc/iqc2/qcms_data/rwis/20230328_1600 2805 10610268
IQC sfc/iqc2/qcms_data/urbanet/20230328_1600 163 791172
IQC sfc/iqc3/qcms_data/buoy/20230328_1600 50 203044
IQC sfc/iqc3/qcms_data/crn/20230328_1600 19 106468
IQC sfc/iqc3/qcms_data/hcn/20230328_1600 78 169220
IQC sfc/iqc3/qcms_data/hfmetar/20230328_1600 2565 4204412
IQC sfc/iqc3/qcms_data/hydro/20230328_1600 1635 2887308
IQC sfc/iqc3/qcms_data/mesonet/20230328_1600 36093 80783508
IQC sfc/iqc3/qcms_data/metar/20230328_1600 1385 2223092
IQC sfc/iqc3/qcms_data/nepp/20230328_1600 0 72928
IQC sfc/iqc3/qcms_data/rwis/20230328_1600 1081 4783148
IQC sfc/iqc3/qcms_data/urbanet/20230328_1600 1561 4509852
IQC sfc/iqc4/qcms_data/buoy/20230328_1600 138 303012
IQC sfc/iqc4/qcms_data/crn/20230328_1600 33 126796
IQC sfc/iqc4/qcms_data/hcn/20230328_1600 90 184532
IQC sfc/iqc4/qcms_data/hfmetar/20230328_1600 2521 4135948
IQC sfc/iqc4/qcms_data/hydro/20230328_1600 2559 3951756
IQC sfc/iqc4/qcms_data/mesonet/20230328_1600 46164 102899424
IQC sfc/iqc4/qcms_data/metar/20230328_1600 1258 2037164
IQC sfc/iqc4/qcms_data/nepp/20230328_1600 0 72928
IQC sfc/iqc4/qcms_data/rwis/20230328_1600 238 1933808
IQC sfc/iqc4/qcms_data/urbanet/20230328_1600 1550 4480592
IQC sfc/iqc5/qcms_data/buoy/20230328_1600 98 257572
IQC sfc/iqc5/qcms_data/crn/20230328_1600 27 118084
IQC sfc/iqc5/qcms_data/hcn/20230328_1600 0 69692
IQC sfc/iqc5/qcms_data/hfmetar/20230328_1600 2317 3818524
IQC sfc/iqc5/qcms_data/hydro/20230328_1600 2102 3425292
IQC sfc/iqc5/qcms_data/mesonet/20230328_1600 48004 106940064
IQC sfc/iqc5/qcms_data/metar/20230328_1600 1087 1786820
IQC sfc/iqc5/qcms_data/nepp/20230328_1600 72 177472
IQC sfc/iqc5/qcms_data/rwis/20230328_1600 1366 5746448
IQC sfc/iqc5/qcms_data/urbanet/20230328_1600 404 1432232