MADIS RT RUN-TIME STATUS for Tue Mar 28 19:32:02 GMT 2023 on vm-bldr-madis-process1a
INFO RT VM STATUS for Node vm-bldr-madis-process1a Time Tue Mar 28 19:32:02 GMT 2023
PASS METAR processing tbl file: 128320 Mar 21 21:03 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
PASS Station update flag file: 0 May 3 2019 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
PASS SNOTEL files Last updates: hourly<120min 102 daily<1800min 102 monthly<45000min 52
PASS Metar station table file: 1121683 May 3 2019 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
PASS Accept/reject|autorej files Cnts: AO: 3 IQC: 3 NON-SFC: 20
PASS Persistent processes Cnts: pqact (1+): 1 ldmd (1+): 13 DECODE (5): 5
PASS Dcmetr files CNT-LDAD.raw: 802
PASS RSAS values range check Total chk: 317840 Outside of range (<100): 0
PASS Aircraft data Check: REC-CNT>300 #TAMDAR 20230328_1800:275 20230328_1700:334 #PUBLIC-DP 20230328_1800:1900 20230328_1700:1747 #PUBLIC-DP-profile 20230328_1800:35 20230328_1700:41
PASS Load resource m15 < 9 cores 98% (goal > 60%) -- % of the time M15 < 9 Samples: 1172 #10=+ 0 #9 16 #8 152 #7 590
PASS Open file resource check: open file < 5000 Max Files: 2833 #<1000: 410 #<10000 1172
PASS Process count check: processes < 1000 Max Processes: 594
PASS AO Throughput 0 -- check: #"conflicts" <20 delayed %Delayed: 0% #Sample: 234 #Delayed: 0
INFO AO runtime avg sfcanl 14 seconds 1168/78
INFO AO runtime avg sfcncdf 1 seconds 78/61
INFO AO runtime avg sfcver 77 seconds 5515/71
INFO AO runtime avg sfcqcmsg 21 seconds 1711/78
INFO AO runtime avg sfcqcout meso 33 seconds 7280/216
INFO AO runtime avg sfcqcout_late meso 83 seconds 24543/293
INFO AO runtime avg FSL_qcout5m_ 45 seconds 10592/235
INFO AO runtime avg FSL_analrun15m_ 112 seconds 8771/78
PASS IQC Throughput 0 -- check: #"conflicts" <30 delayed %Delayed: 0% #Sample: 1175 #Delayed: 9 Competed timely: 1102
INFO IQC runtime avg qcstg1_2 5 seconds 104546/20607
INFO IQC runtime avg qcstg1_2 meso 20 seconds 46816/2327
INFO IQC runtime avg qcstg1_2_late meso 21 seconds 34438/1616
INFO IQC runtime avg sfcing 30 seconds 34956/1163
INFO IQC runtime avg sfchqc 63 seconds 73402/1161
PASS QC applied max value Max applied check should be more than 107, 123 is good: 123, // temperatureQCA(99999)
PASS RT Throughput -179136 secs -- check: Avg time < 180 sec. #Hrs: 19 Max secs: 3666437
PASS '0' length data Station table check: cnt '0' length files #: 0
PASS '0' length master Station table check: cnt '0' length files #: 0
PASS netcdf-4.1 lib NetCDF lib link not in routerStoreNetCDF
WARN Shefdecoder issues. Missing stations in either hydro and mesonet: check 20230328_1800: #CPYRMA: 0 #NJWXNET: 631
PASS QC MAPS SEA-LEVEL PRESSURE (MB) 24.77 % OBS FAILED CHECK ( 8345 OF 33691) AVGTHR 1.7913 RATIO 0.100E+06 AVGEPS 0.2470E-01 FRMSE 40.379 TRMSE 99999.
PASS QC DEW POINT DEPRESSION (K) 4.64 % OBS FAILED CHECK ( 3011 OF 64890) AVGTHR 5.2602 RATIO 0.350 AVGEPS 0.1421E-01 FRMSE 16.154 TRMSE 1.8434
PASS QC POTENTIAL TEMPERATURE (K) 4.07 % OBS FAILED CHECK ( 2855 OF 70093) AVGTHR 3.6391 RATIO 0.100E+06 AVGEPS 0.2561E-01 FRMSE 1459.7 TRMSE 99999.
PASS QC DEW POINT TEMPERATURE (K) 5.53 % OBS FAILED CHECK ( 3587 OF 64906) AVGTHR 3.9352 RATIO 0.342 AVGEPS 0.1562E-01 FRMSE 18.502 TRMSE 1.3468
PASS QC U COMPONENT OF SFC WIND (M/S) 2.63 % OBS FAILED CHECK ( 1708 OF 65041) AVGTHR 4.4191 RATIO 0.352 AVGEPS 0.1427E-01 FRMSE 8.0797 TRMSE 1.5572
PASS QC V COMPONENT OF SFC WIND (M/S) 3.60 % OBS FAILED CHECK ( 2343 OF 65041) AVGTHR 4.5498 RATIO 0.348 AVGEPS 0.1492E-01 FRMSE 8.3353 TRMSE 1.5852
PASS QC 3H ALTIMETER CHANGE (MB) 3.04 % OBS FAILED CHECK ( 915 OF 30108) AVGTHR 1.2167 RATIO 0.240 AVGEPS 0.1721E-01 FRMSE 44.470 TRMSE 0.29141
PASS QC NWS SEA-LEVEL PRESSURE (MB) 6.45 % OBS FAILED CHECK ( 47 OF 729) AVGTHR 1.5621 RATIO 0.428 AVGEPS 0.5827E-01 FRMSE 4.7014 TRMSE 0.66935
PASS QC ALTIMETER (MB) 23.68 % OBS FAILED CHECK ( 7977 OF 33691) AVGTHR 1.9300 RATIO 0.100E+06 AVGEPS 0.2519E-01 FRMSE 41.347 TRMSE 99999.
PASS MXST_P limit 268693 MXST_P: 280000
PASS AO Possible Seg fault 0 #logs checked: 2265
PASS IQC Possible Seg fault 0 #logs checked: 4446
PASS NONSFC Possible Seg fault 0 #logs checked: 807
PASS MADIS DB Connection ok madis/proc/storage/log/MadisDbUpload/MadisDataUpload_20230328_398535.log
INFO MADIS DB Connection cnts Info:485 [2023-Mar-28 19:31:22.587380] - LastUploadStats:mode:upload_sfc,pid:60568,aborted:0,fileCount:1,wholeFileErrors:0,lines:44267,inserts:34099,updates:10168,duplicates:0,errors:0,stationsAdded:0,stationsUpdated:4,publishLines:0,publishErrors:0,notes:./202303281930_receipt.sfcd2.txt,run ms:65449
PASS AFWS files current # in last hour: 20
PASS madis.noaa.gov.stations current # updated in last day: 12
PASS madis.noaa.gov.trans current # updated in last day: 24
WARN MSAS ANAL PRODUCT -- SFCNCDF # Today Day: 23087
madis/proc/sfc/ao/output/sfcncdf.out.230870200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870500_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870900_1:
madis/proc/sfc/ao/output/sfcncdf.out.230871100_1:
madis/proc/sfc/ao/output/sfcncdf.out.230871200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871300_2:
madis/proc/sfc/ao/output/sfcncdf.out.230871300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871400_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871800_4:
WARN MSAS ANAL PRODUCT -- SFCNCDF # Prior Day: 23086
madis/proc/sfc/ao/output/sfcncdf.out.230851900_4:
madis/proc/sfc/ao/output/sfcncdf.out.230852000_4:
madis/proc/sfc/ao/output/sfcncdf.out.230852100_4:
madis/proc/sfc/ao/output/sfcncdf.out.230852300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230860000_2:
madis/proc/sfc/ao/output/sfcncdf.out.230860000_3:
madis/proc/sfc/ao/output/sfcncdf.out.230861200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861300_2:
madis/proc/sfc/ao/output/sfcncdf.out.230861400_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861700_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861800_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861900_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862000_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862100_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862200_2:
madis/proc/sfc/ao/output/sfcncdf.out.230862200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870500_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870900_1:
madis/proc/sfc/ao/output/sfcncdf.out.230871100_1:
madis/proc/sfc/ao/output/sfcncdf.out.230871200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871300_2:
madis/proc/sfc/ao/output/sfcncdf.out.230871300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871400_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871800_4:
WARN SFCANL MATRIX FAILURE # Today Day: 23087
madis/proc/sfc/ao/output/sfcanl.out.230870900_1:
madis/proc/sfc/ao/output/sfcanl.out.230871100_1:
madis/proc/sfc/ao/output/sfcanl.out.230871300_4:
WARN SFCANL MATRIX FAILURE # Prior Day: 23086
madis/proc/sfc/ao/output/sfcanl.out.230860000_2:
madis/proc/sfc/ao/output/sfcanl.out.230860000_3:
madis/proc/sfc/ao/output/sfcanl.out.230862200_2:
COMPARE Mesonet data providers cnt 48 for 20230328_1700:
COMPARE DECODE STN TBLS 149 Newest: 214481 Mar 28 16:30 LDAD/data/RAWSStation.txt Oldest: 8848 Sep 12 2019 LDAD/data/AWS-GANetStation.txt
COMPARE PRE-DECODE STN TBLS 149 Newest: 214481 Mar 28 16:30 LDAD/data/master/RAWSStation.txt Oldest: 1568 Sep 12 2019 LDAD/data/master/AWS-KSNetStation.txt
COMPARE pqsend entries 6 (Only on primary, 1 for each feedtype/distr)
COMPARE Surface QC text products 7320 hour (18): Newest: Mar 28 19:27 madis/proc/sfc/ao/qc122/qcsumry19.dat Oldest:
COMPARE sda IO Writes 2308717 Local Disk MB/s: 24 RS MB/s: 0 PQ MB/s: 25+ Tot secs: 3475+
COMPARE sda IO Reads(2308717 Local Disk MB/s: 1 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3475+
COMPARE NetCDF REC Writes 2308717 Avg DECODE rec/s: 84 Avg AO rec/s: 84 Avg NON-SFC rec/s: 0 Tot secs: 3475
INFO Network 2308717 eth0 15 MB/s TX 3 MB/s RX Tot secs: 3475
COMPARE usr1 Disk 2308717 24 WRITE MB/s | 1 READ MB/s | 6061 Avg WRT TPS | 16051 Max 1-min WRT TPS
COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size. Note: Max #recNum has to be < MXST_P 280000
AO-20230328_17 sfc/ao/qcms_data/buoy/20230328_1700 885 1151604
AO-20230328_17 sfc/ao/qcms_data/crn/20230328_1700 245 434620
AO-20230328_17 sfc/ao/qcms_data/hcn/20230328_1700 102 199844
AO-20230328_17 sfc/ao/qcms_data/hfmetar/20230328_1700 11963 18827700
AO-20230328_17 sfc/ao/qcms_data/hydro/20230328_1700 9955 12471948
AO-20230328_17 sfc/ao/qcms_data/mesonet/20230328_1700 249501 549427476
AO-20230328_17 sfc/ao/qcms_data/metar/20230328_1700 8047 11976260
AO-20230328_17 sfc/ao/qcms_data/nepp/20230328_1700 72 177472
AO-20230328_17 sfc/ao/qcms_data/rwis/20230328_1700 5978 21335008
AO-20230328_17 sfc/ao/qcms_data/urbanet/20230328_1700 5933 16139372
NS-20230328_17 nonsfc/madis_data/LDAD/profiler/netCDF/20230328_1700 44 2256920
NS-20230328_17 nonsfc/madis_data/LDAD/profiler/netCDF/20230328_1730 34 1753480
NS-20230328_17 nonsfc/madis_data/LDAD/snow/netCDF/20230328_1700 2234 2523292
NS-20230328_17 nonsfc/madis_data/point/acars/netcdf/20230328_1700 37337 42972544
NS-20230328_17 nonsfc/madis_data/point/acarsProfiles/netcdf/20230328_1700 402 58548956
NS-20230328_17 nonsfc/madis_data/point/raob/netcdf/20230328_1700 1 180508
AO-20230328_16 sfc/ao/qcms_data/buoy/20230328_1600 830 1089124
AO-20230328_16 sfc/ao/qcms_data/crn/20230328_1600 246 436072
AO-20230328_16 sfc/ao/qcms_data/hcn/20230328_1600 102 199844
AO-20230328_16 sfc/ao/qcms_data/hfmetar/20230328_1600 11972 18841704
AO-20230328_16 sfc/ao/qcms_data/hydro/20230328_1600 10001 12524940
AO-20230328_16 sfc/ao/qcms_data/mesonet/20230328_1600 249547 549528492
AO-20230328_16 sfc/ao/qcms_data/metar/20230328_1600 8091 12040676
AO-20230328_16 sfc/ao/qcms_data/nepp/20230328_1600 72 177472
AO-20230328_16 sfc/ao/qcms_data/rwis/20230328_1600 5940 21206568
AO-20230328_16 sfc/ao/qcms_data/urbanet/20230328_1600 2881 8021052
NS-20230328_16 nonsfc/madis_data/LDAD/profiler/netCDF/20230328_1600 48 2458296
NS-20230328_16 nonsfc/madis_data/LDAD/profiler/netCDF/20230328_1630 37 1904512
NS-20230328_16 nonsfc/madis_data/LDAD/snow/netCDF/20230328_1600 2318 2607628
NS-20230328_16 nonsfc/madis_data/point/acars/netcdf/20230328_1600 39153 45050048
NS-20230328_16 nonsfc/madis_data/point/acarsProfiles/netcdf/20230328_1600 386 56220316
NS-20230328_16 nonsfc/madis_data/point/HDW/netcdf/20230328_1600 3 1272624
NS-20230328_16 nonsfc/madis_data/point/raob/netcdf/20230328_1600 3 317420
DECODE decode/data/fxa/LDAD/crn/netCDF/20230328_1700 245 315420
DECODE decode/data/fxa/LDAD/hcn/netCDF/20230328_1700 102 139672
DECODE decode/data/fxa/LDAD/hfmetar/netCDF/20230328_1700 11963 13254796
DECODE decode/data/fxa/LDAD/hydro/netCDF/20230328_1700 10080 8365536
DECODE decode/data/fxa/LDAD/mesonet/netCDF/20230328_1700 251443 311247216
DECODE decode/data/fxa/LDAD/nepp/netCDF/20230328_1700 72 111828
DECODE decode/data/fxa/LDAD/rwis/netCDF/20230328_1700 6009 15958488
DECODE decode/data/fxa/LDAD/snow/netCDF/20230328_1700 2234 1833884
DECODE decode/data/fxa/LDAD/urbanet/netCDF/20230328_1700 5933 5340828
IQC sfc/iqc1/qcms_data/buoy/20230328_1700 648 882372
IQC sfc/iqc1/qcms_data/crn/20230328_1700 153 301036
IQC sfc/iqc1/qcms_data/hcn/20230328_1700 0 69692
IQC sfc/iqc1/qcms_data/hfmetar/20230328_1700 3524 5696616
IQC sfc/iqc1/qcms_data/hydro/20230328_1700 3708 5275404
IQC sfc/iqc1/qcms_data/mesonet/20230328_1700 87500 193673280
IQC sfc/iqc1/qcms_data/metar/20230328_1700 3778 5726444
IQC sfc/iqc1/qcms_data/nepp/20230328_1700 0 72928
IQC sfc/iqc1/qcms_data/rwis/20230328_1700 1171 5087348
IQC sfc/iqc1/qcms_data/urbanet/20230328_1700 1230 3629392
IQC sfc/iqc2/qcms_data/buoy/20230328_1700 7 154196
IQC sfc/iqc2/qcms_data/crn/20230328_1700 40 136960
IQC sfc/iqc2/qcms_data/hcn/20230328_1700 0 69692
IQC sfc/iqc2/qcms_data/hfmetar/20230328_1700 2850 4647872
IQC sfc/iqc2/qcms_data/hydro/20230328_1700 2209 3548556
IQC sfc/iqc2/qcms_data/mesonet/20230328_1700 62414 138584424
IQC sfc/iqc2/qcms_data/metar/20230328_1700 1501 2392916
IQC sfc/iqc2/qcms_data/nepp/20230328_1700 0 72928
IQC sfc/iqc2/qcms_data/rwis/20230328_1700 2808 10620408
IQC sfc/iqc2/qcms_data/urbanet/20230328_1700 497 1679612
IQC sfc/iqc3/qcms_data/buoy/20230328_1700 54 207588
IQC sfc/iqc3/qcms_data/crn/20230328_1700 19 106468
IQC sfc/iqc3/qcms_data/hcn/20230328_1700 78 169220
IQC sfc/iqc3/qcms_data/hfmetar/20230328_1700 2566 4205968
IQC sfc/iqc3/qcms_data/hydro/20230328_1700 1698 2959884
IQC sfc/iqc3/qcms_data/mesonet/20230328_1700 36002 80583672
IQC sfc/iqc3/qcms_data/metar/20230328_1700 1392 2233340
IQC sfc/iqc3/qcms_data/nepp/20230328_1700 0 72928
IQC sfc/iqc3/qcms_data/rwis/20230328_1700 1082 4786528
IQC sfc/iqc3/qcms_data/urbanet/20230328_1700 2205 6222892
IQC sfc/iqc4/qcms_data/buoy/20230328_1700 135 299604
IQC sfc/iqc4/qcms_data/crn/20230328_1700 32 125344
IQC sfc/iqc4/qcms_data/hcn/20230328_1700 90 184532
IQC sfc/iqc4/qcms_data/hfmetar/20230328_1700 2507 4114164
IQC sfc/iqc4/qcms_data/hydro/20230328_1700 2547 3937932
IQC sfc/iqc4/qcms_data/mesonet/20230328_1700 46216 103013616
IQC sfc/iqc4/qcms_data/metar/20230328_1700 1257 2035700
IQC sfc/iqc4/qcms_data/nepp/20230328_1700 0 72928
IQC sfc/iqc4/qcms_data/rwis/20230328_1700 242 1947328
IQC sfc/iqc4/qcms_data/urbanet/20230328_1700 2610 7300192
IQC sfc/iqc5/qcms_data/buoy/20230328_1700 124 287108
IQC sfc/iqc5/qcms_data/crn/20230328_1700 27 118084
IQC sfc/iqc5/qcms_data/hcn/20230328_1700 0 69692
IQC sfc/iqc5/qcms_data/hfmetar/20230328_1700 2311 3809188
IQC sfc/iqc5/qcms_data/hydro/20230328_1700 2109 3433356
IQC sfc/iqc5/qcms_data/mesonet/20230328_1700 48006 106944456
IQC sfc/iqc5/qcms_data/metar/20230328_1700 1040 1718012
IQC sfc/iqc5/qcms_data/nepp/20230328_1700 72 177472
IQC sfc/iqc5/qcms_data/rwis/20230328_1700 1398 5854608
IQC sfc/iqc5/qcms_data/urbanet/20230328_1700 1208 3570872