MADIS RT RUN-TIME STATUS for Tue Mar 28 20:32:02 GMT 2023 on vm-bldr-madis-process1a
INFO RT VM STATUS for Node vm-bldr-madis-process1a Time Tue Mar 28 20:32:02 GMT 2023
PASS METAR processing tbl file: 128320 Mar 21 21:03 madis/proc/decode/data/fxa/LDAD/data/NonFedAWOSStation.tbl
PASS Station update flag file: 0 May 3 2019 madis/proc/decode/awips/fxa/GO-UPDATE-STATIONS
PASS SNOTEL files Last updates: hourly<120min 102 daily<1800min 102 monthly<45000min 52
PASS Metar station table file: 1121683 May 3 2019 madis/proc/sfc/iqc1/fslparms/lsfc.tbl
PASS Accept/reject|autorej files Cnts: AO: 3 IQC: 3 NON-SFC: 20
PASS Persistent processes Cnts: pqact (1+): 1 ldmd (1+): 18 DECODE (5): 5
PASS Dcmetr files CNT-LDAD.raw: 805
PASS RSAS values range check Total chk: 317840 Outside of range (<100): 0
PASS Aircraft data Check: REC-CNT>300 #TAMDAR 20230328_1900:169 20230328_1800:275 #PUBLIC-DP 20230328_1900:1658 20230328_1800:1900 #PUBLIC-DP-profile 20230328_1900:29 20230328_1800:35
PASS Load resource m15 < 9 cores 98% (goal > 60%) -- % of the time M15 < 9 Samples: 1232 #10=+ 0 #9 16 #8 166 #7 624
PASS Open file resource check: open file < 5000 Max Files: 2833 #<1000: 433 #<10000 1232
PASS Process count check: processes < 1000 Max Processes: 594
PASS AO Throughput 0 -- check: #"conflicts" <20 delayed %Delayed: 0% #Sample: 246 #Delayed: 0
INFO AO runtime avg sfcanl 14 seconds 1228/82
INFO AO runtime avg sfcncdf 1 seconds 82/64
INFO AO runtime avg sfcver 76 seconds 5668/74
INFO AO runtime avg sfcqcmsg 21 seconds 1796/82
INFO AO runtime avg sfcqcout meso 33 seconds 7664/228
INFO AO runtime avg sfcqcout_late meso 83 seconds 25865/309
INFO AO runtime avg FSL_qcout5m_ 45 seconds 11142/247
INFO AO runtime avg FSL_analrun15m_ 110 seconds 9085/82
PASS IQC Throughput 0 -- check: #"conflicts" <30 delayed %Delayed: 0% #Sample: 1235 #Delayed: 9 Competed timely: 1162
INFO IQC runtime avg qcstg1_2 5 seconds 110023/21743
INFO IQC runtime avg qcstg1_2 meso 20 seconds 49177/2447
INFO IQC runtime avg qcstg1_2_late meso 21 seconds 36262/1705
INFO IQC runtime avg sfcing 30 seconds 36780/1224
INFO IQC runtime avg sfchqc 63 seconds 77183/1221
PASS QC applied max value Max applied check should be more than 107, 123 is good: 123, // temperatureQCA(99999)
PASS RT Throughput -170169 secs -- check: Avg time < 180 sec. #Hrs: 20 Max secs: 3666437
PASS '0' length data Station table check: cnt '0' length files #: 0
PASS '0' length master Station table check: cnt '0' length files #: 0
PASS netcdf-4.1 lib NetCDF lib link not in routerStoreNetCDF
WARN Shefdecoder issues. Missing stations in either hydro and mesonet: check 20230328_1900: #CPYRMA: 0 #NJWXNET: 636
WARN QC MAPS SEA-LEVEL PRESSURE (MB) 25.18 % OBS FAILED CHECK ( 8535 OF 33893) AVGTHR 1.7930 RATIO 0.100E+06 AVGEPS 0.2495E-01 FRMSE 40.759 TRMSE 99999.
PASS QC DEW POINT DEPRESSION (K) 4.63 % OBS FAILED CHECK ( 3014 OF 65046) AVGTHR 5.2652 RATIO 0.339 AVGEPS 0.1445E-01 FRMSE 16.705 TRMSE 1.7859
PASS QC POTENTIAL TEMPERATURE (K) 3.86 % OBS FAILED CHECK ( 2710 OF 70260) AVGTHR 3.6414 RATIO 0.100E+06 AVGEPS 0.2587E-01 FRMSE 1307.8 TRMSE 99999.
PASS QC DEW POINT TEMPERATURE (K) 5.42 % OBS FAILED CHECK ( 3524 OF 65059) AVGTHR 3.9339 RATIO 0.323 AVGEPS 0.1553E-01 FRMSE 19.087 TRMSE 1.2716
PASS QC U COMPONENT OF SFC WIND (M/S) 2.84 % OBS FAILED CHECK ( 1875 OF 65922) AVGTHR 4.4214 RATIO 0.366 AVGEPS 0.1440E-01 FRMSE 7.8875 TRMSE 1.6166
PASS QC V COMPONENT OF SFC WIND (M/S) 3.51 % OBS FAILED CHECK ( 2313 OF 65922) AVGTHR 4.5446 RATIO 0.360 AVGEPS 0.1468E-01 FRMSE 8.0571 TRMSE 1.6370
PASS QC 3H ALTIMETER CHANGE (MB) 3.62 % OBS FAILED CHECK ( 1094 OF 30259) AVGTHR 1.2162 RATIO 0.296 AVGEPS 0.1781E-01 FRMSE 36.108 TRMSE 0.36046
PASS QC NWS SEA-LEVEL PRESSURE (MB) 6.82 % OBS FAILED CHECK ( 47 OF 689) AVGTHR 1.5564 RATIO 0.428 AVGEPS 0.5604E-01 FRMSE 4.1257 TRMSE 0.66604
PASS QC ALTIMETER (MB) 23.98 % OBS FAILED CHECK ( 8128 OF 33893) AVGTHR 1.9298 RATIO 0.100E+06 AVGEPS 0.2499E-01 FRMSE 41.799 TRMSE 99999.
PASS MXST_P limit 270800 MXST_P: 280000
PASS AO Possible Seg fault 0 #logs checked: 2265
PASS IQC Possible Seg fault 0 #logs checked: 4507
PASS NONSFC Possible Seg fault 0 #logs checked: 816
PASS MADIS DB Connection ok madis/proc/storage/log/MadisDbUpload/MadisDataUpload_20230328_398548.log
INFO MADIS DB Connection cnts Info:487 [2023-Mar-28 20:30:57.362212] - LastUploadStats:mode:upload_sfc,pid:51660,aborted:0,fileCount:1,wholeFileErrors:0,lines:41541,inserts:32148,updates:9393,duplicates:0,errors:4,stationsAdded:0,stationsUpdated:4,publishLines:0,publishErrors:0,notes:./202303282029_receipt.sfcd2.txt,run ms:60231
PASS AFWS files current # in last hour: 20
PASS madis.noaa.gov.stations current # updated in last day: 12
PASS madis.noaa.gov.trans current # updated in last day: 24
WARN MSAS ANAL PRODUCT -- SFCNCDF # Today Day: 23087
madis/proc/sfc/ao/output/sfcncdf.out.230870200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870500_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870900_1:
madis/proc/sfc/ao/output/sfcncdf.out.230871100_1:
madis/proc/sfc/ao/output/sfcncdf.out.230871200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871300_2:
madis/proc/sfc/ao/output/sfcncdf.out.230871300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871400_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871800_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871900_4:
WARN MSAS ANAL PRODUCT -- SFCNCDF # Prior Day: 23086
madis/proc/sfc/ao/output/sfcncdf.out.230852000_4:
madis/proc/sfc/ao/output/sfcncdf.out.230852100_4:
madis/proc/sfc/ao/output/sfcncdf.out.230852300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230860000_2:
madis/proc/sfc/ao/output/sfcncdf.out.230860000_3:
madis/proc/sfc/ao/output/sfcncdf.out.230861200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861300_2:
madis/proc/sfc/ao/output/sfcncdf.out.230861400_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861700_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861800_4:
madis/proc/sfc/ao/output/sfcncdf.out.230861900_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862000_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862100_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862200_2:
madis/proc/sfc/ao/output/sfcncdf.out.230862200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230862300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870500_4:
madis/proc/sfc/ao/output/sfcncdf.out.230870900_1:
madis/proc/sfc/ao/output/sfcncdf.out.230871100_1:
madis/proc/sfc/ao/output/sfcncdf.out.230871200_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871300_2:
madis/proc/sfc/ao/output/sfcncdf.out.230871300_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871400_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871800_4:
madis/proc/sfc/ao/output/sfcncdf.out.230871900_4:
WARN SFCANL MATRIX FAILURE # Today Day: 23087
madis/proc/sfc/ao/output/sfcanl.out.230870900_1:
madis/proc/sfc/ao/output/sfcanl.out.230871100_1:
madis/proc/sfc/ao/output/sfcanl.out.230871300_4:
WARN SFCANL MATRIX FAILURE # Prior Day: 23086
madis/proc/sfc/ao/output/sfcanl.out.230860000_2:
madis/proc/sfc/ao/output/sfcanl.out.230860000_3:
madis/proc/sfc/ao/output/sfcanl.out.230862200_2:
COMPARE Mesonet data providers cnt 48 for 20230328_1800:
COMPARE DECODE STN TBLS 149 Newest: 214481 Mar 28 16:30 LDAD/data/RAWSStation.txt Oldest: 8848 Sep 12 2019 LDAD/data/AWS-GANetStation.txt
COMPARE PRE-DECODE STN TBLS 149 Newest: 214481 Mar 28 16:30 LDAD/data/master/RAWSStation.txt Oldest: 1568 Sep 12 2019 LDAD/data/master/AWS-KSNetStation.txt
COMPARE pqsend entries 6 (Only on primary, 1 for each feedtype/distr)
COMPARE Surface QC text products 7320 hour (19): Newest: Mar 28 20:27 madis/proc/sfc/ao/qc122/qcsumry20.dat Oldest:
COMPARE sda IO Writes 2308718 Local Disk MB/s: 24 RS MB/s: 0 PQ MB/s: 23+ Tot secs: 3446+
COMPARE sda IO Reads(2308718 Local Disk MB/s: 1 RS MB/s: 0 PQ MB/s: 0+ Tot secs: 3446+
COMPARE NetCDF REC Writes 2308718 Avg DECODE rec/s: 85 Avg AO rec/s: 80 Avg NON-SFC rec/s: 11 Tot secs: 3446
INFO Network 2308718 eth0 15 MB/s TX 3 MB/s RX Tot secs: 3446
COMPARE usr1 Disk 2308718 24 WRITE MB/s | 1 READ MB/s | 5992 Avg WRT TPS | 15396 Max 1-min WRT TPS
COMPARE Rough compare sizes for same hour on known good system. The ext fields are #recNum and file-size. Note: Max #recNum has to be < MXST_P 280000
AO-20230328_18 sfc/ao/qcms_data/buoy/20230328_1800 972 1250436
AO-20230328_18 sfc/ao/qcms_data/crn/20230328_1800 245 434620
AO-20230328_18 sfc/ao/qcms_data/hcn/20230328_1800 102 199844
AO-20230328_18 sfc/ao/qcms_data/hfmetar/20230328_1800 11987 18865044
AO-20230328_18 sfc/ao/qcms_data/hydro/20230328_1800 9796 12288780
AO-20230328_18 sfc/ao/qcms_data/mesonet/20230328_1800 250192 550944912
AO-20230328_18 sfc/ao/qcms_data/metar/20230328_1800 7923 11794724
AO-20230328_18 sfc/ao/qcms_data/nepp/20230328_1800 72 177472
AO-20230328_18 sfc/ao/qcms_data/rwis/20230328_1800 5863 20946308
AO-20230328_18 sfc/ao/qcms_data/urbanet/20230328_1800 5917 16096812
NS-20230328_18 nonsfc/madis_data/LDAD/profiler/netCDF/20230328_1800 46 2357608
NS-20230328_18 nonsfc/madis_data/LDAD/profiler/netCDF/20230328_1830 36 1854168
NS-20230328_18 nonsfc/madis_data/LDAD/snow/netCDF/20230328_1800 2258 2547388
NS-20230328_18 nonsfc/madis_data/point/acars/netcdf/20230328_1800 36020 41465896
NS-20230328_18 nonsfc/madis_data/point/acarsProfiles/netcdf/20230328_1800 349 50835336
NS-20230328_18 nonsfc/madis_data/point/HDW/netcdf/20230328_1800 28 1274924
NS-20230328_18 nonsfc/madis_data/point/raob/netcdf/20230328_1800 19 1412716
AO-20230328_17 sfc/ao/qcms_data/buoy/20230328_1700 885 1151604
AO-20230328_17 sfc/ao/qcms_data/crn/20230328_1700 245 434620
AO-20230328_17 sfc/ao/qcms_data/hcn/20230328_1700 102 199844
AO-20230328_17 sfc/ao/qcms_data/hfmetar/20230328_1700 11963 18827700
AO-20230328_17 sfc/ao/qcms_data/hydro/20230328_1700 9955 12471948
AO-20230328_17 sfc/ao/qcms_data/mesonet/20230328_1700 249501 549427476
AO-20230328_17 sfc/ao/qcms_data/metar/20230328_1700 8047 11976260
AO-20230328_17 sfc/ao/qcms_data/nepp/20230328_1700 72 177472
AO-20230328_17 sfc/ao/qcms_data/rwis/20230328_1700 5978 21335008
AO-20230328_17 sfc/ao/qcms_data/urbanet/20230328_1700 5933 16139372
NS-20230328_17 nonsfc/madis_data/LDAD/profiler/netCDF/20230328_1700 44 2256920
NS-20230328_17 nonsfc/madis_data/LDAD/profiler/netCDF/20230328_1730 34 1753480
NS-20230328_17 nonsfc/madis_data/LDAD/snow/netCDF/20230328_1700 2303 2592568
NS-20230328_17 nonsfc/madis_data/point/acars/netcdf/20230328_1700 37403 43048048
NS-20230328_17 nonsfc/madis_data/point/acarsProfiles/netcdf/20230328_1700 403 58694496
NS-20230328_17 nonsfc/madis_data/point/HDW/netcdf/20230328_1700 228 1293324
NS-20230328_17 nonsfc/madis_data/point/raob/netcdf/20230328_1700 3 317420
DECODE decode/data/fxa/LDAD/crn/netCDF/20230328_1800 245 315420
DECODE decode/data/fxa/LDAD/hcn/netCDF/20230328_1800 102 139672
DECODE decode/data/fxa/LDAD/hfmetar/netCDF/20230328_1800 11987 13281004
DECODE decode/data/fxa/LDAD/hydro/netCDF/20230328_1800 9918 8246952
DECODE decode/data/fxa/LDAD/mesonet/netCDF/20230328_1800 251973 311900176
DECODE decode/data/fxa/LDAD/nepp/netCDF/20230328_1800 72 111828
DECODE decode/data/fxa/LDAD/rwis/netCDF/20230328_1800 5894 15673748
DECODE decode/data/fxa/LDAD/snow/netCDF/20230328_1800 2261 1852784
DECODE decode/data/fxa/LDAD/urbanet/netCDF/20230328_1800 5917 5327196
IQC sfc/iqc1/qcms_data/buoy/20230328_1800 728 973252
IQC sfc/iqc1/qcms_data/crn/20230328_1800 153 301036
IQC sfc/iqc1/qcms_data/hcn/20230328_1800 0 69692
IQC sfc/iqc1/qcms_data/hfmetar/20230328_1800 3535 5713732
IQC sfc/iqc1/qcms_data/hydro/20230328_1800 3692 5256972
IQC sfc/iqc1/qcms_data/mesonet/20230328_1800 87816 194367216
IQC sfc/iqc1/qcms_data/metar/20230328_1800 3744 5676668
IQC sfc/iqc1/qcms_data/nepp/20230328_1800 0 72928
IQC sfc/iqc1/qcms_data/rwis/20230328_1800 1164 5063688
IQC sfc/iqc1/qcms_data/urbanet/20230328_1800 1224 3613432
IQC sfc/iqc2/qcms_data/buoy/20230328_1800 8 155332
IQC sfc/iqc2/qcms_data/crn/20230328_1800 40 136960
IQC sfc/iqc2/qcms_data/hcn/20230328_1800 0 69692
IQC sfc/iqc2/qcms_data/hfmetar/20230328_1800 2847 4643204
IQC sfc/iqc2/qcms_data/hydro/20230328_1800 2000 3307788
IQC sfc/iqc2/qcms_data/mesonet/20230328_1800 62497 138766692
IQC sfc/iqc2/qcms_data/metar/20230328_1800 1495 2384132
IQC sfc/iqc2/qcms_data/nepp/20230328_1800 0 72928
IQC sfc/iqc2/qcms_data/rwis/20230328_1800 2779 10522388
IQC sfc/iqc2/qcms_data/urbanet/20230328_1800 497 1679612
IQC sfc/iqc3/qcms_data/buoy/20230328_1800 53 206452
IQC sfc/iqc3/qcms_data/crn/20230328_1800 18 105016
IQC sfc/iqc3/qcms_data/hcn/20230328_1800 78 169220
IQC sfc/iqc3/qcms_data/hfmetar/20230328_1800 2572 4215304
IQC sfc/iqc3/qcms_data/hydro/20230328_1800 1458 2683404
IQC sfc/iqc3/qcms_data/mesonet/20230328_1800 36215 81051420
IQC sfc/iqc3/qcms_data/metar/20230328_1800 1349 2170388
IQC sfc/iqc3/qcms_data/nepp/20230328_1800 0 72928
IQC sfc/iqc3/qcms_data/rwis/20230328_1800 1068 4739208
IQC sfc/iqc3/qcms_data/urbanet/20230328_1800 2205 6222892
IQC sfc/iqc4/qcms_data/buoy/20230328_1800 141 306420
IQC sfc/iqc4/qcms_data/crn/20230328_1800 33 126796
IQC sfc/iqc4/qcms_data/hcn/20230328_1800 90 184532
IQC sfc/iqc4/qcms_data/hfmetar/20230328_1800 2512 4121944
IQC sfc/iqc4/qcms_data/hydro/20230328_1800 2551 3942540
IQC sfc/iqc4/qcms_data/mesonet/20230328_1800 46478 103588968
IQC sfc/iqc4/qcms_data/metar/20230328_1800 1197 1947860
IQC sfc/iqc4/qcms_data/nepp/20230328_1800 0 72928
IQC sfc/iqc4/qcms_data/rwis/20230328_1800 239 1937188
IQC sfc/iqc4/qcms_data/urbanet/20230328_1800 2604 7284232
IQC sfc/iqc5/qcms_data/buoy/20230328_1800 125 288244
IQC sfc/iqc5/qcms_data/crn/20230328_1800 27 118084
IQC sfc/iqc5/qcms_data/hcn/20230328_1800 0 69692
IQC sfc/iqc5/qcms_data/hfmetar/20230328_1800 2323 3827860
IQC sfc/iqc5/qcms_data/hydro/20230328_1800 2158 3489804
IQC sfc/iqc5/qcms_data/mesonet/20230328_1800 48058 107058648
IQC sfc/iqc5/qcms_data/metar/20230328_1800 1020 1688732
IQC sfc/iqc5/qcms_data/nepp/20230328_1800 72 177472
IQC sfc/iqc5/qcms_data/rwis/20230328_1800 1322 5597728
IQC sfc/iqc5/qcms_data/urbanet/20230328_1800 1212 3581512