HADS Intranet Processing Status for Thu Apr 17 03:32:01 GMT 2025 on vm-bldr-madis-ops-hadsi01.ncep.noaa.gov
03:32:01 up 27 days, 15:23, 0 users, load average: 0.13, 0.25, 0.33
************************************
*** IS LDM RUNNING ***
************************************
ldmd procs....PASS
pqact proc....PASS
******************************************
*** ARE DATABASE TABLES LOADED/LOADING ***
******************************************
dcp table loaded?...................................PASS
plat4mat table loaded?..............................PASS
data_param table loaded?............................PASS
report_defs table loaded?...........................PASS
wfo_pe_codes table loaded?..........................PASS
rfc_pe_codes table loaded?..........................PASS
pp4pp table loaded and current?.....................PASS
pc4pp table loaded and current?.....................PASS
ta4tntx table loaded and current?...................PASS
rawdata table loaded and current?...................PASS
hg table loaded and current?........................PASS
New afws_data_partners table loaded and current?....PASS
New afws_contacts table loaded and current?.........PASS
Checking Database Performance (may take a min or 2)...
*******************************************************
*** NEXHADS and DECODEDHADS DBs PERFORMANCE SUMMARY ***
*******************************************************
nexhads time elapsed is 1.345, which is < 5.0 seconds?....PASS
decodedhads time elapsed is 19.110, which is < 60.0 seconds?....PASS
*****************************
*** INCOMING DATA ***
*****************************
/opt/ldm/madis/hads/data/DOMSAT1_incoming....PASS
/opt/ldm/madis/hads/data/NWSTG_incoming......PASS
/opt/ldm/madis/hads/data/USGS_incoming.......PASS
*************************************************
*** ARE WE DECODING THE INCOMING RAW DCP DATA ***
*************************************************
/opt/ldm/madis/hads/data/StatusLog.......PASS
The Processed Directory........PASS
The Processed SQL Directory....PASS
*****************************************
*** Are RRS Products being generated? ***
*****************************************
RRS products from decoders.............................................PASS
RRS from deriving PPH from PC..........................................PASS
RRS from deriving PPC from PP..........................................PASS
RRS from deriving max/min temperature on the hydrologic day 12Z-12Z....find: ‘/opt/ldm/madis/hads/shef_reports/temp/saved_reports/*_003’: No such file or directory
FAIL
*** Please note if install occurs between 20Z-12Z, FAIL will be seen until after 12Z.
**********************************
*** PLATFORM DESCRIPTION TABLE ***
**********************************
Is pdts flat file being updated in DADDS?.............PASS
Is pdts flat file being updated in network............PASS
Is DADDS pdts flat file size greater than zero?.......PASS
Is network pdts flat file size greater than zero?.....PASS
*************************************
*** CHECK DATABASE BACKUP CURRENT ***
*************************************
Are there more than 50 files in /opt/ldm/madis/hads/database/backup/daily...PASS
Are the files in /opt/ldm/madis/hads/database/backup/daily current..........PASS
Are the files in /opt/ldm/madis/hads/database/backup/daily/Sun current......PASS
Are the files in /opt/ldm/madis/hads/database/backup/daily/Mon current......PASS
Are the files in /opt/ldm/madis/hads/database/backup/daily/Tue current......PASS
Are the files in /opt/ldm/madis/hads/database/backup/daily/Wed current......PASS
Are the files in /opt/ldm/madis/hads/database/backup/daily/Thu current......PASS
Are the files in /opt/ldm/madis/hads/database/backup/daily/Fri current......PASS
Are the files in /opt/ldm/madis/hads/database/backup/daily/Sat current......PASS
***Please note that immediately after an install and out to 7 days, FAIL may be seen.
******************************************
*** CHECK USGS CROSS REFERENCE CURRENT ***
******************************************
Is the USGS Cross Reference SQL being created M-F..........PASS
Is the USGS Cross Reference flat file being created M-F....PASS
****************************************************
*** CHECK PERIODIC TABLE PUSHES TO PROCESSING VM ***
****************************************************
Is the nowant table updating weekly (Tue)...........PASS
Is the newdcps table updating weekly (Tue)..........PASS
Is the route_dcps table updating weekly (Tue).......PASS
Is the filters table updating weekly (Tue)..........PASS
Is the dcp_deletions table updating weekly (Tue)....PASS
Is the report_record table updating weekly (Tue)....PASS
Is the nesdis_owner_id_xref table updating daily....PASS
*** Please note after install, a FAIL may be seen for up to 7 days, if Tues update, and 24 hours for a daily update.
****************************************
*** ARE DATABASE TABLES BEING PURGED ***
****************************************
hg table purging?........PASS
pc table purging?........PASS
vb table purging?........PASS
rawdata table purging?...PASS
pc4pp table purging?.....PASS
pp4pp table purging?.....PASS
ta4tnx table purging?....PASS
*************************************
*** ARE SYSTEM FILES BEING PURGED ***
*************************************
Is /opt/ldm/madis/hads/logs being purged?....................PASS
Is /opt/ldm/madis/hads/shef_reports/archive being purged?....PASS
Is /opt/ldm/madis/hads/data/NWSTG_incoming being purged?.....PASS
Is /opt/ldm/madis/hads/data/processed?.......................PASS
Is /opt/ldm/madis/hads/data/queuedSQL?.......................PASS
Is /opt/ldm/madis/hads/data/processedSQL?....................PASS
Is /opt/ldm/madis/hads/data/SutronSTBinary/processed?........PASS
Is /opt/ldm/madis/hads/data/Sutron8200ASCII/processed?.......PASS
Is /opt/ldm/madis/hads/data/ASCII_NonSutron/processed?.......PASS
Is /opt/ldm/madis/hads/data/Binary_NoOffset/processed?.......PASS
pullUSGS purging
Is /opt/ldm/madis/hads/pullUSGS/original_USGS_SHEF/long_term_archive?.....................................PASS
Is /opt/ldm/madis/hads/pullUSGS/original_USGS_SHEF/short_term_archive?....................................PASS
Is /opt/ldm/madis/hads/pullUSGS/SQL/original_USGS/?.......................................................PASS
Is /opt/ldm/madis/hads/pullUSGS/SQL/processed_USGS/?......................................................PASS
Is /opt/ldm/madis/hads/pullUSGS/processing/process_height/archive/?.......................................PASS
Is /opt/ldm/madis/hads/pullUSGS/processing/process_height/corrected_files?................................PASS
Is /opt/ldm/madis/hads/pullUSGS/processing/process_PP/archive/?...........................................PASS
Is /opt/ldm/madis/hads/pullUSGS/processing/process_PP/corrected_files?....................................PASS
Is /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF/archive/adjusted_USGS_sheffiles?...........PASS
Is /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF/archive/CorrectedFiltered?.................PASS
Is /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF/archive/corrected_height_pp?...............PASS
Is /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF/archive/Filtered?..........................PASS
Is /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF/archive/SHEFFiles?.........................PASS
Is /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF/long_term_archive/adjusted_USGS_sheffiles?.PASS
Is /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF/long_term_archive/CorrectedFiltered?.......PASS
Is /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF/long_term_archive/corrected_height_pp?.....PASS
Is /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF/long_term_archive/Filtered?................PASS
Is /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF/long_term_archive/SHEFFiles?...............PASS
************************************
*** IS COMMON DATA BEING PURGED ***
************************************
push results files being purged.......PASS
rollback being purged from history....PASS
************************************
*** ARE EMAILS CREATED & SENT? ***
************************************
Are "Changes in USGS DCP definitions - from the USGS" being created and sent...PASS
Are "Results of DADDS Processing"..............................................PASS
Are "15 minute - HADS vs USGS Drawdowns and Corrections".......................PASS
Are "DAILY - HADS vs USGS Drawdowns and Corrections"...........................PASS
************************************
*** Directory Check ***
************************************
/opt/ldm/madis/hads/data
Does /opt/ldm/madis/hads/data/DOMSAT1_incoming exist?............................PASS
Does /opt/ldm/madis/hads/data/NWSTG_incoming exist?..............................PASS
Does /opt/ldm/madis/hads/data/REPROCESS_incoming exist?..........................PASS
Does /opt/ldm/madis/hads/data/USGS_incoming exist?...............................PASS
Does /opt/ldm/madis/hads/data/sortraw_input exist?...............................PASS
Does /opt/ldm/madis/hads/data/sortraw_input/newdcps symlink exist?...............PASS
Does /opt/ldm/madis/hads/data/sortraw_input/nowant symlink exist?................PASS
Does /opt/ldm/madis/hads/data/sortraw_input/route_dcps symlink exist?............PASS
Database backup dirs
Does /opt/ldm/madis/hads/database/backup/daily/Sun exist?........................PASS
Does /opt/ldm/madis/hads/database/backup/daily/Mon exist?........................PASS
Does /opt/ldm/madis/hads/database/backup/daily/Tue exist?........................PASS
Does /opt/ldm/madis/hads/database/backup/daily/Wed exist?........................PASS
Does /opt/ldm/madis/hads/database/backup/daily/Thu exist?........................PASS
Does /opt/ldm/madis/hads/database/backup/daily/Fri exist?........................PASS
Does /opt/ldm/madis/hads/database/backup/daily/Sat exist?........................PASS
shef_reports dirs
Does /opt/ldm/madis/hads/shef_reports/temp/current_reports exist?................PASS
Does /opt/ldm/madis/hads/shef_reports/temp/saved_reports exist?..................PASS
Does /opt/ldm/madis/hads/shef_reports/archive exist?.............................PASS
Does /opt/ldm/madis/hads/shef_reports/forBackupFeed exist?.......................PASS
/opt/ldm/madis/hads/network
Does /opt/ldm/madis/hads/network/ASCII_Sutron8200 symlink exist?.................PASS
Does /opt/ldm/madis/hads/network/BINARY_Sutron symlink exist?....................PASS
Does /opt/ldm/madis/hads/network/newdcps symlink exist?..........................PASS
Does /opt/ldm/madis/hads/network/nowant symlink exist?...........................PASS
Does /opt/ldm/madis/hads/network/route_dcps symlink exist?.......................PASS
Does /opt/ldm/madis/hads/network/HADSvsUSGS_USACECoords symlink exist?...........PASS
pullUSGS dirs
Does /opt/ldm/madis/hads/pullUSGS exist?.........................................PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files exist?..............................PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/daily_backups exist?................PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/daily_backups/Sun exist?............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/daily_backups/Mon exist?............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/daily_backups/Tue exist?............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/daily_backups/Wed exist?............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/daily_backups/Thu exist?............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/daily_backups/Fri exist?............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/daily_backups/Sat exist?............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/realtime backups exist?.............PASS
Does /opt/ldm/madis/hads/pullUSGS/logs exist?....................................PASS
Does /opt/ldm/madis/hads/pullUSGS/original_USGS_SHEF exist?......................PASS
Does /opt/ldm/madis/hads/pullUSGS/original_USGS_SHEF/long_term_archive exist?....PASS
Does /opt/ldm/madis/hads/pullUSGS/original_USGS_SHEF/short_term_archive exist?...PASS
Does /opt/ldm/madis/hads/pullUSGS/processing exist?..............................PASS
Does /opt/ldm/madis/hads/pullUSGS/processing/preprocess_USGS_SHEF exist?.........PASS
Does /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF exist?..........PASS
Does /opt/ldm/madis/hads/pullUSGS/processing/processed_USGS_SHEF exist?..........PASS
Does /opt/ldm/madis/hads/pullUSGS/processing/process_height exist?...............PASS
Does /opt/ldm/madis/hads/pullUSGS/processing/process_PP exist?...................PASS
Does /opt/ldm/madis/hads/pullUSGS/processing/temp_SHEF exist?....................PASS
Does /opt/ldm/madis/hads/pullUSGS/processing/temp_SHEF/corrections exist?........PASS
Does /opt/ldm/madis/hads/pullUSGS/processing/temp_SHEF/filter exist?.............PASS
filter_files & flat_files dirs
Does /opt/ldm/madis/hads/flat_files exist?.......................................PASS
Does /opt/ldm/madis/hads/filter_files exist?.....................................PASS
Does /opt/ldm/madis/hads/filter_files/ASCII_Sutron8200 symlink exist?............PASS
Does /opt/ldm/madis/hads/filter_files/BINARY_Sutron symlink exist?...............PASS
Does /opt/ldm/madis/hads/filter_files/HADSvsUSGS_USACECoords symlink exist.......PASS
/common/data/apps/madis/current/private/to_ingest/ dirs
Does /common/data/apps/madis/current/private/to_ingest//UsgsToShef exist?.........PASS
Does /common/data/apps/madis/current/private/to_ingest//idb exist?................PASS
Does /common/data/apps/madis/current/private/to_ingest//pullUSGS exist?...........PASS
Does /common/data/apps/madis/current/private/to_ingest//dump_a_table exist?.......PASS
/common/data/apps/madis/current/private/ihads/hot2cold dirs
Does hot2cold/filter_files exist?.........................................PASS
Does hot2cold/log exist?..................................................PASS
Does hot2cold/pullUSGS exist?.............................................PASS
Does hot2cold/pullUSGS/flat_files exist?..................................PASS
Does hot2cold/pullUSGS/flat_files/daily_backups exist?....................PASS
Does hot2cold/pullUSGS/flat_files/daily_backups/Sun exist?................PASS
Does hot2cold/pullUSGS/flat_files/daily_backups/Mon exist?................PASS
Does hot2cold/pullUSGS/flat_files/daily_backups/Tue exist?................PASS
Does hot2cold/pullUSGS/flat_files/daily_backups/Wed exist?................PASS
Does hot2cold/pullUSGS/flat_files/daily_backups/Thu exist?................PASS
Does hot2cold/pullUSGS/flat_files/daily_backups/Fri exist?................PASS
Does hot2cold/pullUSGS/flat_files/daily_backups/Sat exist?................PASS
Does hot2cold/pullUSGS/flat_files/realtime_backups exist?.................PASS
Does hot2cold/pullUSGS/logs exist?........................................PASS
Does hot2cold/pullUSGS/processing exist?..................................PASS
Does hot2cold/pullUSGS/processing/processed_USGS_SHEF exist?..............PASS
Does hot2cold/pullUSGS/processing/processed_USGS_SHEF/archive exist?......PASS
Does hot2cold/pullUSGS/processing/processed_USGS_SHEF/archive/adjusted_USGS_sheffiles exist?...PASS
Does hot2cold/pullUSGS/processing/processed_USGS_SHEF/archive/CorrectedFiltered exist?.........PASS
Does hot2cold/pullUSGS/processing/processed_USGS_SHEF/archive/corrected_height_pp exist?.......PASS
Does hot2cold/pullUSGS/processing/processed_USGS_SHEF/archive/Filtered exist?..................PASS
Does hot2cold/pullUSGS/processing/processed_USGS_SHEF/archive/SHEFFiles exist?.................PASS
************************************
*** File Checks ***
************************************
/opt/ldm/madis/hads/pullUSGS/flat_files
Does /opt/ldm/madis/hads/pullUSGS/flat_files/filter_sites exist?....................PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/dis_list exist?........................PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/PPI_corrections exist?.................PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/hgira_corrections exist?...............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/hgirb_corrections exist?...............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/hgirc_corrections exist?...............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/hpira_corrections exist?...............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/hpirb_corrections exist?...............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/hpirc_corrections exist?...............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/htira_corrections exist?...............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/htirb_corrections exist?...............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/htirc_corrections exist?...............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/hbira_corrections exist?...............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/hbirb_corrections exist?...............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/hmira_corrections exist?...............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/hmirb_corrections exist?...............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/pre_corrections exist?.................PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/SHEF_corrections exist?................PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/xref exist?............................PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/NwsUsgsLookup.json exist?..............PASS
Files destined for Ingest during a failover:
Does /opt/ldm/madis/hads/pullUSGS/flat_files/available_USGSIDs exist?...............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/Daily_USGS_Data_Stats exist?...........PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/PerformanceTiming exist?...............PASS
Does /opt/ldm/madis/hads/pullUSGS/flat_files/reliability.log exist?.................PASS
Tar file created for public to be used for sync'ing dev/QA with OPS:
/common/data/apps/madis/current/public/hads/iproc/pullUSGS.tar.gz exist and updating?......PASS
************************************************
*** ARE /COMMON HOT2COLD FILES BEING UPDATED ***
************************************************
Files in /common/data/apps/madis/current/private/ihads/hot2cold:
hot2cold/data/StatusLog...............................................PASS
hot2cold/pullUSGS/logs................................................PASS
hot2cold/pullUSGS/flat_files..........................................PASS
hot2cold/pullUSGS/processing/processed_USGS_SHEF/archive/SHEFFiles....PASS
********************************************************************
*** FILES UPDATED VIA INTRANET WEB AND TRANSFERRED TO PROCESSING ***
********************************************************************
Is nowant flat file being updated?
NOTE: This file could be days old, but should update within 30 minutes
of an update to nowant via HADS intranet web.
-rw-rw-r--. 1 ldm ldm 135918 Apr 16 20:55 /opt/ldm/madis/hads/flat_files/nowant
------------------------------------------------------
Is newdcps flat file being updated?
NOTE: This file could be days old, but should update within 30 minutes
of an update to newdcps via HADS intranet web.
-rw-rw-r--. 1 ldm ldm 56826 Apr 16 17:00 /opt/ldm/madis/hads/flat_files/newdcps
------------------------------------------------------
Is route_dcps flat file being updated?
NOTE: This file could be days old, but should update within 10 minutes
of an update to route_dcps via HADS intranet web.
-rw-rw-r--. 1 ldm ldm 7562 Apr 11 20:36 /opt/ldm/madis/hads/flat_files/route_dcps
------------------------------------------------------
Is ASCII_Sutron8200 flat file being updated?
NOTE: This file could be days old, but should update within 30 minutes
of a manual entry into ASCII-Sutron filters via HADS intranet web
-rw-rw-r--. 1 ldm ldm 3933 Apr 17 03:00 /opt/ldm/madis/hads/flat_files/ASCII_Sutron8200
------------------------------------------------------
Is BINARY_Sutron flat file being updated?
NOTE: This file could be days old, but should update within 30 minutes
of a manual entry into BINARY-Sutron filters via HADS intranet web.
-rw-rw-r--. 1 ldm ldm 8325 Apr 17 03:00 /opt/ldm/madis/hads/flat_files/BINARY_Sutron
------------------------------------------------------
Is HADSvsUSGS_USACECoords flat file being?
NOTE: This file could be days old, but should update within 30 minutes
of a manual entry into HADSvsUSGS-USACECoords via HADS intranet web.
-rw-rw-rw-. 1 ldm ldm 27 Dec 12 23:12 /opt/ldm/madis/hads/flat_files/HADSvsUSGS_USACECoords
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ +
+ Summary of the number of FAILs flagged: 1 +
+ +
+ Finished at: Thu Apr 17 03:32:29 GMT 2025 +
+ +
+ Notes: +
+ +
+ 1. WFO-KEY is often a FAIL due to no DCPs defined in their area. If these are the only FAILS, +
+ then this is good +
+ 2. If early in the install, there will be some FAILs until enough time has passed for jobs to run +
+ (examples.. backups, some web page updates, purging) +
+ +
+ +
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++