Formatted Title

Standardizing High-Resolution Site Characterization Data Processing and Visualization

Background/Objectives

Background/Objectives:  

Environmental Resources Management (ERM) uses high-resolution site characterization (HRSC) data to visualize detailed geologic, hydrogeologic and contaminant profiles in a combined log format. These combined logs display results from multiple methodologies, at the requisite scale, as a singular output providing a more complete picture of conditions governing contaminant distribution and transport at monitored locations across a client’s site. The raw data for the combined logs are often submitted by multiple subcontractors, each using different file formats which frequently contain inconsistent parameter names. This imposed a significant data cleansing burden before upload to an ERM EQUIS™ database. As a result, an end-to-end workflow was needed to address these problems proactively and efficiently. The aim is to use existing tools to minimize customization and manual intervention while standardizing the processing, storage, and reporting of HRSC data.

Approach/Activities

Approach/Activities:

HRSC subject matter experts (SMEs) worked with data management and visualization practitioners to identify the HRSC methods used and the data they yield at client sites. This information was used to create a data dictionary which mapped the source data to standard EQUIS™ tables and fields while establishing valid value ranges and field descriptions to be used in the target database. By aligning incoming data with EQUIS™ standards, data managers are able use EQUIS’s out-of-the-box reports instead of creating custom reports. The use of standard reporting capabilities eliminates the cost and effort associated with maintaining custom reports through the application upgrade cycles.  

Next, experienced data managers and visualization practitioners reviewed the proposed workflow and identified a data processing gap associated with the creation of EQUIS™-compatible import files. It was determined that a sufficient workflow existed for most data types, but automating the transformation of geophysical data through a Python script would eliminate the tedious manual processing used in the current workflow. This automation increases speed and accuracy and allows data managers to leverage the use of the standard exports while automating the addition or removal of valid values as required. 

With reliable and consistent data upload and processing accomplished, the visualization team and HRSC SMEs finalized calculations needed to create the combined log visualizations and completed the new standard operating procedure for the handling of HRSC data at ERM.

Results/Lessons Learned

Results/Lessons Learned: 

ERM used a multi-disciplinary team to capture institutional knowledge and develop a new workflow. By proactively aligning incoming data with a standard database, we dramatically reduced the extent of manual intervention, improved data quality, and can consistently deliver an impactful side-by-side comparison of HRSC data. The creation of a standard approach can be more easily supported by data managers than its ad-hoc predecessors and reduces data analysis and visualization costs to project teams and clients while still delivering the robust visualization required by SMEs. Feedback will be collected as the workflow is adopted and used to strengthen data governance around data input variations and future refinements to the combined log output.