Each year billions of dollars are spent monitoring the environment providing valuable data that enables us to manage our resources effectively. What happens to this valuable data once it is collected ? To quote from the US EPA:
"On the order of 20 percent of the budget allocated for the monitoring program should be reserved for data management and data analysis activities. Failure to plan for these costs can result in the loss of information due to inadequate data preservation and limited analysis of the monitoring data that are collected."
This document discusses the value of data and how to get the most out of it.
Collecting Data is Expensive - It costs between $5000 and $10000 to run an average monitoring station for a year. It is important to protect this investment.
Data is valuable - Reliable data is essential to manage your resources effectively. It is important to get the most out of your data.
Data Management - To get the most out of your data you need a system in place to provide quick and easy access to all of your data. For small amounts of data you could use Excel. When you have to deal with large amounts of data you may encounter special problems that need a specialized solution.
Managing large amounts of Data
Some common problems with managing large amounts of data are:
consolidating data from different sources
combining different types of data
keeping all of your data online and accessible
analysing data quickly
integrating different systems
assessing the quality of your data
A good data management system must
provide quick and easy access to all of your data.
Hydstra stores data efficiently reducing the amount of storage required and improving the speed at which it can be processed. Hydstra can keep tens of thousands of station years of data on a local hard disk and analyse a year of continuous data in under 3 seconds on a Pentium.
consolidate data from different sources.
Hydstra can import data from a variety of sources including any data logger, digitized charts, telemetry systems and spreadsheets. You can easily import data on a one-off basis or set up the system to routinely import data.
consolidate different types of data.
Hydstra allows you to combine your time-series data, water quality data, groundwater bore information and geographical information into a single integrated archive.
provide facilities for editing and reviewing data.
In a perfect world data would always be clean and accurate but in reality data needs to be reviewed after it is collected and often edited. Hydstra provides a powerful tool for editing and reviewing data. Facilities include freehand editing, rescaling, zooming, cut & paste and stretching.
allow you to assess the quality of your data.
When you are making multi-million dollar decisions based on your data, you need to know how good the data is. Hydstra allows you to guarantee the quality of your data by storing quality information with each data point. Hydstra also includes facilities for routinely performing data audits and highlighting potential problems in your archive.
make it easy to publish data from your archive.
Hydstra provides a wide range of outputs that can be automatically assembled to publish in Microsoft Word format or on the World Wide Web.
store other information related to your data.
Hydstra stores supporting information including Station Details, Rating Tables and Shifts, Gauging Measurements, Instruments, Cross-Sections, Variable Descriptions, Variable Conversions, Data Quality Descriptions etc.
easily integrate with other systems.
Hydstra makes it easy for you to get data out of your archive to use with other packages. With Hydstra you can extract data from your archive in text format, dBase III database format, CSV format for spreadsheets as well as through a DLL.
Hydstra overcomes the problems involved with managing large amounts of data, giving you access to the information you need to effectively manage water systems.
