Dataverse

What is Data Purification?

Data Purification is the act of cleaning up your Dataverse.

This includes deduplication and merging as well as normalizing bad values, but can also include enrichment of your data with information from other sources and verifying email addresses and phone numbers. 

Another part of this process is identifying ingestion points of bad or redundant data, in addition to an archiving strategy to move historical or inactive data to a more appropriate store, and deleting data with no legitimate business value.

Dataverse Capacity Management

Dataverse capacity is divided into 3 primary pools, database, logs and documents. Any one of these 3 pools being over-capacity can present issues for your organization.

If your organization has the need to audit user access to records for example, you would have activated audit logging. Historical logs are what consume the log capacity pool. These may not be something you can simply delete. Instead these can be automatically archived to low-cost storage on a schedule.

Many organizations create documents in the business application process. Documents can consume quite a bit of capacity. There are several options for storing the actual capacity consuming documents elsewhere or simply an automated deletion procedure for no longer needed documents… like previous versions of an accepted agreement for example.

Database capacity is the pool that most organizations struggle with. This is made up of the actual pieces of data that you have collected of course, but also includes solutions files and application programs that you may be using. While Data Purification will handle the former, the latter requires analysis. Inefficient solution design can have a significant impact on your database capacity. Many solutions, or components of solutions were designed before capacity was a concern. Today, “Capacity-based design”, as well as “Licensed-based design” have been added as new strategies in solution development circles.