> For the complete documentation index, see [llms.txt](https://docs.dh3.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.dh3.io/specs/other/disaster-recovery.md).

# Disaster recovery

#### Introduction

This disaster recovery plan details the steps to be taken in response to reorganizations (reorgs) and disruptions within the Solana blockchain network as they affect [DH3.io](http://DH3.io) operations. The aim is to ensure a systematic and reliable approach to managing potential inconsistencies and maintaining the integrity of both real-time streams and historical data within the [DH3.io](http://DH3.io) ecosystem.

#### Regular Monitoring Procedures

* Validator Consistency Check: Every 5 seconds, a consistency verification is carried out among our validators to ensure they are in agreement on slot positions and block hashes.
* External RPC Verification: At 30-second intervals, an additional check with a randomly selected RPC from our list of trusted sources (e.g., QuickNode) is performed to confirm data consistency with external networks.

#### Disaster Recovery Plan for Real Time Streams and Historical Database

* Initial Response:
  * In the event of detected inconsistencies, the Kubernetes operator will halt all related instances to prevent the spread of erroneous data.
* Issue Assessment:
  * Determine whether the inconsistency is due to a global blockchain event or localized to our validators.
* Localized Event Handling:
  * If the issue is found to be local, the affected validator is removed from the pool.
* Identifying Last Correct Block ID:
  * Establish the most recent valid block ID before the discrepancy was detected.
* Data Correction:
  * Delete incorrect records from HBase (RPC) and HDFS archives. Thanks to our LiteRPC, this process allows for the straightforward rollback and deletion of inaccurate data.

#### Disaster Recovery Plan for User Generated Data Sets

* Data Set Generation Delay:
  * For most scenarios, it is recommended to delay the generation of data sets for 5-10 minutes. This delay, combined with the cessation of all related downstream processes and jobs by our Kubernetes operator, usually suffices to address any inconsistencies.
* Real-Time Data Set Handling:
  * For real-time data sets, the same corrective measures are applied as with the historical database, involving the deletion of erroneous blocks of data.
* Data Integrity and Duplication Prevention:
  * Each record within the data sets is assigned a unique ID, derived from the block timestamp, transaction position within the block, and several other characteristics. This unique identifier helps quickly eliminate unwanted data and acts as a measure against data duplication.

#### Conclusion

This disaster recovery plan is crucial for maintaining the operational integrity and reliability of [DH3.io](http://DH3.io) in the face of Solana blockchain reorganizations. By following these procedures, [DH3.io](http://DH3.io) ensures the consistency, accuracy, and security of both the real-time and historical data within its Web3 data hub.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.dh3.io/specs/other/disaster-recovery.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.