# Welcome WEB3 frens ### What is DH3 A fresh new way of how data should be managed, processed, served, and owned. In web2 days in most cases, you were just a person who generated data but had no clue what was going on behind that, here we are trying to make a real difference by empowering all web3 users to participate, educate themself, and earn from it. > *Question: "Could you explain Web3 to a 5-year-old?"* > > *The answer:* > > *Web1 -> Web2 -> Web3 = Read-Only -> Read-Write -> Read-Write-Own* > > *by* [*Chris Dixon*](https://twitter.com/cdixon) *@ a16z* ### Main objectives * Effective storage - [OSS](broken://pages/MxQPMUuMttqBniXePkdv), [No cloud](broken://pages/TCZC9PXLcvOtuTB4FXGV) * Minimize time to create data sets, save weeks or even months for users - [No code tools](broken://pages/j0KTl0jciF4V1csr13h4) * [Data ownership](/dh3-focus/data-ownership) and royalties from your contributions * Prefer rapid development. Distributed from the start, [decentralized](broken://pages/eDp3ZCksPXegqZgIuv7N) in the near feature # Effective storage #### Smart Data Management Strategies 🚀 Managing multi-petabyte databases in the cloud is a real challenge, especially when costs can spiral unexpectedly. While cloud computing promised affordability and simplicity, the reality turned out to be more complex. Cloud prices can shock you, skyrocketing up to 100 times the cost of traditional hosting solutions. This landscape can force you to constantly strategize about cost savings, putting a damper on your innovation and big data projects. #### Leveraging Modern Software Foundations Thankfully, the past decade has ushered in robust solutions like Kubernetes, and container-optimized OSes like FlatCar, revolutionizing how we manage large-scale bare metal Kubernetes clusters. These innovations drastically reduce the manpower needed, empowering us to maintain complex infrastructure more efficiently. #### A Decentralized Vision for Future-Proof Software Our approach embraces Kubernetes and smaller, agile data centers. This decentralized approach ensures no single entity holds the keys to the kingdom, unlike monoliths like BigQuery. It's about distributing power across a tapestry of data centers, small yet mighty. When crafting software, we prioritize vendor-neutral designs to ensure freedom and flexibility for all. Let's build a future where your data scales seamlessly, and your cloud thrives on innovation without boundaries. 🌟 # Multichain The data worlds of EVM and Solana are markedly different. Transactions in Solana are transparent, allowing observers to see all transaction effects and account changes on the blockchain. For EVM, this level of transparency is achieved only by programming all transactions and indexing changes that occur during transaction execution. DH3.io goes beyond indexing events for EVM chains, delving deeper into indexing the entire call tree, with all parameters, inter-contract interactions, balance changes, and other critical details needed by analysts. This approach brings the data of EVM chains into close correspondence with the transparency level available for Solana transactions. * DPC # Quick indexing We aim to achieve not only efficient data storage but also rapid data retrieval speeds. The majority of data providers offer ready-to-use data sets or pre-parsed base data sets, constraining users to their binary structure parsing capabilities and imposing a learning curve related to data mapping. This approach often stems from the prohibitive costs associated with processing raw binary data and the resources necessary for constructing these data sets. Direct data fetching from RPC or utilizing cloud storage, which incurs charges for every IO operation, can significantly escalate costs. It is crucial to ensure unrestricted access to foundational unstructured data, mitigating concerns about the number of iterations required to refine the data. We observe numerous teams resorting to the development of data offloaders for their private infrastructure to emulate similar capabilities. This endeavor can be particularly challenging, especially for high throughput chains like Solana, leading teams to expend weeks, or even months, on tasks that our solution aims to reduce to mere minutes. At DH3, we have devoted additional effort to ensure that our core methodologies are scalable and cost-efficient. We have chosen to store all unstructured data in Hadoop, employing our ETL powered by a custom Kubernetes operator that oversees all consumer and data processor activities. Our current infrastructure allows for data rescans at an extraordinary speed of 50 million transactions per second (Tps) at a throughput of 30GBps. Notably, this capacity is not the upper limit, as DH3.io's infrastructure is designed for near-linear scalability. Our entire system is deployed on bare metal, optimizing operational efficiency while eliminating additional IO or network expenses. Impressively, our system can rescan all Solana transactions within a range of 10 to 30 minutes. Access to our rapid indexing capabilities is provided through either the **Backfilling API** or the **GUI Builder**. # Parsing and Decoding #### Advanced Data Decryption and Accessibility This is not just a simple storage of Solana transactions. DH3.io has decrypted a majority of these transactions, making them available in both binary serialized and deserialized formats. By mastering the ability to quickly and cost-effectively rescan the entire Solana, DH3.io has developed a sophisticated transaction and account decoder. This decoder is based on a three-level approach: sourcing data from the blockchain, repositories, and code inference. #### Utilization of IDL/ABI and Decoding Strategies In Solana, the Interface Description Language (IDL) is the standard for meta-information about account and transaction structures. DH3.io harnesses this standard, extracting IDL schemas directly from the blockchain when available. The team has also curated a list of the most popular Solana programs, tracking their GitHub repositories. This allows the analyzer to access the source codes of corresponding programs. In cases where IDL is not published on-chain, DH3.io either connects it from the project itself or, if unavailable, the decoder heuristically analyzes the code to deduce the necessary IDL, enabling the decryption of on-chain data. Here we are using state-of-the-art LLM-based extraction techniques with subsequent cleaning, testing, and recompiling. #### Adaptive and Error-Tolerant Data Processing The platform is built to adapt to the evolution of unfinalized programs, accommodating changes in their structure. Erroneous deserializations are not a concern, as constructing a new dataset derived from the binary immutable dataset of transactions is a relatively inexpensive operation for DH3.io. Currently, the platform is capable of performing arbitrary calculations for each transaction using the Lua language, and there are plans to enhance this feature by enabling the compilation of transaction processing scripts using WebAssembly. #### Data Visualization and Advanced Filtering Capabilities DH3.io offers a vast array of well-structured data, providing the community with tools to build useful, derivative live datasets on top of this array. These datasets can be beautifully visualized through interactive dashboards, APIs, and other data consumption tools. The platform features rapid data filtering capabilities. The filtering condition is intelligently divided into two parts: easily-verifiable conditions that are compiled into prefiltering predicates to discard unnecessary data at the scanning stage, and structural decryption to check the remainder of the condition on already structured and possibly transformed data. The obtained data can then be aggregated, sorted, saved in derivative datasets, and even joined together in some cases. **Private Data** TBD\
# Exploring Innovative Block Explorer Features DH3.io’s block explorer deserves special mention. It allows users to familiarize themselves with the contents of blocks in decrypted form, navigate between programs and accounts, and instantly switch to composing queries. Users can use a transaction under consideration as a sample to derive a filter, capturing similar transactions in a certain sense. The block explorer emphasizes deep deserialization of data structures as needed, with conditions and transformations including expressions containing nested elements {% embed url="" %} # Building #### Revolutionizing Data Management for Developers In the digital universe where data reigns supreme, repetitive tasks can become a colossal time-sink. Picture the all-too-common scenario: you're tasked with perpetually generating dataset X from source Y. With the surge of data flow, especially on platforms like Solana, this repetitive cycle becomes daunting—any hiccup could send you spiraling back to square one. #### DH3—your game-changing ally. Envision a world where you're liberated from the drudgery of backend concerns. We're here to future-proof your data needs, handling the heavy lifting of base datasets maintenance with cost-efficient precision, allowing you to browse, preserve, and deliver data seamlessly. #### Breaking Barriers to Accessibility The true innovation isn't just in managing data efficiently—it's about making these powerful tools universally accessible. In the current climate, advanced blockchain know-how, coupled with SQL or Graph query writing prowess, is a prerequisite. But what if it wasn't? DH3 is redefining this paradigm. Our suite of tools is designed with simplicity and speed at its core. Envision going from a glimpse of data to executing a dataset in under an hour—rescanning the entirety of Solana included. This is the empowerment developers deserve, accelerating your iteration cycle, minimizing cost worries, and unlocking a universe of potential for every individual dev, researcher, and data virtuoso. We pave the path toward a more developer-centric, inspiration-fueled data ecosystem. With DH3, you're not just coding; you're crafting the future. Short demo from MVP: {% embed url="" %} # Data ownership **Developer-Centric Data Discovery Ecosystem** As developers and data enthusiasts, we're often confined to using datasets that gain popularity or are deemed useful by the masses. Typically, the cycle is straightforward: an application gains traction, and only then do the associated datasets become readily accessible. Or alternatively, developers invest time and resources to construct datasets without necessarily being rewarded for their efforts. **Break the Cycle: Ownership and Revenue Sharing** **Introducing a Transformative Approach:** We're breaking away from the norm by offering a platform that not only motivates users to delve into data discovery but also compensates them for their contributions. This system aligns with the principles of open-source development, where the sharing of resources benefits the creator as well as the community at large. **Data Set Ownership:** Creators can claim ownership of their data sets upon creation. These data sets can be made publicly available and accessible via API, allowing seamless integration and combination with other datasets to form more sophisticated data constructs. **Monetization through Utilization:** With a mechanism similar to artist's royalties in the music industry, every instance where a data set is utilized generates revenue for its creator. This not only incentivizes the creation of more datasets but also ensures that creators are rewarded proportionately to the value their work brings to the ecosystem. **Practical Example: Data Sets in the Wild** Consider the early users of DH3, navigating through vast uncharted data territories. One pioneer assembles a data set compiling price indices from on-chain oracle data. This type of data set could become a hot commodity, integrated by countless others into their applications and analyses. As use cases surge, so too does the creator's revenue - a perfect illustration of the scalability and potential rewards of this user-centric data discovery model. # Data Streams At the core of contemporary data solutions, the journey begins with raw data streams. Providers, operating robust RPC servers, embark on the meticulous collection of this invaluable raw data. They are tasked not only with its efficient storage but also with ensuring it remains accessible. The goal is to facilitate the creation of datasets in a manner that is both cost-effective and swift. Data streams, such as transactions, are preserved in their original form, retaining all interactions with programs or smart contracts in their executed state, typically encapsulated within serialized binary structures. In certain instances, like with the Ethereum Virtual Machine (EVM), data streams are augmented with comprehensive traces of smart contract calls. This enrichment process incorporates data that could otherwise not be recreated from the default raw outputs. A recurrent challenge observed amongst numerous data-centric startups is their approach towards data preparation. These entities endeavor to process and offer foundational datasets based on currently available information. However, this information is subject to rapid changes due to updates in ABI or IDL interface descriptions, new releases, or standards. Adapting to these alterations often proves to be not only costly but resource-intensive. It entails extensive data rescans and the reconstruction of datasets. Moreover, for startups leveraging cloud services, this necessity translates into significant expenses and delayed progression. DH3 stands at the forefront of addressing and overcoming these challenges. Our innovative approach ensures unparalleled efficiency in rebuilding and modifying datasets. We have developed a system that permits the continual querying of raw data, incorporating the most current parsing methodologies. This enables developers to focus solely on the strategic aspects of data management, such as the selection and indexing of fields, thereby streamlining the entire process. Through our efforts, we are setting a new standard in data management, one that promises agility, cost-effectiveness, and technological resilience in the ever-evolving landscape of data utilization and analysis. # Storage ### Choice of the storage backend Hadoop has become the backbone of many organizations' big data operations due to its scalable, flexible, and cost-effective nature. Born out of a need to process an ever-growing amount of data, as we see now in WEB3. Advantages of Hadoop: * **Scalability**: Hadoop clusters can be easily scaled up by simply adding more nodes. This means providers can start with the data they have and scale up as their data grows. * **Cost-effective**: Hadoop uses commodity hardware to store large quantities of data, which dramatically reduces the cost per terabyte of storage. * **Flexibility**: Hadoop is not limited to MapReduce, and we use our own Kubernetes operator which can schedule jobs inside and outside the Hadoop cluster letting us maximize all available CPU and network resources. ### DH3 Hadoop operator Deploying Hadoop to Kubernetes can enhance Hadoop's scalability and deployment flexibility further. Kubernetes, an open-source platform for managing containerized workloads and services, facilitates both declarative configuration and automation. Implementing Hadoop on Kubernetes involves containerizing Hadoop services, configuring persistent storage solutions to manage HDFS, and setting up network configurations for seamless communication between the Hadoop components. Data providers already running a custom Kubernetes operator specifically designed for Hadoop deployments, ensuring a smooth deployment process across different bare-metal environments. This allows [providers](/specs/providers) to shift their emphasis towards the expansion of data lakes, thereby minimizing the redundancy of infra. In conclusion, Hadoop's design philosophy of high fault tolerance, cost-effectiveness, and scalability, combined with Kubernetes' dynamic and flexible nature, presents a formidable solution for managing big data. # Streams ### Overview The fundamental structure of our data handling system consists of raw block streams, transaction streams, and account streams to be added soon. Each component is meticulously designed to optimize data retrieval and ensure efficient resource utilization. #### Blocks Raw block streams serve as the bedrock for our data architecture. Blocks are organized in approximately 256MB batches. This organization facilitates targeted data retrieval by segregating data into yearly, monthly, and daily segments, thereby optimizing read operations and minimizing input/output resource strain. #### Transactions Our transaction streams evolve from the raw block streams and are categorized into numerous channels. This segregation helps in isolating specific data types, such as oracles, transactions with incomplete states, and others, to reduce false matches and increase rescan speeds during data extraction. As our system evolves, we anticipate the introduction of additional channels to address specific data retrieval needs with precision. Check out [channels](/specs/data-streams/channels) for details. #### Accounts In development... # Channels ### Streamlining Data Access through Categorized Channels In the realm of blockchain data management, streamlining the process of building datasets is crucial for enhancing efficiency and reducing the computational overhead involved in data retrieval. One effective strategy to achieve this involves categorizing data into various channels. This approach not only simplifies data access but also significantly cuts down on the number of read operations required. ### **Understanding Block and Transaction Streams** Block streams are tailored to operate on a per-network basis, such as `mainnet` and `devnet`. This means that each network has a dedicated channel, streamlining access to its block data. On the other hand, more granular data types, such as transactions and accounts, are divided into multiple streams. These categorizations are designed with specificity in mind; for instance, separating oracle-related transactions from general transaction streams or distinguishing token accounts from more general account streams. This level of organization facilitates targeted data retrieval, making it easier for developers to access the precise information they need without wading through irrelevant data. ### **Navigating the Challenges of Data Duplication** A byproduct of creating these separate channels is the inevitability of data duplication. Given that a single transaction may interact with multiple programs and, therefore, belong to more than one category, it might appear in several streams. While at first glance, duplicating data across channels could seem like a costly strategy in terms of storage, it's a trade-off that pays dividends. For the ecosystems we're contributing to and the data providers integrating with our solutions, handling duplicated data isn't a notable concern. In fact, it's a small price to pay compared to the alternative. Processing vast quantities of unrelated records places a significant strain on network resources and CPU usage. When compared, the cost of additional storage is minimal, especially given the efficiency gains in data retrieval and the consequent reduction in processing overhead. ### **A bit of Storage Philosophy** In an environment where the specifics of decoding and parsing binary data evolves daily, it's crucial for us to adapt and align our strategies accordingly. All data within our channels is stored in its original, unparsed form. This approach underscores a foundational principle of our operation—**parse data on-the-fly**. Relying on real-time parsing allows us to ensure that data sets are reconstructed as swiftly as possible, utilizing the most up-to-date techniques and knowledge derived from the latest updates and contributions within the **DH3 community**. This method not only maximizes efficiency and speed but also ensures that our datasets are enriched using the freshest insights, keeping us and our developers at the leading edge of data handling practices. ### **Current channels** | ID | Description | Providers | | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | --------- | | sol.blocks.mainnet | Full mainnet blocks | DexterLab | | sol.blocks.devnet | Full devnet blocks | DexterLab | | sol.transactions.oracles | Oracles, Pyth, ChainLink. | DexterLab | | sol.transactions.failed | Transactions that failed to execute, are still relevant data for some, as they can reduce lamports or fail just partially. | DexterLab | | sol.transactions.other | Transactions that do not fall into any other category, for now, this is the most relevant stream as it contains the majority of transactions | DexterLab | | sol.accounts.token | TokenProgram owner accounts | DexterLab | | sol.accounts.system | SystemProgram owner accounts | DexterLab | | sol.accounts.other | All other accounts that do not fall in specific channels | DexterLab | | eth.blocks.mainnet | Full mainnet blocks with enriched traces. | Parsiq | | eth.transactions.oracles | Oracles | Parsiq | | eth.transactions.other | Transactions that do not fall into any other category | Parsiq | | eth.transactions.mem | Mempool transactions are only applied with real-time streams. | Parsiq | # Stream interceptors Interceptors are designed to be modular and customizable, enabling developers to assemble them with ease, akin to building Lego blocks. This architecture not only enhances adaptability but also streamlines integration with AI technologies for automation purposes. {% hint style="info" %} A significant portion of dataset configuration is facilitated through a user-friendly Graphical User Interface (GUI), ensuring a seamless and intuitive setup process for stream interceptors. The system is engineered to auto-configure interceptors based on GUI inputs, simplifying initial setup procedures from weeks or months to minutes if compared to some well-known ways of building data sets like subgraphs. {% endhint %} For scenarios requiring advanced logic or data augmentation from third-party sources, developers have the flexibility to introduce custom modules. These modules can be crafted using WebAssembly (WASM), allowing for the integration of complex functionalities and external data into the stream processing workflow. This capability ensures that developers can tailor interceptors to meet specific requirements, providing an avenue for sophisticated data processing and enrichment tasks. Explore more details: * [High overview](/specs/stream-interceptors/interceptors) * [Module types](broken://pages/3F1VwVt7OOWJyeREYX5a) # Interceptors This document provides a brief overview of the construction of StreamInterceptors. Note that this is an initial draft; we are currently collaborating with our early clients to establish the foundational specifications for StreamInterceptors. Before proceeding, it is important to highlight that in most cases, neither dataset builders nor end-users will interact with StreamInterceptors in their raw form. Instead, all interactions will be facilitated through a graphical user interface (GUI). ### Anatomy of interceptor Stream interceptors define how data sets are created and stored. This can range from simply selecting desired transactions and streaming them via webhooks to constructing a full data set structure managed by DH3. Let's break down each section: `spec.stream` This section specifies the initial input of raw data. Depending on the type of interceptor used, it might include various options. `spec.compute` Here you configure the computing resources allocated to the interceptor. As of now, only `ArchiveStreamInterceptor` requires this configuration. `spec.filter` This acts as a filtering layer before utilizing more compute-intensive resources. It efficiently filters through streams without fully extracting or transforming transaction data. `spec.extract` -> `spec.transform` -> `spec.load` When the filter identifies required matches, it sends the matching data through [Extract](/specs/stream-interceptors/interceptors/extracting), [Transform](/specs/stream-interceptors/interceptors/transforming) and [Load](/specs/stream-interceptors/interceptors/loading) steps. This process handles everything from simple transaction parsing and sending through webhooks to complex data extraction involving additional queries to other datasets or live RPCs. For these last three, we offer useful starters, such as extractors for parsing full Solana transactions with extensive program support, transformers to map transaction data to database fields, and various sinks for loaders that support both external and DH3-managed databases. The community can enhance this further by replacing any part with custom logic written in WASM, providing a flexible and powerful platform for tailored solutions. ### Types of interceptors ArchiveStreamInterceptor StreamInterceptor DedicatedStreamInterceptor ### Basic example ```yaml apiVersion: interceptor.dh3.io/v1beta kind: ArchiveStreamInterceptor metadata:    name: 'filter-meta-updates' spec:   stream:     channel: sol.transactions.regular options:      timeRange: [111, 222]     blockRange: [111, 222]   compute:     replicas: 10     limit: # limits per replica       cpu: 1       mem: 1Gi   filter:     - beam: 'dh3/solanaTransactionFilter'       instructions: [{},{}]   extract:     - beam: 'dh3/solanaTransactionParser'       includeFieldsMeta: true       includeAccountsMeta: true       instructions: true       events: true   transform: []   load:     - beam: 'dh3/webhook'       url:         - url1         - urlbc       batch: 100       gzip: true ``` # Selecting stream # Defining compute resources # Filtering # Extracting # Transforming # Loading # Beams Interceptors are responsible for connecting to massive data streams and ensuring stable consumption of the data. They act as a conduit, executing Beams on the data flows. Beams, in this context, represent small, modular pieces of code that are designed to perform specific data processing tasks. Beams, in this context, represent small, modular pieces of code that are designed to perform specific data processing tasks. They can be compared to serverless functions, as they can be combined and orchestrated to achieve the desired data processing results. Beams are designed to be lightweight, flexible, and easy to integrate into data processing pipelines. Currently, the majority of Beams are built within the Interceptor framework by the DH3 team, which provides a structured environment for managing data flows and processing. Shortly anyone will be able to provide their Beams through WASM (WebAssembly) modules, enabling custom logic and extending the capabilities of the data processing system. This will allow for more complex and specialized data processing tasks, such as incorporating external data sources, dictionaries, and other custom logic. # Available # Create a beam coming in Q3-Q4 of 2024 # Helpers * can be used from mapping modules * can be used from WASM # Dictionaries Dictionaries, also known as associative arrays or hash maps, are essential data structures in modern programming, allowing for the storage and retrieval of key-value pairs. * **Data Mapping**: One primary role of dictionaries in data set building processes is to map unknown or source-specific data fields to known, standardized fields in the target schema. This ensures consistency and eases data integration from various sources. * **Transformations**: Dictionaries can be used to perform data transformations, such as converting abbreviations to full forms or translating codes to readable text. * **Data Cleaning and Validation**: They facilitate cleaning and validating data by mapping incorrect or incomplete data to valid entries. For instance, a dictionary can convert common misspellings or error codes to correct values. * **Extendability**: By maintaining mapping rules in dictionaries, data set building processes become easily extendable. New mappings and transformations can be added or updated without changing the core logic, making the process adaptable to evolving data requirements. * **Efficiency**: Using dictionaries for lookups is computationally efficient, which speeds up data processing, especially with large datasets. # Execution * how it's deployed, and executed on Kubernetes through providers # Examples # Data sets At DH3, we position ourselves not as mere data providers, but as facilitators of robust **data infrastructure**. Our focus is on scalability and efficiency, ensuring developers and users can access updates and new data sets promptly, unlike many providers that manage most data sets in-house and slowly. **Our Responsibilities:** We clearly define our role in the data ecosystem. DH3 is committed to delivering raw data, parsing its binary structures, and providing the necessary tools—from comprehensive infrastructure solutions to intuitive GUIs. This approach allows us to avoid the bottleneck often seen in traditional data management systems. Unlike data providers who maintain an extensive amount of data sets in-house, DH3 believes in not limiting users and developers. We understand that maintaining and updating numerous data sets internally can be counter-productive, leading to slow fixes, delayed updates, and limited new data availability. Our model is designed to alleviate these issues: 1. **Community-Driven Data Sets:** With DH3, the community plays a pivotal role in designing and creating data sets. This ensures a broader range of data sets, which are more relevant and up-to-date according to user needs. 2. **Faster Integration:** Developers and users can integrate new data sets and updates more rapidly without waiting for internal processes to catch up. 3. **Scalability:** By not being restricted to a specific number of data sets, DH3 offers unparalleled scalability, making it easier to adapt to the ever-growing demands of data-centric applications. **Types of Data Sets:** * **Base Data Sets:** Foundational data sets that serve as the building blocks for various applications. * **Bring Your Data:** Allows users to integrate their data seamlessly into the DH3 infrastructure. * **Views:** Combined base data sets with your data if needed. # Base data sets # Bring your data # Views # Providers * atomic teams * chain specialists * contributors # API's # Backfilling Backfill offers an unprecedentedly swift method to sift through the entire history of the Solana blockchain to pinpoint and retrieve specific transactions necessary for your application. Imagine possessing the functionality of command-line tools like cat and grep, but supercharged to operate across our robust infrastructure. By harnessing our system, you can execute these operations without the overhead of significant resource consumption on your end. Combing through transactional data has never been more efficient, saving time and computational expense, ensuring that you can focus on what matters most—analyzing the data, not struggling to acquire it. ### Backfiller role at DH3 The Backfiller role constitutes a fundamental component of the DH3 ecosystem, designed to facilitate the backfilling of historical data within user-generated datasets and their subsequent reconstruction or enhancement. Our aim is not to constrain innovation by supplying predefined or ready-made datasets. Rather, we dedicate significant resources to empowering users to define the structure and application of data according to their unique requirements and creative vision. ### How you could use backfilling API for your needs? Developers who wanna go beyond traditionally served data sets which mostly cover NFTs and Tokens with their trading-related info usually end up writing some private data pipelines, burning RPC requests on raw data retrieval, writing binary structure parsers, and so on... It can end up as a costly process not only from a money perspective but also from burning tons of time and missing other opportunities along the way. * save time and resources on raw data collecting * no need to learn and allocate resources for transaction data parsing * no need for fancy infrastructure to scan fast through raw data Currently, backfilling streams all matched and parsed data to your HTTP endpoints and you just need to have a web server and backend for data storage. In feature releases alongside GUI for building data sets, we will introduce the ability to design and store data sets fully on the DH3 data providers' side. #### Leveraging the Backfilling API for Enhanced Data Management For developers seeking to extend their reach beyond the conventional data sets largely comprising NFTs and Tokens, along with their associated trading information, the journey can often involve the arduous task of establishing private data pipelines. This typically entails a significant expenditure of resources towards raw data retrieval, development of binary structure parsers, and an array of additional complexities. Such endeavors not only incur substantial financial costs but also demand extensive time investment, potentially diverting attention from other valuable opportunities. * **Streamline Data Collection Processes**: Eliminate the need for exhaustive raw data collection efforts. * **Simplify Transaction Data Parsing**: Forego the necessity to invest in learning and resource allocation for transaction data parsing. * **Minimize Infrastructure Requirements**: Avoid the complexity of developing sophisticated infrastructure to rapidly navigate through raw data. The current implementation of the backfilling API efficiently delivers all matched and parsed data directly to your HTTP endpoints, requiring only the presence of a web server and a backend for data storage on your part. Future updates will further enhance user experience by introducing a graphic user interface (GUI) for data set construction, coupled with the provision for designing and storing data sets entirely through DH3 data providers.
# HTTP Methods ## API Calls {% openapi src="/files/C28GPGR88YlAJQ0zVX6Y" path="/job/" method="get" %} [backfill\_api.yaml](https://3523412580-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FOmM15VvdyDSaZuim6EH0%2Fuploads%2FtrRGnRMgBzXs3p2Bwqug%2Fbackfill_api.yaml?alt=media\&token=18c9ac3f-6098-482b-8e1b-173fd3e7ad65) {% endopenapi %} {% openapi src="/files/C28GPGR88YlAJQ0zVX6Y" path="/job/" method="post" %} [backfill\_api.yaml](https://3523412580-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FOmM15VvdyDSaZuim6EH0%2Fuploads%2FtrRGnRMgBzXs3p2Bwqug%2Fbackfill_api.yaml?alt=media\&token=18c9ac3f-6098-482b-8e1b-173fd3e7ad65) {% endopenapi %} {% openapi src="/files/C28GPGR88YlAJQ0zVX6Y" path="/job/{id}" method="get" %} [backfill\_api.yaml](https://3523412580-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FOmM15VvdyDSaZuim6EH0%2Fuploads%2FtrRGnRMgBzXs3p2Bwqug%2Fbackfill_api.yaml?alt=media\&token=18c9ac3f-6098-482b-8e1b-173fd3e7ad65) {% endopenapi %} {% openapi src="/files/C28GPGR88YlAJQ0zVX6Y" path="/job/{id}" method="delete" %} [backfill\_api.yaml](https://3523412580-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FOmM15VvdyDSaZuim6EH0%2Fuploads%2FtrRGnRMgBzXs3p2Bwqug%2Fbackfill_api.yaml?alt=media\&token=18c9ac3f-6098-482b-8e1b-173fd3e7ad65) {% endopenapi %} {% openapi src="/files/C28GPGR88YlAJQ0zVX6Y" path="/job/{id}" method="patch" %} [backfill\_api.yaml](https://3523412580-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FOmM15VvdyDSaZuim6EH0%2Fuploads%2FtrRGnRMgBzXs3p2Bwqug%2Fbackfill_api.yaml?alt=media\&token=18c9ac3f-6098-482b-8e1b-173fd3e7ad65) {% endopenapi %} {% openapi src="/files/C28GPGR88YlAJQ0zVX6Y" path="/job/{id}/start" method="patch" %} [backfill\_api.yaml](https://3523412580-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FOmM15VvdyDSaZuim6EH0%2Fuploads%2FtrRGnRMgBzXs3p2Bwqug%2Fbackfill_api.yaml?alt=media\&token=18c9ac3f-6098-482b-8e1b-173fd3e7ad65) {% endopenapi %} {% openapi src="/files/C28GPGR88YlAJQ0zVX6Y" path="/job/{id}/stop" method="patch" %} [backfill\_api.yaml](https://3523412580-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FOmM15VvdyDSaZuim6EH0%2Fuploads%2FtrRGnRMgBzXs3p2Bwqug%2Fbackfill_api.yaml?alt=media\&token=18c9ac3f-6098-482b-8e1b-173fd3e7ad65) {% endopenapi %} {% openapi src="/files/C28GPGR88YlAJQ0zVX6Y" path="/job/{id}/metrics" method="get" %} [backfill\_api.yaml](https://3523412580-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FOmM15VvdyDSaZuim6EH0%2Fuploads%2FtrRGnRMgBzXs3p2Bwqug%2Fbackfill_api.yaml?alt=media\&token=18c9ac3f-6098-482b-8e1b-173fd3e7ad65) {% endopenapi %} # WebHooks payload When using webhooks, your endpoint will receive a POST request containing a `payload` field in the JSON body. This `payload` will always be an array that holds transaction data, even if batch processing is disabled. The transactions are represented in raw, unprocessed format, similar to what you would receive from an RPC node. For advanced parsing, we offer an internal parser capable of interpreting well-known Interface Definition Languages (IDLs). More details on the parsing feature are available in the [*Parsed Transaction*](broken://pages/s7uWcgx88bqtutXH9WJY) section. **Example Payload:** ```json { "payload": [ { ...transaction }, { ...transaction } ] } ``` # Monitoring processes ### Metrics Our backfilling tasks are under constant surveillance, with metrics refreshing at one-second intervals. This ensures you have the most up-to-date insights into the performance and progress of your jobs. Developers have two options for accessing these metrics: 1. **On-Demand Access**: At any time during the backfilling process, you can request the latest metrics directly, allowing you to integrate this data into your monitoring systems on your schedule. 2. **Webhook Integration**: Set up a dedicated webhook URL when you create your job, and our system will push metrics regularly to your endpoint. This seamless integration is designed for developers looking to incorporate metrics into their existing infrastructure without the need for polling. Whichever method you choose, you'll be equipped with the information you need to optimize your application's performance and monitor your backfilling processes effectively. Available metrics: | Field | Description | | --------------------- | ------------------------------------- | | txt\_forward | Transactions that got sent to webhook | | txt\_scanned | Amount of transactions scanned | | txt\_per\_sec | Scan speed | | completed\_percentage | Progress in percentage | ### Job Status Lifecycle Throughout the execution of your backfilling job, you can expect the job status to transition through several states. To keep track of your job's current status, you have a couple of convenient options: * **Fetch the Latest Status:** Query our API to retrieve the most recent status information for your job. This is perfect for those who prefer pulling data on demand. * **Webhook Notifications:** Provide a webhook URL when setting up your job, and our system will proactively send status updates to your endpoint. This method suits those who prefer an automated, push-based approach to stay informed. #### **Understanding Status Transitions** The job status moves through the following stages during backfilling operations: * `CREATED`: Your job is set up and in the queue, awaiting its start. * `STARTING`: The job is initiated. This stage includes preparing the necessary resources and configurations. * `RUNNING`: Your backfilling process is active. You can check its progress via real-time metrics. * `STOPPING`: The job is winding down gracefully, saving its current state. * `PAUSED`: The job is on hold. You can resume at a future time from this state. * `COMPLETED`: The job has finished processing the specified range of blocks. Stay in tune with your backfilling progress by opting for either of these methods for job status updates. # Transaction Filter When handling transaction streams across various APIs, including webhooks and backfilling operations, we consistently employ the `TransactionFilter` specification to ensure that filters are set up uniformly. This specification outlines the data structures and criteria used to select the desired transactions. Using a standardized `TransactionFilter` across different API interfaces streamlines the process and avoids redundancy. Below is the JSON representation of the complete `TransactionFilter` structure for reference. Developers can adapt this template to meet the specific filtering requirements of their individual use cases. ### TransactionFilter Overview Our `TransactionFilter` currently supports granular instruction filtering based on program or individual instruction parameters. In upcoming releases, we plan to expand its functionality to include filters for events and accounts. To facilitate the discovery of programs and instruction activities, we recommend utilizing our `Programs API`, which provides the necessary discriminators. **TransactionFilter JSON Schema** Below is the `TransactionFilter` structure in its JSON format. Note the inclusion of optional parameters `id` and `name`, which can aid in event stream identification if needed. ```json [ { "id": "", "name": "", "include": { "instructions": [ { "address": "", "discriminator": "", "discriminatorType": "" } ] } } ] ``` **Key Definitions** * `id`: (optional) A unique identifier for the transaction filter. * `name`: (optional) A descriptive name for the transaction filter. * `include`: Defines the rules for matching transactions. * `instructions`: An array of objects to specify instruction filters. * `address`: The address of the program (required). * `discriminator`: The unique identifier for the instruction (optional). * `discriminatorType`: The type of discriminator, such as `u8`, `u16`, `u32`, or `anchor` (optional). Leverage this `TransactionFilter` structure to customize your filtering needs according to the specifications of the individual use cases. Keep an eye out for our updates, which will enhance the filtering capabilities with additional features. ### Examples #### Example 1: Basic Program Address Filter To track transactions associated with a specific program, you can set the address field in the instructions array: ```json [ { "id": "metaplex-program-filter", "name": "All Metaplex Transactions", "include": { "instructions": [ { "address": "metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s" } ] } } ] ``` #### Example 2: Instruction Discriminator Filter To filter transactions based on a particular type of instruction within a program, include the discriminator: ```json [ { "include": { "instructions": [ { // filter calls to CreateMetadaAccountV2 from Metaplex program "address": "metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s", "discriminator": "10", "discriminatorType": "u8" } ] } } ] ``` #### Example 3: Combining Multiple Required instructions For more complex filtering, you can combine multiple instruction criteria in the instructions array: ```json [ { "include": { "instructions": [ { // filter calls to CreateMasterEditionV3 from Metaplex program "address": "metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s", "discriminator": "11", "discriminatorType": "u8" }, { // filter calls to CreateMetadaAccountV3 from Metaplex program "address": "metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s", "discriminator": "21", "discriminatorType": "u8" } ] } } ] ``` #### Example 4: Combining Multiple Filters Want to monitor several instruction types? Simply stack your filters in one go like this: ```json [ // Your first instruction filter { "include": { "instructions": [ { // Details for the first instruction } ] } }, // Your second instruction filter { "include": { "instructions": [ { // Details for the second instruction } ] } }, // Add more filters as needed ] ``` Customize each filter block with the specific instruction details. This way, you capture all relevant instruction types in a single filter configuration. Something like this: ```json [ { "include": { "instructions": [ { // filter calls to CreateMetadaAccountV2 from Metaplex program "address": "metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s", "discriminator": "10", "discriminatorType": "u8" } ] } }, { "include": { "instructions": [ { // filter calls to CreateMetadaAccountV3 from Metaplex program "address": "metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s", "discriminator": "21", "discriminatorType": "u8" } ] } }, { "include": { "instructions": [ { // filter calls to CreateMetadaAccount from Metaplex program "address": "metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s", "discriminator": "00", "discriminatorType": "u8" } ] } } ] ``` These examples should get you started on using `TransactionFilter` to track transactions that are most relevant to your application logic. Adjust the filters to your specific needs by tweaking the parameters provided. # Other # Disaster recovery Disaster Recovery Plan for DH3.io (Data Hub for Web3) Regarding Solana Chain Reorganizations #### Introduction This disaster recovery plan details the steps to be taken in response to reorganizations (reorgs) and disruptions within the Solana blockchain network as they affect [DH3.io](http://DH3.io) operations. The aim is to ensure a systematic and reliable approach to managing potential inconsistencies and maintaining the integrity of both real-time streams and historical data within the [DH3.io](http://DH3.io) ecosystem. #### Regular Monitoring Procedures * Validator Consistency Check: Every 5 seconds, a consistency verification is carried out among our validators to ensure they are in agreement on slot positions and block hashes. * External RPC Verification: At 30-second intervals, an additional check with a randomly selected RPC from our list of trusted sources (e.g., QuickNode) is performed to confirm data consistency with external networks. #### Disaster Recovery Plan for Real Time Streams and Historical Database * Initial Response: * In the event of detected inconsistencies, the Kubernetes operator will halt all related instances to prevent the spread of erroneous data. * Issue Assessment: * Determine whether the inconsistency is due to a global blockchain event or localized to our validators. * Localized Event Handling: * If the issue is found to be local, the affected validator is removed from the pool. * Identifying Last Correct Block ID: * Establish the most recent valid block ID before the discrepancy was detected. * Data Correction: * Delete incorrect records from HBase (RPC) and HDFS archives. Thanks to our LiteRPC, this process allows for the straightforward rollback and deletion of inaccurate data. #### Disaster Recovery Plan for User Generated Data Sets * Data Set Generation Delay: * For most scenarios, it is recommended to delay the generation of data sets for 5-10 minutes. This delay, combined with the cessation of all related downstream processes and jobs by our Kubernetes operator, usually suffices to address any inconsistencies. * Real-Time Data Set Handling: * For real-time data sets, the same corrective measures are applied as with the historical database, involving the deletion of erroneous blocks of data. * Data Integrity and Duplication Prevention: * Each record within the data sets is assigned a unique ID, derived from the block timestamp, transaction position within the block, and several other characteristics. This unique identifier helps quickly eliminate unwanted data and acts as a measure against data duplication. #### Conclusion This disaster recovery plan is crucial for maintaining the operational integrity and reliability of [DH3.io](http://DH3.io) in the face of Solana blockchain reorganizations. By following these procedures, [DH3.io](http://DH3.io) ensures the consistency, accuracy, and security of both the real-time and historical data within its Web3 data hub.