🧠
DH3 Docs
  • Welcome WEB3 frens
  • 🧠DH3 Focus
    • Effective storage
    • Multichain
    • Quick indexing
    • Parsing and Decoding
    • Exploring
    • Building
    • Data ownership
  • ⚒️Specs
    • Data Streams
      • Storage
      • Streams
      • Channels
    • Stream interceptors
      • Interceptors
        • Selecting stream
        • Defining compute resources
        • Filtering
        • Extracting
        • Transforming
        • Loading
      • Beams
        • Available
        • Create a beam
      • Helpers
        • Dictionaries
      • Execution
      • Examples
    • Data sets
      • Base data sets
      • Bring your data
      • Views
    • Providers
    • API's
      • Backfilling
        • HTTP Methods
        • WebHooks payload
        • Monitoring processes
        • Transaction Filter
    • Other
      • Disaster recovery
Powered by GitBook
On this page
  1. Specs
  2. API's
  3. Backfilling

HTTP Methods

Last updated 1 year ago

API Calls

⚒️

List all jobs

get

Retrieves the list of backfilling jobs.

Query parameters
tokenstringOptionalExample: f469559d4cc55170a26031a4a5123eb2
Body
Responses
200
Successful response
application/json
get
GET /backfill/job/ HTTP/1.1
Host: api.dh3.io
Accept: */*
200

Successful response

[
  {
    "id": "44",
    "name": "scan-1",
    "status": "COMPLETED",
    "batch": "1",
    "hooks": {
      "data": "https://my.project.com/callback",
      "status": null,
      "metrics": null
    },
    "cpu_limit": "1",
    "range": {
      "blocks": {
        "start": 164776968,
        "end": 164754296
      },
      "time": {
        "start": 1670110514,
        "end": 1670120614
      }
    },
    "filter": [
      {
        "include": {
          "instructions": [
            {
              "address": "metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s",
              "discriminator": "10",
              "discriminatorType": "u8"
            }
          ]
        }
      }
    ],
    "metrics": {
      "cpu_time": 220,
      "run_time": 220,
      "txt_forward": 1313,
      "txt_per_sec": 4279,
      "txt_scanned": 941482,
      "callback_error_count": 0,
      "completed_percentage": 100,
      "callback_success_count": 137,
      "callback_bytes_transmit": 0
    }
  }
]

Get a job

get

Retrieves full job information

Path parameters
idintegerOptionalExample: 44
Query parameters
tokenstringOptionalExample: f469559d4cc55170a26031a4a5123eb2
Body
Responses
200
Successful response
application/json
get
GET /backfill/job/{id} HTTP/1.1
Host: api.dh3.io
Accept: */*
200

Successful response

{
  "id": "44",
  "name": "scan-1",
  "status": "COMPLETED",
  "batch": "1",
  "hooks": {
    "data": "https://my.project.com/callback",
    "status": null,
    "metrics": null
  },
  "cpu_limit": "1",
  "range": {
    "blocks": {
      "start": 164776968,
      "end": 164754296
    },
    "time": {
      "start": 1670110514,
      "end": 1670120614
    }
  },
  "filter": [
    {
      "include": {
        "instructions": [
          {
            "address": "metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s",
            "discriminator": "10",
            "discriminatorType": "u8"
          }
        ]
      }
    }
  ],
  "metrics": {
    "cpu_time": 220,
    "run_time": 220,
    "txt_forward": 1313,
    "txt_per_sec": 4279,
    "txt_scanned": 941482,
    "callback_error_count": 0,
    "completed_percentage": 100,
    "callback_success_count": 137,
    "callback_bytes_transmit": 0
  }
}

Delete job

delete

Delete backfilling job.

Path parameters
idintegerOptionalExample: 44
Query parameters
tokenstringOptionalExample: f469559d4cc55170a26031a4a5123eb2
Body
Responses
200
Successful response
application/json
delete
DELETE /backfill/job/{id} HTTP/1.1
Host: api.dh3.io
Accept: */*
200

Successful response

{
  "message": "ok"
}

Update job

patch

Updates job params.

Path parameters
idintegerOptionalExample: 44
Query parameters
tokenstringOptionalExample: f469559d4cc55170a26031a4a5123eb2
Body
Responses
200
Successful response
application/json
patch
PATCH /backfill/job/{id} HTTP/1.1
Host: api.dh3.io
Accept: */*
200

Successful response

{
  "message": "ok"
}

Start job

patch

Starts a backfilling process. If the time range is big, it could take up to a few minutes to start.

Path parameters
idintegerOptionalExample: 44
Query parameters
tokenstringOptionalExample: f469559d4cc55170a26031a4a5123eb2
Body
Responses
200
Successful response
application/json
patch
PATCH /backfill/job/{id}/start HTTP/1.1
Host: api.dh3.io
Accept: */*
200

Successful response

{
  "message": "ok"
}

Stop job

patch

Stops backfilling process. After that job will have PAUSE status.

Path parameters
idintegerOptionalExample: 44
Query parameters
tokenstringOptionalExample: f469559d4cc55170a26031a4a5123eb2
Body
Responses
200
Successful response
application/json
patch
PATCH /backfill/job/{id}/stop HTTP/1.1
Host: api.dh3.io
Accept: */*
200

Successful response

{
  "message": "ok"
}

Get a job metrics

get

Metrics can be received on demand via this call or by getting full job information. Also you can defined metrics web hook callback when creating job.

Path parameters
idintegerOptionalExample: 44
Query parameters
tokenstringOptionalExample: f469559d4cc55170a26031a4a5123eb2
Body
Responses
200
Successful response
application/json
get
GET /backfill/job/{id}/metrics HTTP/1.1
Host: api.dh3.io
Accept: */*
200

Successful response

{
  "cpu_time": 220,
  "run_time": 220,
  "txt_forward": 1313,
  "txt_per_sec": 4279,
  "txt_scanned": 941482,
  "callback_error_count": 0,
  "completed_percentage": 100,
  "callback_success_count": 137,
  "callback_bytes_transmit": 0
}
  • API Calls
  • GETList all jobs
  • POSTCreate Job
  • GETGet a job
  • DELETEDelete job
  • PATCHUpdate job
  • PATCHStart job
  • PATCHStop job
  • GETGet a job metrics

Create Job

post

Creates a backfilling task that retrieves transactions based on specified filters and subsequently submits them to a predefined URL in the hook's data field.

Query parameters
tokenstringOptionalExample: f469559d4cc55170a26031a4a5123eb2
Body
namestringRequired

The name of job.

filterobjectRequired

An array of filters to apply. Check the Transaction Filter for details.

batchinteger · min: 1 · max: 500Optional

Batch webhook calls. Specify the number of transactions to be batched in one call. Defaults to 1

cpu_limitinteger · min: 1 · max: 20Optional

Limit CPU for JOB, one CPU does ~20 000 transaction rescans. Speed can vary based on how many records are picked up and sent back. You can look at it as concurrency or threads, one CPU will always send one hook at the moment. Defaults to 1

Responses
200
Successful response
application/json
post
POST /backfill/job/ HTTP/1.1
Host: api.dh3.io
Content-Type: application/json
Accept: */*
Content-Length: 360

{
  "name": "scan-1",
  "hooks": {
    "data": "https://my.project.com/callback/"
  },
  "range": {
    "time": {
      "start": 1670110514,
      "end": 1670120614
    }
  },
  "filter": [
    {
      "name": "metaplex-CreateMetadataV2",
      "include": {
        "instructions": [
          {
            "address": "metaqbxxUerdq28cj1RbAWkYQm3ybzjb6a8bt518x1s",
            "discriminator": "10",
            "discriminatorType": "u8"
          }
        ]
      }
    }
  ],
  "parser": {
    "enabled": [
      "data"
    ]
  },
  "cpu_limit": 1,
  "batch": 10
}
200

Successful response

{
  "id": 1
}