Manage ClickHouse schemas
and sync API data.

01 / Schema

Define tables and views.

Define ClickHouse tables, views, materialized views, and dictionaries in code. Import the schema from an existing database.

Define a schema ↗

02 / Migrations & backfills

Migrate and backfill data.

Handle complex schema changes and backfill data across materialized views. Track progress and resume interrupted runs.

Explore backfills ↗

03 / API Sync

Sync data from any source.

Build reliable syncs from HTTP APIs, databases, and other sources into ClickHouse.

Browse app integrations ↗

Schema and API sync example

From a schema to a working data sync

Start with a table, load 100 demo API records, then query posts per author. Change the view later using the data already stored in ClickHouse.

Install the packages and configure a ClickHouse connection before running the example. Add the TypeScript snippets to src/chkit.ts. For schema management without API sync, use the schema tutorial.

1. Define

Define a table

Define the columns once; the reader in step 3 writes into this table. The replacement engine and stable key let queries resolve repeated syncs to the latest version of each post.

src/chkit.ts

import { table } from '@chkit/core'
import { ingestionColumns } from '@chkit/plugin-ingest'

export const posts = table({
  database: 'default', name: 'posts',
  columns: [
    { name: 'id', type: 'String' },
    { name: 'title', type: 'String' },
    { name: 'user_id', type: 'UInt64' },
    ...ingestionColumns,
  ],
  engine: 'ReplacingMergeTree(_chkit_ingested_at)',
  primaryKey: ['id'], orderBy: ['id'],
})

Explore the schema DSL. Schema management is available in TypeScript and Python; this walkthrough uses TypeScript for API sync.

2. Migrate

Generate and review a migration

Point the project config at src/chkit.ts and register ingest() when adding API sync. With a ClickHouse connection configured, generate the migration and inspect it before applying.

Terminal

bunx chkit generate --name create-posts
bunx chkit migrate

# After reviewing the generated SQL and preview:
bunx chkit migrate --apply

Commit the schema, migration SQL, and snapshot together. The setup guide includes the complete config and package installation.

See the generated table SQL

Generated migration: table statement

CREATE TABLE IF NOT EXISTS default.posts
(
  `id` String,
  `title` String,
  `user_id` UInt64,
  `_chkit_batch_id` String,
  `_chkit_run_id` String,
  `_chkit_ingested_at` DateTime64(6, 'UTC') DEFAULT now64(6)
) ENGINE = ReplacingMergeTree(_chkit_ingested_at)
PRIMARY KEY (`id`)
ORDER BY (`id`);

3. Sync

Load API records into the table

Add this reader and pipeline to src/chkit.ts. It fetches a small public API and maps records into the table’s columns.

src/chkit.ts: add to the schema above

import { definePipeline, defineStream, HttpError } from '@chkit/plugin-ingest'

type Post = { id: number; title: string; userId: number }

const postStream = defineStream({
  id: 'demo.posts', destination: posts,
  async *read(context) {
    const page = await context.attempt(async (signal) => {
      const response = await fetch('https://jsonplaceholder.typicode.com/posts', { signal })
      if (!response.ok) throw await HttpError.fromResponse(response)
      return await response.json() as Post[]
    })
    yield { rows: page.map((post) => ({
      id: String(post.id), title: post.title, user_id: post.userId,
    })) }
  },
})

export const content = definePipeline({ id: 'content', streams: [postStream] })

Run bunx chkit ingest run --tag pipeline:content. This bounded demo reads all posts; add pagination and incremental state when the source needs them.

Result: 100 posts in default.posts, ready to query. This reader supplies the data; the default loader supplies batching and ingestion metadata.

4. Query

Query posts by author

Views belong to the same schema. Add this definition, generate and apply its migration, then query posts per author. FINAL reconciles repeated versions before the count.

src/chkit.ts: add a view

import { view } from '@chkit/core'

export const postsByAuthor = view({
  database: 'default', name: 'posts_by_author',
  as: `SELECT user_id, count() AS posts
       FROM default.posts FINAL
       GROUP BY user_id`,
})

bunx chkit generate --name posts-by-author
bunx chkit migrate
bunx chkit migrate --apply
bunx chkit query "SELECT * FROM default.posts_by_author ORDER BY user_id LIMIT 3"

Expected result for the demo dataset:

user_id	posts
1	10
2	10
3	10

If the final schema is still evolving, retain raw objects and transform them in ClickHouse instead of mapping every field up front.

5. Evolve

Add a measure to the view

Update the view definition from step 4 to count titles containing a search term.

src/chkit.ts: replace the previous view definition

import { view } from '@chkit/core'

export const postsByAuthor = view({
  database: 'default', name: 'posts_by_author',
  as: `SELECT user_id, count() AS posts,
         countIf(positionCaseInsensitive(title, 'qui') > 0) AS matching_posts
       FROM default.posts FINAL
       GROUP BY user_id`,
})

Generate another migration, review it, and apply it. chkit recreates this ordinary view with the new SQL. Learn about transformations.

Query the new matching_posts measure over the records loaded in step 3. This view change requires no API re-fetch or table backfill.

6. Verify

Check for drift and schedule syncs

Use the same CLI locally and in CI. Check migration state and schema drift, inspect the exported sources, and run the sync on an external schedule.

Terminal / CI

# Check migration state and live schema drift.
bunx chkit check

# Inspect the source definitions.
bunx chkit ingest list

# Run once; cron or CI controls the cadence.
bunx chkit ingest run --tag pipeline:content

Serialize ingestion processes per target. When using an incremental source, the next run starts from its committed state. See scheduling and recovery and source testing.

Plugins and agent skills

Add the tools your project needs.

Import an existing schema, generate application types, or backfill stored data. Configure the plugins in the same project.

• Adopt an existing databasePull live schema into local definitions.
• Keep application types alignedGenerate TypeScript row types and Zod schemas.
• Backfill stored dataRun SQL data migrations with checkpoints.
• Work with your coding agentInstall skills for schema and source authoring.

Compatible with

• 

  ObsessionDB

• 

  ClickHouse Cloud

• 

  Altinity

• 

  Self-hosted ClickHouse

• 

  …and any other ClickHouse

Set up chkit.

Manage a schema →Sync your first source →