Persistent Storage

How to store data and images you can re-use and query across model runs

👍

Want to help us test?

Our storage features are still in active development, and as such, may change. You're welcome to help us test, or to wait until they are generally available. To request access, sign in to your Summit account and send us a message from the Storage page.

Database

External

You can bring your own Postgres-compatible database into a Summit model by passing a connection string to a Storage event:

=Storage("{{ DB_CONN }}")

This DB_CONN should be stored in your Vault.

🚧

Your data, your responsibility.

Summit will automatically flag and forbid queries that contain DROP, DELETE, TRUNCATE, or ALTER in order to prevent accidental data loss.

However, you should still follow the principle of least permission and connect to your database with a user that has been explicitly granted the smallest set of permissions.

Summit DB

Summit provides customers with their own database schema for persistent storage. This database is a CockroachDB hosted by Cockroach Labs on Amazon Web Services.

You are free to CREATE your own tables inside this database.

For convenience, each account is automatically provided with four tables:

Table Name	Purpose
kv_store	Key-value storage of JSON objects.
media_store	Tracking and manage content uploads to the CDN.
logs	Automatically populates with new rows when any of your models run.
tasks	Allows you to create scheduled and one-off tasks to run at specific times.

You may of course also create any other tables you'd like using the Query event.

Using key-value storage

Summit makes it very easy to take advantage of key-value storage. Simply send any event to a Storage event like so:

=Storage("my_special_key")

Whatever data is sent to this event will be automatically inserted into the kv_store table with a key of my_special_key. If data already exists at that key, it will be overwritten.

In programming terms, this makes the above a setter.

But this is also a getter, meaning you can use the same syntax to retrieve the data from this location in the kv_store table simply by pointing this storage event at another event, or referencing this storage event using liquid syntax.

The structure of your kv_store table:

SQL

CREATE TABLE kv_store (
  id BIGSERIAL PRIMARY KEY,
  key TEXT NOT NULL UNIQUE,
  value JSONB NOT NULL,
  created_at TIMESTAMPTZ DEFAULT NOW(),
  updated_at TIMESTAMPTZ DEFAULT NOW()
);

As you can see above, the key must be unique, and the value stored at a key must always be of a JSON format, like an Object.

The creation and update timestamps will track changes automatically.

Using media storage

Image files are stored in the CDN, but a copy of each uploaded file is recorded in the media_store table.

The structure of your media_store table:

SQL

CREATE TABLE media_store (
  id BIGSERIAL PRIMARY KEY,
  cdn_id UUID,
  url TEXT NOT NULL UNIQUE,
  description TEXT,
  file_format TEXT NOT NULL,
  width INT,  -- Optional for images and videos
  height INT,  -- Optional for images and videos
  duration INT,  -- Duration in seconds, applicable for audio and video
  bitrate INT,  -- Bitrate in kbps, applicable for audio and video
  codec TEXT,  -- Codec used for encoding, applicable for audio and video
  file_size BIGINT,  -- Size of the file in bytes
  created_by TEXT,
  created_at TIMESTAMPTZ DEFAULT NOW(),
  updated_at TIMESTAMPTZ DEFAULT NOW()
);

🚧

What's allowed (and not allowed)

Your user is allowed to perform creative and destructive operations, which includes CREATE, DELETE, SELECT, INSERT,ALTER, and TRUNCATE. You may also use SHOW CREATE to look at existing table structures.

Your user cannot create schemas, nor delete the provisioned tables, nor create users.

As we have handed you some fairly sharp scissors, please be careful. If any queries destroy your data, we are not able to recover your data, as we do not provide a restore-from-backups service (yet).

Your user database username, password, and schema name are stored in your Vault. In general, you should not need to modify these. If you accidentally change or delete one of these credentials, please contact support.

Using logs

New rows are inserted into the logs table any time a model runs, providing you with a record of what is (and isn't) working as part of your automations. This visibility can be leveraged to create all manner of apps and alerts.

The structure of the logs table is:

SQL

CREATE TABLE logs (
    run_pk UUID PRIMARY KEY,                        -- run_id as UUID, must not be NULL
    created_at TIMESTAMPTZ NULL DEFAULT now(),      -- Timestamp when the task was created, defaults to the current time
    widget_pk UUID,                                 -- widget_pk as UUID, can be NULL
    model_name STRING,                              -- name of this widget (model) as of this log, can be NULL
    input_parameters JSONB,                         -- input parameters as JSONB, can be NULL
    request_user_pk UUID,                           -- request_user_pk as UUID, can be NULL
    api_key_pk UUID,                                -- api_key_pk as UUID, can be NULL
    access_type STRING,                             -- access type as a string, can be NULL
    duration_ms NUMERIC,                            -- duration in milliseconds, can be NULL
    ip_address TEXT,                                -- IP address as text, can be NULL
    logs TEXT                                       -- console logs as text, can be NULL
);

Using tasks

Rows added to tasks get picked up and run automatically. Tasks point to models you've created in your account by referencing the unique ID of your published model.

The SQL to add any given model as a task is provided to you on that model's "API & Webhooks" screen, for example:

SQL

INSERT INTO tasks (schedule_expression, task_id) 
VALUES 
 ('*/5 * * * *', 'd27792d3-0e08-4192-b1e4-76b1deb17483');

This will cause the model with this ID to run every 5 minutes, until and unless it fails, in which case it will be flagged as failed status.

The structure of the tasks table is:

SQL

CREATE TABLE tasks (
    job_id UUID NOT NULL DEFAULT gen_random_uuid(),  -- Unique identifier for each task, auto-generated UUID
    schedule_expression STRING NULL,                -- Cron-like schedule expression, can be NULL for one-time tasks
    run_at TIMESTAMPTZ NULL,                        -- Timestamp for one-time tasks, can be NULL for recurring tasks
    task_type TEXT NOT NULL DEFAULT 'cron'          -- Task type, defaults to 'cron' and can only be 'cron' or 'one-time'
        CHECK (task_type IN ('cron', 'one-time')),  -- Ensure task_type is either 'cron' or 'one-time'
    status STRING NOT NULL DEFAULT 'pending',       -- Task status, defaults to 'pending' (e.g., 'pending', 'running', 'completed')
    created_at TIMESTAMPTZ NULL DEFAULT now(),      -- Timestamp when the task was created, defaults to the current time
    last_run_at TIMESTAMPTZ NULL,                   -- Timestamp for when the task was last run, can be NULL initially
    next_run_at TIMESTAMPTZ NULL,                   -- Timestamp for when the task is scheduled to run next, can be NULL
    updated_at TIMESTAMPTZ NULL DEFAULT now(),      -- Timestamp for when the task was last updated, defaults to the current time
    task_id UUID NOT NULL,                          -- Task identifier, cannot be NULL, not required to be unique
    payload JSONB NULL,                             -- JSONB column for storing structured data (e.g., parameters or task metadata)
    CONSTRAINT tasks_pkey PRIMARY KEY (job_id),     -- Primary key constraint on job_id
    INDEX idx_task_schedule (schedule_expression, run_at)  -- Index for optimizing queries based on schedule_expression and run_at
);

CDN

The Storage event also allows you to upload images generated using AI or screenshot Requests. To do so, you can declare an uploader like so:

=Storage("upload")

When this event receives a base64 encoded string, it will upload it and insert a record into your media_store table for later reference.

Since the img method of our Requests event returns a base64 representation of the website screenshot, you can also point it directly at an Storage uploader.