Persistent Storage
How to store data and images you can re-use and query across model runs
Want to help us test?
Our storage features are still in active development, and as such, may change. You're welcome to help us test, or to wait until they are generally available. To request access, sign in to your Summit account and send us a message from the Storage page.
Database
External
You can bring your own Postgres-compatible database into a Summit model by passing a connection string to a Storage
event:
=Storage("{{ DB_CONN }}")
This DB_CONN
should be stored in your Vault.
Your data, your responsibility.
Summit will automatically flag and forbid queries that contain
DROP
,DELETE
,TRUNCATE
, orALTER
in order to prevent accidental data loss.However, you should still follow the principle of least permission and connect to your database with a user that has been explicitly granted the smallest set of permissions.
Summit DB
Summit provides customers with their own database schema for persistent storage. This database is a CockroachDB hosted by Cockroach Labs on Amazon Web Services.
You are free to CREATE
your own tables inside this database.
For convenience, each account is automatically provided with four tables:
Table Name | Purpose |
---|---|
kv_store | Key-value storage of JSON objects. |
media_store | Tracking and manage content uploads to the CDN. |
logs | Automatically populates with new rows when any of your models run. |
tasks | Allows you to create scheduled and one-off tasks to run at specific times. |
You may of course also create any other tables you'd like using the Query event.
Using key-value storage
Summit makes it very easy to take advantage of key-value storage. Simply send any event to a Storage
event like so:
=Storage("my_special_key")
Whatever data is sent to this event will be automatically inserted into the kv_store
table with a key of my_special_key
. If data already exists at that key, it will be overwritten.
In programming terms, this makes the above a setter.
But this is also a getter, meaning you can use the same syntax to retrieve the data from this location in the kv_store
table simply by pointing this storage event at another event, or referencing this storage event using liquid syntax.
The structure of your kv_store
table:
CREATE TABLE kv_store (
id BIGSERIAL PRIMARY KEY,
key TEXT NOT NULL UNIQUE,
value JSONB NOT NULL,
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW()
);
As you can see above, the key must be unique, and the value stored at a key must always be of a JSON format, like an Object.
The creation and update timestamps will track changes automatically.
Using media storage
Image files are stored in the CDN, but a copy of each uploaded file is recorded in the media_store
table.
The structure of your media_store
table:
CREATE TABLE media_store (
id BIGSERIAL PRIMARY KEY,
cdn_id UUID,
url TEXT NOT NULL UNIQUE,
description TEXT,
file_format TEXT NOT NULL,
width INT, -- Optional for images and videos
height INT, -- Optional for images and videos
duration INT, -- Duration in seconds, applicable for audio and video
bitrate INT, -- Bitrate in kbps, applicable for audio and video
codec TEXT, -- Codec used for encoding, applicable for audio and video
file_size BIGINT, -- Size of the file in bytes
created_by TEXT,
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW()
);
What's allowed (and not allowed)
Your user is allowed to perform creative and destructive operations, which includes
CREATE
,DELETE
,SELECT
,INSERT
,ALTER
, andTRUNCATE
. You may also useSHOW CREATE
to look at existing table structures.Your user cannot create schemas, nor delete the provisioned tables, nor create users.
As we have handed you some fairly sharp scissors, please be careful. If any queries destroy your data, we are not able to recover your data, as we do not provide a restore-from-backups service (yet).
Your user database username, password, and schema name are stored in your Vault. In general, you should not need to modify these. If you accidentally change or delete one of these credentials, please contact support.
Using logs
New rows are inserted into the logs
table any time a model runs, providing you with a record of what is (and isn't) working as part of your automations. This visibility can be leveraged to create all manner of apps and alerts.
The structure of the logs
table is:
CREATE TABLE logs (
run_pk UUID PRIMARY KEY, -- run_id as UUID, must not be NULL
created_at TIMESTAMPTZ NULL DEFAULT now(), -- Timestamp when the task was created, defaults to the current time
widget_pk UUID, -- widget_pk as UUID, can be NULL
model_name STRING, -- name of this widget (model) as of this log, can be NULL
input_parameters JSONB, -- input parameters as JSONB, can be NULL
request_user_pk UUID, -- request_user_pk as UUID, can be NULL
api_key_pk UUID, -- api_key_pk as UUID, can be NULL
access_type STRING, -- access type as a string, can be NULL
duration_ms NUMERIC, -- duration in milliseconds, can be NULL
ip_address TEXT, -- IP address as text, can be NULL
logs TEXT -- console logs as text, can be NULL
);
Using tasks
Rows added to tasks
get picked up and run automatically. Tasks point to models you've created in your account by referencing the unique ID of your published model.
The SQL to add any given model as a task is provided to you on that model's "API & Webhooks" screen, for example:
INSERT INTO tasks (schedule_expression, task_id)
VALUES
('*/5 * * * *', 'd27792d3-0e08-4192-b1e4-76b1deb17483');
This will cause the model with this ID to run every 5 minutes, until and unless it fails, in which case it will be flagged as failed
status.
The structure of the tasks
table is:
CREATE TABLE tasks (
job_id UUID NOT NULL DEFAULT gen_random_uuid(), -- Unique identifier for each task, auto-generated UUID
schedule_expression STRING NULL, -- Cron-like schedule expression, can be NULL for one-time tasks
run_at TIMESTAMPTZ NULL, -- Timestamp for one-time tasks, can be NULL for recurring tasks
task_type TEXT NOT NULL DEFAULT 'cron' -- Task type, defaults to 'cron' and can only be 'cron' or 'one-time'
CHECK (task_type IN ('cron', 'one-time')), -- Ensure task_type is either 'cron' or 'one-time'
status STRING NOT NULL DEFAULT 'pending', -- Task status, defaults to 'pending' (e.g., 'pending', 'running', 'completed')
created_at TIMESTAMPTZ NULL DEFAULT now(), -- Timestamp when the task was created, defaults to the current time
last_run_at TIMESTAMPTZ NULL, -- Timestamp for when the task was last run, can be NULL initially
next_run_at TIMESTAMPTZ NULL, -- Timestamp for when the task is scheduled to run next, can be NULL
updated_at TIMESTAMPTZ NULL DEFAULT now(), -- Timestamp for when the task was last updated, defaults to the current time
task_id UUID NOT NULL, -- Task identifier, cannot be NULL, not required to be unique
payload JSONB NULL, -- JSONB column for storing structured data (e.g., parameters or task metadata)
CONSTRAINT tasks_pkey PRIMARY KEY (job_id), -- Primary key constraint on job_id
INDEX idx_task_schedule (schedule_expression, run_at) -- Index for optimizing queries based on schedule_expression and run_at
);
CDN
The Storage event also allows you to upload images generated using AI or screenshot Requests. To do so, you can declare an uploader like so:
=Storage("upload")
When this event receives a base64 encoded string, it will upload it and insert a record into your media_store
table for later reference.
Since the img
method of our Requests event returns a base64 representation of the website screenshot, you can also point it directly at an Storage uploader.
Updated 3 days ago