Tool to perform snapshot testing on an SQLite database, using rusqlite.

The goal is to expose both data and the schema in the snapshots. It is also compatible with Insta Snapshots

This is experimental software, expect breaking changes between 0.x versions, consistent with the semver rules for Rust.

Background reading on snapshot testing.

• https://ianthehenry.com/posts/my-kind-of-repl/
• https://tigerbeetle.com/blog/2024-05-14-snapshot-testing-for-the-masses/
• https://blog.janestreet.com/the-joy-of-expect-tests/

Version 2.5.0

Dependencies

Rusqlite was updated from 0.38.0 to 0.39.0. Please see the release notes for 0.39.0.

Update other dev dependencies, see git history for details.

Version 2.4.1

Documentation

Attempt to fix the docs.rs build which started failing in 2.4.0.

Dependencies

Update some dev dependencies, see git history for details.

Version 2.4.0

Dependencies

Rusqlite was updated from 0.37.0 to 0.38.0. Please see the release notes for 0.38.0, there are a few breaking changes in this one.

Update other dev dependencies, see git history for details.

Features

Rusqlite 0.38 makes the cache statement optional. This feature was used for foreign key checks. The rusqlite_migration library does not enable any features from rusqlite, not even the default one. This way, downstream users can freely chose which feature to enable based on their needs. As a result, rusqlite_migration now handles statement caching for foreign key checks internally now. As side benefit, this also makes caching more efficient: the prepared statement is kept for the minimum time where it is needed and freed immediately, without taking space in the global cache used by rusqlite.

Other

• Use scoped GitHub tokens in actions, with as little privileges as possible.
• Improve test build time by removing some unused optional deps.
• Update most dependencies after a cooldown (this does not apply to security updates).
• Errors don’t use debug output anymore, they should be more friendly to humans. This is not considered a breaking change because Dislay of errors is meant for human consuption and not for flow control in the program.

Version 2.3.0

Dependencies

Rusqlite was updated from 0.36.0 to 0.37.0. Please see the release notes for 0.37.0.

Other

• Misc. clippy fixes
• Minor improvements to the example in the Readme

Version 2.2.0

Features

• Implement the Display trait for M. This makes it easier to print errors pertaining to a particular migration (this feature is planned for the future, in the context of more extensive migration checks)

Dependencies

Rusqlite was updated from 0.35.0 to 0.36.0. Please see the release notes for 0.36.0.

Other

• Update development dependencies
• Improve tests to cover more cases, in particular around downward migrations
• Add docs.rs link to Cargo metadata
• Fix clippy warning in rust 1.87.0

Version 2.2.0 Beta 1

Features

• Implement the Display trait for M. This makes it easier to print errors pertaining to a particular migration (this feature is planned for the future, in the context of more extensive migration checks)

Dependencies

Rusqlite was updated from 0.35.0 to 0.36.0. Please see the release notes for 0.36.0.

Other

• Update development dependencies
• Improve tests to cover more cases, in particular around downward migrations
• Add docs.rs link to Cargo metadata
• Fix clippy warning in rust 1.87.0

Version 2.1.0

Dependencies

Rusqlite was updated from 0.34.0 to 0.34.0. Please see the release notes for 0.35.0.

Version 2.0.0

Breaking changes

Remove the alpha-async-tokio-rusqlite Feature

As the name of the feature suggest, we have had experimental support for async using tokio for a while now. Supporting that feature has been quite a big burden, introducing some duplicated code in the AsyncMigrations struct in particular, as well as a whole set of very similar tests. Plus the benefit of async is limited here, because everything gets executed in a blocking fashion in sqlite anyway.

It turns out that we don’t need the async support in rusqlite_migration for folks to use async libraries. For instance, with tokio-rusqlite, you can define migrations like in the sync context and run:

    async_conn
        .call_unwrap(|conn| MIGRATIONS.to_latest(conn))
        .await?;

See the updated async example for details, in particular why it’s fine to call a method with unwrap in its name.

Make the Builder Finalizer Method Not Generic

On a related note, now that we have removed the AsyncMigrations (see the section right above) struct, we only have Migrations so there is no need for the MigrationsBuilder.finalize method to be generic. Thus we removed the generic argument. To update your code, you can just do this:

-        .finalize::<Migrations>());
+        .finalize());

Remove Migrations::new_iter

This function has been deprecated for a while now, remove it as a part of the major version bump. You can use the standard FromIter trait implementation instead.

Behavior Change

• When the user version field is altered by other code in your application, we are now returning an explicit error (Error::InvalidUserVersion) when this can be detected. Previously, the library would silently misbehave.

Features

• Add the new Migrations::from_slice constructor, which is const and takes a slice, so that it can be constructed in global constant, without using LazyLock or similar. Internally, this is possible because we now use a Cow structure to hold migrations.
• Add Migrations::pending_migrations which returns the number of migrations that would be applied. This is mostly useful to take a backup of the database prior to applying migrations (and do nothing if no migrations will be applied).

Dependencies

Rusqlite was updated from 0.32.1 to 0.34.0. Please see the release notes for 0.34.0 and the release notes for 0.33.0. Tokio Rusqlite was removed as a dependency.

Minimum Rust Version

Rust 1.84.

Moving forward, we expect to keep this aligned with rusqlite itself, now that it has a policy (introduced in october 2024).

Version 2.0.0 Beta 1

Features

• Add the new Migrations::from_slice constructor, which is const and takes a slice, so that it can be constructed in global constant, without using LazyLock or similar. Internally, this is possible because we now use a Cow structure to hold migrations.
• Add Migrations::pending_migrations which returns the number of migrations that would be applied. This is mostly useful to take a backup of the database prior to applying migrations (and do nothing if no migrations will be applied).

Version 2.0.0 Alpha 1

Breaking changes

Remove the alpha-async-tokio-rusqlite Feature

As the name of the feature suggest, we have had experimental support for async using tokio for a while now. Supporting that feature has been quite a big burden, introducing some duplicated code in the AsyncMigrations struct in particular, as well as a whole set of very similar tests. Plus the benefit of async is limited here, because everything gets executed in a blocking fashion in sqlite anyway.

It turns out that we don’t need the async support in rusqlite_migration for folks to use async libraries. For instance, with tokio-rusqlite, you can define migrations like in the sync context and run:

    async_conn
        .call_unwrap(|conn| MIGRATIONS.to_latest(conn))
        .await?;

See the updated async example for details, in particular why it’s fine to call a method with unwrap in its name.

Make the Builder Finalizer Method Not Generic

On a related note, now that we have removed the AsyncMigrations (see the section right above) struct, we only have Migrations so there is no need for the MigrationsBuilder.finalize method to be generic. Thus we removed the generic argument. To update your code, you can just do this:

-        .finalize::<Migrations>());
+        .finalize());

Remove Migrations::new_iter

This function has been deprecated for a while now, remove it as a part of the major version bump. You can use the standard FromIter trait implementation instead.

Behavior Change

• When the user version field is altered by other code in your application, we are now returning an explicit error (Error::InvalidUserVersion) when this can be detected. Previously, the library would silently misbehave.

Dependencies

Rusqlite was updated from 0.32.1 to 0.34.0. Please see the release notes for 0.34.0 and the release notes for 0.33.0. Tokio Rusqlite was removed as a dependency.

Features

• Migrations::new is now const

Minimum Rust Version

Rust 1.84.

Moving forward, we expect to keep this aligned with rusqlite itself, now that it has a policy (introduced in october 2024).

Version 1.3.1

The only change is a fix to the deps.rs badge in the documentation.

Version 1.3.0

Rusqlite was updated from 0.31.0 to 0.32.1. Please see the release notes for 0.32.0 and for 0.32.1. Tokio Rusqlite was updated from 0.5.1 to 0.6.0. Please see the release notes.

Minimum Rust Version

Rust 1.77

Documentation

Various documentation improvements and clarification. In particular, call out that if a rusqlite error is encountered during a migration, the next migrations in the list are not applied.

Other

• Apply minor or patch updates to the dependencies
• Update development dependencies
• Make CI testing more reproducible by forcing the use of Cargo.lock

Version 1.3.0 Beta 1

This reintroduces the async features temporarily removed from Version 1.3.0 Alpha-Without-Tokio 1

Rusqlite was updated from 0.31.0 to 0.32.1. Please see the release notes for 0.32.0 and for 0.32.1. Tokio Rusqlite was updated from 0.5.1 to 0.6.0. Please see the release notes.

Minimum Rust Version

Rust 1.77

Documentation

Various documentation improvements and clarification. In particular, call out that if a rusqlite error is encountered during a migration, the next migrations in the list are not applied.

Other

• Apply minor or patch updates to the dependencies
• Update development dependencies
• Make CI testing more reproducible by forcing the use of Cargo.lock

Version 1.3.0 Alpha-Without-Tokio 1

Major Changes

This is an alpha version to start integrating rusqlite 0.32.1. Unfortunately, at this time, tokio-rusqlite is did not update to rusqlite 0.32.1. So we are temporarily removing the async features, while we figure out a way to bring them back. To be clear, we intend to support the async features going forward, this is a temporary change in a specifically tagged version.

Rusqlite was updated from 0.31.0 to 0.32.1. Please see the release notes for 0.32.0 and for 0.32.1

Minimum Rust Version

Rust 1.77

Documentation

Various documentation improvements and clarification. In particular, call out that if a rusqlite error is encountered during a migration, the next migrations in the list are not applied.

Other

• Apply minor or patch updates to the dependencies
• Update development dependencies
• Make CI testing more reproducible by forcing the use of Cargo.lock

Version 1.2.0

Same code as version 1.2.0-beta.1

Documentation

• Improved the badges a little bit

Version 1.2.0 Beta 1

Small release, mainly to update dependencies.

Minimum Rust Version

Now using edition 2021, but the minimum rust version is still 1.70

New Features

No new features.

Other

• Update rusqlite to 0.31
• Update various development dependencies
• Improve CI build time
• Impove documentation
• Fix some broken examples

See also

Rusqlite was updated from 0.30 to 0.31. Please see its release notes

Version 1.1.0

Same code as version 1.1.0-beta.1

Minimum Rust Version

Rust 1.70

New Features

• Support for tokio-rusqlite behind the feature named alpha-async-tokio-rusqlitethanks to @czocher. See the example. This feature is alpha, meaning that compatibility in future minor versions is not guaranteed.
• Create migrations from directories holding SQL files thanks to @czocher. See the example.
• Add up/down hooks to run custom Rust code during migrations (PR thanks to @matze)
• Add foreign_key_check method to migrations (PR thanks to @Jokler)
• Make Migration functions const (PR thanks to @fkaa)
• Make Migrations serializable (using the Debug serializer) with insta.

Depreciation

• Mark Migrations::from_iter as deprecated

Other

• Documentation improvements
  • Repository metadata improvements
• Code quality improvements
  • Introduce cargo mutants & fix bugs found
  • Clippy warning fixes and other linter improvements
  • Report on test coverage & improve test coverage
  • Add benchmarks
• Made errors returned more precise
• Updated dependencies

See also

Rusqlite was updated from 0.29.0 to 0.30.0. Please see its release notes

Version 1.1.0 Beta 1

⚠️ The APIs exposed in this version may be unstable.

Summing up all the changes from the previous Alpha versions.

Minimum Rust Version

Rust 1.70

New Features

• Support for tokio-rusqlite behind the feature named alpha-async-tokio-rusqlitethanks to @czocher. See the example. This feature is alpha, meaning that compatibility in future minor versions is not guaranteed.
• Create migrations from directories holding SQL files thanks to @czocher. See the example.
• Add up/down hooks to run custom Rust code during migrations (PR thanks to @matze)
• Add foreign_key_check method to migrations (PR thanks to @Jokler)
• Make Migration functions const (PR thanks to @fkaa)
• Make Migrations serializable (using the Debug serializer) with insta.

Depreciation

• Mark Migrations::from_iter as deprecated

Other

• Documentation improvements
  • Repository metadata improvements
• Code quality improvements
  • Introduce cargo mutants & fix bugs found
  • Clippy warning fixes and other linter improvements
  • Report on test coverage & improve test coverage
  • Add benchmarks
• Made errors returned more precise
• Updated dependencies

See also

Rusqlite was updated from 0.29.0 to 0.30.0. Please see its release notes

Version 1.1.0 Alpha 2

⚠️ The APIs exposed in this version may be unstable.

Minimum Rust Version

Rust 1.64

New Features

• Create migrations from directories holding SQL files. See the example.

Depreciation

• Mark Migrations::from_iter as deprecated

Other

• Documentation improvements
• Code quality improvements
  • Introduce cargo mutants & fix bugs found
  • Clippy warning fixes
  • Report on test coverage & improve test coverage
  • Add benchmarks
• Made errors returned more precise
• Update dependencies

Version 1.1.0 Alpha 1

⚠️ The APIs exposed in this version may be unstable.

Minimum Rust Version

Rust 1.61

New Features

• Add up/down hooks to run custom Rust code during migrations (PR thanks to @matze)
  • The purpose of this release is to get feedback on the new API. Please feel free to comment on this discussion!
• Add foreign_key_check method to migrations (PR thanks to @Jokler)
  • Please beware of the follow up work needed on this
• Make Migration functions const (PR thanks to @fkaa)

Other

• CI improvements
• Linter improvements
• Repository metadata improvements
• Documentation improvements
• Dev dependencies update (not dependencies of the library when used in another crate)

Version 1.0.2

Bug fix

• fix: adapt to rusqlite 0.29 and tighten dependency requirements for rusqlite (see this discussion)

Version 1.0.1

Bug Fix

• fix: error instead of panicking on higher migration level (see commit ad57d92d1677420eb81c4e25635be1884f9b7ce7)

Other

• Documentation improvements

Version 1.0.0

Breaking changes

• Remove deprecated symbols (Migrations.latest, SchemaVersionError::MigrateToLowerNotSupported)

Other

• Documentation improvements

Version 0.5.1

Potentially Breaking Changes

• Update the rusqlite crate (to protect agaisnt RUSTSEC-2020-0014)

Other

• Improve the documentation

Version 0.5.0

• Update the env_logger dependency
• Improve the documentation

Version 0.4.1 / 0.4.2

• Update documentation

Version 0.4.0

New features

• Add downward migrations, i.e. migrations to go to past schema version of the database. Thanks @MightyPork!
• Unsafe code is now forbidden.

Breaking changes

• Rename latest to to_latest. The old symbol is deprecated and will be removed eventually.
• An error is now returned when a migration is attempted while no migrations exist.

Other

• Improve general rust API documentation.
• Generate parts of the readme based on rust comments, for increased consistency with the docs.rs content.
• Various refactoring and clean-ups.

Version 0.3.1

Fix in readme, for crates.io

Version 0.3

New features

• Multi line sql statements like:

  M::up(r#"
  CREATE TABLE t1(a, b);
  CREATE TABLE t2(a, b);
  "#)
  

  are now fully supported

Other

• Various doc & CI improvements
• Fix a case of failure with silent errors.

Rusqlite Migration is a performant and simple schema migration library for rusqlite.

• Performance:
  • Fast database opening: to keep track of the current migration state, most tools create one or more tables in the database. These tables require parsing by SQLite and are queried with SQL statements. This library uses the user_version value instead. It’s much lighter as it is just an integer at a fixed offset in the SQLite file.
  • Fast compilation: this crate is very small and does not use macros to define the migrations.
• Simplicity: this crate strives for simplicity. Just define a set of SQL statements as strings in your Rust code. Add more SQL statements over time as needed. No external CLI required. Additionally, rusqlite_migration works especially well with other small libraries complementing rusqlite, like serde_rusqlite.

Example

Here, we define SQL statements to run with Migrations::new() and run these (if necessary) with Migrations::to_latest().

use rusqlite::{params, Connection};
use rusqlite_migration::{Migrations, M};

// 1️⃣ Define migrations
const MIGRATIONS_SLICE: &[M<'_>] = &[
    M::up("CREATE TABLE friend(name TEXT NOT NULL);"),
    // In the future, add more migrations here:
    //M::up("ALTER TABLE friend ADD COLUMN email TEXT;"),
];
const MIGRATIONS: Migrations<'_> = Migrations::from_slice(MIGRATIONS_SLICE);

fn main() {
    let mut conn = Connection::open_in_memory().unwrap();

    // Apply some PRAGMA, often better to do it outside of migrations
    conn.pragma_update_and_check(None, "journal_mode", &"WAL", |_| Ok(()))
        .unwrap();

    // 2️⃣ Update the database schema, atomically
    MIGRATIONS.to_latest(&mut conn).unwrap();

    // 3️⃣ Use the database 🥳
    conn.execute("INSERT INTO friend (name) VALUES (?1)", params!["John"])
        .unwrap();
}

Please see the examples folder for more, in particular:

• migrations with multiple SQL statements (using for instance r#"…" or include_str!(…))
• migrations defined from a directory with SQL files
• migrations to previous versions (downward migrations)
• migrations when using async

I’ve also made a cheatsheet of SQLite pragma for improved performance and consistency.

Built-in tests

To test that the migrations are working, you can add this in your test module:

#[test]
fn migrations_test() {
    assert!(MIGRATIONS.validate().is_ok());
}

The migrations object is also suitable for serialisation with insta, using the Debug serialisation. You can store a snapshot of your migrations like this:

#[test]
fn migrations_insta_snapshot() {
    let migrations = Migrations::new(vec![
        // ...
    ]);
    insta::assert_debug_snapshot!(migrations);
}

Optional Features

Rusqlite migration provides several Cargo features. They are:

• from-directory: enable loading migrations from *.sql files in a given directory

Active Users

This crate is actively used in a number of projects. You can find up-to-date list of those on:

• crates.io / lib.rs
• GitHub’s list of dependent repositories

A number of contributors are also reporting issues as they arise, another indicator of active use.

Minimum Supported Rust Version (MSRV)

This crate extends rusqlite and as such is tightly integrated with it. Thus, it supports the same MSRV as rusqlite. At the time of writing, this means:

Limits

1. Since this crate uses the user_version field, if your program or any other library changes it, this library will behave in an unspecified way: it may return an error, apply the wrong set of migrations, do nothing at all…

2. The user_version field is effectively a i32, so there is a theoretical limit (about two billion) on the number of migrations that can be applied by this library. You are likely to hit memory limits well before that though, so in practice, you can think of the number of migrations as limitless. And you would need to create 10 000 new migrations, every day, for over 5 centuries, before getting close to the limit.

Contributing

Contributions (documentation or code improvements in particular) are welcome, see contributing!

We use various tools for testing that you may find helpful to install locally (e.g. to fix failing CI checks):

• cargo-insta
• cargo-mutants

Acknowledgments

I would like to thank all the contributors, as well as the authors of the dependencies this crate uses.

Thanks to Migadu for offering a discounted service to support this project. It is not an endorsement by Migadu though.

⚡ TL;DR

When Opening the DB

PRAGMA journal_mode = wal; -- different implementation of the atomicity properties
PRAGMA synchronous = normal; -- synchronise less often to the filesystem
PRAGMA foreign_keys = on; -- check foreign key reference, slightly worst performance

And check user_version to apply any migrations, for instance with this Rust library.

When Closing the DB

PRAGMA analysis_limit=400; -- make sure pragma optimize does not take too long
PRAGMA optimize; -- gather statistics to improve query optimization

Introduction

SQL pragma are statements (like SELECT … or CREATE TABLE …) that change the database behaviors or call a special functions. This post is a short list of SQLite pragma I use in my projects built on SQLite, to get better performance and more consistency.

Performance

File system Interactions

The following pragma statements set the way SQLite deals with the file system.

journal_mode wal

By default, when applying changes, SQLite uses a rollback journal, which is basically a copy of the data before the changes. Changing the journal mode to “Write-Ahead Log” is known to bring significantly better performance in most cases. It also allows concurrent readers with one writer. To activate it, run:

PRAGMA journal_mode = wal;

Some less common use cases are incompatible with this journal mode, for instance having a database on a network file system. The full list of drawbacks is listed in the documentation¹.

synchronous normal

To ensure integrity of the database, one of the mechanism SQLite uses is the file system synchronization operations. However, these synchronizations are quite costly. With WAL journal mode enabled, your database will still be consistent while synchronizing less often:

PRAGMA synchronous = normal;

Compared to the default synchronous = full, committed transactions could be rolled back if there is a power loss (although not if the application crashes).

Optimize

To execute statement as efficiently as possible, SQLite has a query planner, which tries to read the tables to provide good performance (for instance by evaluating the WHERE clauses that select the fewest rows first).

This query planner sometimes needs to know whether a column has many values or only a few, repeated values (like a boolean column would). To know this, it can’t read the whole table, as that may prove as costly as running the part of the query that is being optimized, so it uses some statistics collected in internal tables.

Using optimize right before closing the database connexion collects the statistics for some table columns. These columns are chosen mainly based on the queries executed during the connexion: if a query had benefited from more accurate statistics, the corresponding columns are analyzed.

For instance:

PRAGMA analysis_limit=400;
PRAGMA optimize;

In the above example, pragma analysis_limit ensures that optimize won’t run for too long, by limiting the number of rows read.

Allow Using More Memory

These two pragma could result in better performance, depending on your hardware and software configuration.

• Keep temporary storage in memory: PRAGMA temp_store = 2. However, setting this pragma does not guarantee that the temporary storage will be held in memory. Please refer to the documentation for other parameters that may change the final outcome.
• Keep more of the database pages in memory: for instance, use 32 MiB of memory for this purpose with PRAGMA cache_size = -32000. Note that the OS cache is already keeping parts of the database file in RAM, so this might end up wasting memory.

Consistency

user_version

The database’s schema can evolve over time. Your application could start with a car table to represent a car but later on, you realize you want to represent bicycles, so you add a bicycle table.

To keep track of the different versions of the schema, some libraries maintain an internal table with a single row with a version number. It then performs migrations as needed. In our example, version 1 would have only the car table and version 2, also the bicycle table. The migration from version 1 to version 2 add the bicycle table.

SQLite offers the user_version pragma, to keep track of these versions. It is an integer at a fixed offset in the database file. It is simpler and more efficient than maintaining a table with versions, in particular because the table has to be found in the database file while the integer is available right away.

The drawback is that this is not portable between database engines. If you only use SQLite though, it is almost always the best option. To use it, you can write some code to check the value of the user_version pragma value right after opening the database. If it is lower than expected, then atomically 1) run the necessary migrations and 2) increment the user_version pragma value. I wrote a library to ease this task in rust and here is an example in python.

foreign_keys on

This may come as a surprise for folks with experience with other database system, but SQLite does not enforce foreign key constraints by default. Consequently, a foreign key can point to a row that does not exist.

This can be fixed with pragma foreign_keys:

PRAGMA foreign_keys = ON;

This comes at a performance cost however, because more checks are performed when inserting values with foreign keys. The cost is usually negligible but your mileage may vary.

Note: pragma foreign_key_check can be used to check a particular table for violated foreign key constraint. This can be useful before enabling foreign_keys on a database with existing data.

STRICT

Another peculiarity of SQLite is dynamic typing through type affinity. When you define a column of type INTEGER in most other database engines, a value of type TEXT inserted in that column will be converted or return an error. But with SQLite, type affinities mean that if the value is not of the expected type, it will be converted if possible or stored with a different type if necessary. That’s handy when prototyping, but you may want a stricter behavior for consistency with other major SQL databases or when working with strictly typed languages.

To this end, SQLite supports the STRICT keyword at table creation. For instance, if a table t is created like so:

CREATE TABLE t(a INTEGER) STRICT;

then

INSERT INTO t VALUES(1);
INSERT INTO t VALUES('1');

are successful but

INSERT INTO t VALUES('f');

is not and returns Runtime error: cannot store TEXT value in INTEGER column t.a (19).

Note: Without the STRICT keyword at table creation, this last INSERT INTO … statement would have been successful and would have just returned a string.

Go deeper

The full list of supported pragma with detailed descriptions is available in the documentation. Some more options can be tweaked for the specific scenario of running a SQLite for a server.