⚡ TL;DR

You can use negative filters like

git diff ':!*.lock'
git status ':!*.lock'

and on attributes:

git status ':(attr:!linguist-generated !linguist-vendored)'

You can even combine those patterns.

Excluding on a glob

I sometimes have many changes over multiple files in a Git repository. And then I want to display only the changes made to files in the src/ directory:

git diff '*/src/*'

Or, more interesting, I want all changes except those made to lock files:

git diff ':!*.lock'

Note that in the above examples, patterns are quoted, so that the shell does not expand the glob itself. Here, Git is interpreting the patterns¹. This feature is called “pathspec”.

Diving into pathspec

That feature can be used with commands like git ls-files, git status, git grep… And you can do more than exclude globs.

The above :!*.lock is actually a short form of :(exclude)*.lock. And quite a few other keywords exist, like top to force a match on the full path from the root of the repository, icase for a case insensitive match or attr to place particular constraints on the attributes of a file.

For example, you can use the linguist attributes understood by GitHub and other forges locally with:

• ':(attr:!linguist-generated !linguist-vendored)' matches files that have neither of these linguist attributes,
• ':(attr:linguist-generated)' ':(attr:linguist-vendored)' matches files with either attribute²,
• and that can be combined with file patterns and other attributes, like ':(icase,attr:linguist-vendored)*/LOCK' which would match a file named lock or Lock or even LoCk if it has the linguist-vendored attribute.

The manual contains even more details, like attributes gotchas, matching without interpreting globs…. Have fun!

💬 This is a comment on

Neovim 0.12 News (archive)

Neovim 0.12 was released earlier today. I don’t have enough time just now, but I’m sharing here the list of things from the release page that I want to try first.

Whole new features

Here are new features to replace some plugins and simplify my configuration:

• There is a new 'autocomplete' option. When set, completion suggestions appear in insert mode, without pressing a triggering shortcut. The 'complete' option also gains the ability to call arbitrary functions, as long as they follow the complete-functions interface. These two features combined could replace the many completion plugins that came and went over the years. I certainly hope to use them to remove mini.completion from my config.
  More importantly, they might help the ecosystem to standardize around a common interface for completion sources. At the moment, LSP servers play this role, but a full-blown LSP server implementation in every plugin is quite heavy. For example, crates.nvim supports some completion plugins (nvim-cmp, coq.nvim) and also exposes an LSP server for cross-compatibility support. In the future, such a crate might only implement the complete-functions interface and it could be used as a source by either native nvim completions or plugins.
• The other big highlight of this release is the native plugin manager, vim.pack. It was contributed mainly by Evgeni Chasnovski (known for mini.nvim). His guide of this new component is worth a read. In particular, you can pin plugins to a particular hash, for a version you have audited and update only once you have audited the changes in any new version.
• The default status line was reworked and integrates with vim.diagnostic.status(), vim.ui.progress_status() and an indicator for the new busy state. This will allow me to remove some custom logic to produce diagnostic status (E:2 W:3 for 2 errors and 3 warnings) and maybe even drop my custom status line code entirely.
• I will try to use the MarkSet to make user-placed marks visible in the gutter of the current buffer. I used plugins for that feature in the past, but it should be simple to implement it in configuration with MarkSet now.
• I’ll use vim.net.request() to replace some calls to external commands. It will make the configuration more portable and shorter — it might even be marginally faster, eliminating an external command call.
• 'diffopt' inline and inline:word will provide richer diff at the line level, a bit like delta.
• It’s nice to have native plugins for the undo tree (:Undotree) and to compare whole folders (:DiffTool).
• I plan to use the new treesitter selection shortcuts (an, in, [n and ]n in visual mode) to replace the deprecated treehopper plugin: I don’t really care about jumping to a particular highlight node, visual mode should work better for me.

Polish of existing features

This release introduces some nice performance improvements, for instance on Ctrl+r in insert mode and packadd.

Some handy shortcuts were added:

Finally, the UI was reworked: it is now less coupled with the core. So the core can be :restarted or :connected to with the same UI. The ui2 opt-in should also remove the dreaded “Press ENTER” message. I plan to give it a try and go back to cmdheight=0. Both of these settings are marked experimental, but if they are stable enough for me, I might get one more line for files.

💬 This is a comment on

DNS-PERSIST-01: A New Model for DNS-based Challenge Validation

The Source of Truth

Connecting to a website? Sending an email? Which server¹ you reach depends on DNS records. And CAA or SSHFP records establish trust for public key cryptographic protocols.

DNS records are the source of truth.

DNS API Tokens

Currently, wildcard certificates with Let’s Encrypt require to write an arbitrary string in a DNS record, for every single certificate renewed or issued. In practice, this often involves sharing a write API token to change the DNS. This is risky: should an attacker obtain the token, they can do a lot of damage: hijack traffic (and get valid certificates for it), receive and send emails using the corresponding domain…

And often API tokens are not scoped just to access one domain, through human error or because the provider lacks the feature. Then, they might give access to all domains in an account at the DNS provider and the attacker can move laterally to more targets.

With this in mind, I have not set up wildcard certificates with Let’s Encrypt so far. The risk simply outweighed the gains for the use cases I had. And I’m not alone: agnos was written to address the same concern. And the DNS challenge documentation calls this out: “Keeping API credentials on your web server is risky”.

DNS Challenges Without Tokens

So I’m really excited about this new challenge type, DNS-PERSIST-01. It is a static DNS TXT record that authorizes a particular Let’s Encrypt account² to get certificates, even wildcard ones, for a particular domain. No need to share very sensitive DNS API token!
It also means that Let’s Encrypt clients don’t need to support the API of your DNS provider to emit wildcard certificates. Pick any client and DNS provider you want, including a self hosted one.

Looking forward to this being fully implemented and available to everyone!

💬 This is a comment on

6-day and IP Address Certificates are Generally Available by Matthew McPherrin (via)

Let’s Encrypt has just announced that short-lived¹ certificates are generally available. They can also be used for IP addresses, which is especially useful for DNS over HTTPS. Those certificates could be smaller in the future, if information for validity checks is omitted. However, for now at least, these certificates still include revocation information.

Using Those New Certificates With Caddy

I wanted to give those certificates a try with Caddy. Caddy needs to use the shortlived Let’s Encrypt profile. The profile feature isn’t explicitly mentioned in the docs, but some community posts use a profile directive. This also works in the global configuration:

{
	cert_issuer acme {
		profile shortlived
	}
}

Reloading the Caddy configuration was not enough to prompt an early renewal, so I deleted the underlying certificate files and restarted Caddy:

sudo rm -rf /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/example.com
sudo systemctl restart caddy

And I got a brand new certificate for my domain name, valid only for a few days. It’s marginally smaller² than with the classic profile, mostly because these short-lived certificates also use the new generation hierarchy. But they should be roughly the same size as the tlsserver profile.

Conclusion

With Caddy, certificate renewal should be automated enough that short-lived certificates don’t cause any problems. But I’ll see with this experiment if there are any surprising pain points.

My understanding is that browsers only cache TLS sessions, not certificates: when a full handshake is performed, the full certificate chain is then sent anyway³. So even if the certificate expires more frequently, it is not sent more often. The slightly smaller certificate chain is thus a small net benefit.

💬 This is a comment on

Short URLs: why and how by Derek Sivers

In the Beginning Were Short URLs

I recently read Derek Sivers’s Short URLs: why and how, again. He makes a compelling case for using very short, but meaningful, URLs on your website. Very short here means one or two words at most, or even just 2 to 4 characters. On his website, that’s https://sive.rs/anna, https://sive.rs/plaintext https://sive.rs/1s, or indeed https://sive.rs/su for that particular post. That’s very different from typical URLs like https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/¹ or https://github.blog/changelog/2021-10-11-improved-notification-email-titles-for-issues-and-prs/.

Similarly, before I even read that post, the first few pages I created on my website had similar URLs: /cv or /open-source. Why? It’s easy to remember and you can type it (quickly!). They are also easy to share on the phone or face to face. Someone asks for my CV? Easy, go to cj.rs/cv (Charlie Juliett DOT Romeo Sierra SLASH Charlie Victor² on the phone).

Short URLs also look nice and tight. I even purchased cj.rs to replace joly.pw as the main domain in 2019³.

Derek makes similar points:

He then adds:

I did not really think about those last points⁴, but they make sense as nice complementary benefits.

Losing My Way and Finding It Back

If we were thinking alike about short URLs, then why did I use https://cj.rs/blog/lets-encrypt-caa-records-with-caddy/ for my last blog post? Why did I add the full title at the end of a path that is nested inside blog? Shouldn’t I have used something like letsencrypt-caa or caddy-caa?

Mainly I lost sight of the benefits of concise URLs. And Hugo has nice sections to organise content, set cascading parameters, list related pages…

But regardless of the tool, I certainly think in terms of categories for my pages. I remember things like “I have written this blog post” or “I have created this page at the root to showcase my open source work”.

A bit of hierarchical organisation makes sense even for the user. Some pages share a common theme, but in different categories. For instance, I have a blog post on an Asciinema module for Hugo and the page of that project both pertaining to the same underlying project. So ideally, they would be at cj.rs/hugo-asciinema and cj.rs/blog/hugo-asciinema instead of cj.rs/hugo-asciinema and cj.rs/announcement-hugo-asciinema.

Finally, even when there is only one page on a particular topic, some hierarchy is useful. Something like cj.rs/su can look cryptic in a tweet, whereas cj.rs/blog/su makes more sense. Both can still be easily remembered and typed. And having /blog/ for blog posts leaves space right under / for proper pages on the same topic that may exist later. While there are over one million 4-letter combinations in a-z and 0-9, meaningful words are much rarer. On the same example as above, I’m keeping /asciinema in store for something more closely related to Asciinema in the future. If I had spent that one word on the Hugo module, it would have been gone forever⁵.

Striking a Balance

Moving forward, I’ll revert to much shorter URLs using (for the path part):

• One or two words at the top level, for projects and generally pages that are not blog posts. Think /contact, /rusqlite-migration or /open-source.
• At most one folder, like /blog/ for blog posts. The exception would be multi-parts articles, which could be under /blog/topic/1, /blog/topic/2…
• That list may evolve if there are strong reasons, but I’ll keep in mind the memorability and direct typing benefits.

I host my own instance of miniflux, an RSS reader. I do it as a hobby, I enjoy the learning opportunities that come along the way.

One such opportunity presented itself in November 2023.

Back then, a Man-in-the-Middle attack was reported against jabber.ru. You can go read the full details on that blog post, but let’s go over its main aspects. Without the attacker, a client connects directly to the jabber.ru server over TLS:

TLS encrypts the connection. A certificate is also presented when the connection is established, to ensure authenticity and integrity. For this to work, the Certificate Authority (CA), a trusted third party, signs the certificate. In doing so, it affirms that it saw a given certificate associated with a particular domain¹. This protects against some Man-in-the-Middle attacks, where an attacker inserts itself between a client (say on a public wifi network) and the website, as in the next diagram. If that attacker does not manage to similarly be between the CA and the website, then it will be unable to decrypt or alter the exchange without being detected, because it can’t present a certificate signed by the CA.

Nonetheless in November 2023, a Man-in-the-Middle attack against jabber.ru succeeded. In that instance, the attacker leveraged a privileged position on the network. They intercepted all TLS traffic to the victim’s server, then tricked² the Let’s Encrypt CA into issuing a certificate for the attacker server. Then, they could decrypt the traffic from the client, re-encrypt it, potentially alter it and send it to the server.

There was no need to compromise the server for this attack to work. It simply leveraged a privileged network position.

In a reply on his blog, ACME developer Hugo Landau points out the following mitigation:

This got me thinking: how would I implement this mitigation for my self-hosted services? In this post, we will use the CAA DNS record to limit issuance to one particular account registered at one particular CA, as suggested above.

The Setup

TLS Proxy

My setup looks like this:

Here caddy acts as a TLS proxy. It manages the TLS certificate, requesting a new one from Let’s Encrypt as needed. It also terminates the TLS connection from the browser and then connects to the underlying service, like miniflux. This example could of course be generalized to any service proxied by caddy, in particular services handling more personal data, like a self-hosted webmail or contact server.

DNS

The domain hosting the RSS reader, r.cj.rs, is in fact a CNAME record pointing to the underlying machine hosting the various services, Olivine. It looks like this:

dog r.cj.rs

CNAME r.cj.rs.             5m00s   "olivine.joly.eu.org."
    A olivine.joly.eu.org. 5m00s   132.145.68.59

CAA Records

Going back to the mitigation we want to implement:

Let’s Encrypt provides some documentation for the CAA record (which stands for Certification Authority Authorization). The key points are:

• CAA records closest to the domain take priority: a CAA on r.cj.rs takes precedence over one on cj.rs.
• CAA follows CNAME redirects.
• The issue and issuewild properties control the certificate authority allowed. If only issue is present, then the same constraints apply to both normal and wildcard certificates.
• The accounturi parameter can restrict issuance to a particular account, like https://acme-v02.api.letsencrypt.org/acme/acct/1234567890

So in our case, we can simply add an issue CAA record with the accounturi parameter on the DNS record directly pointing to the machine (olivine.joly.eu.org). Then other domains can alias with CNAME to that DNS record and will inherit the CAA constraints. And if a service moves to a different server, the CAA constraint will be that of the new server, if any.

Finding the accounturi Caddy Uses

Caddy automatically creates a Let’s Encrypt account if none is configured. It then issues certificates against that account. But where is the account number?

By default on Ubuntu, the parameters for that account are stored in /var/lib/caddy/.local/share/caddy/acme/acme-v02.api.letsencrypt.org-directory/users/default/default.json³. There, the location key contains the accounturi:

{
  // ...
  "location": "https://acme-v02.api.letsencrypt.org/acme/acct/1968958296"
}

Putting It All Together

Summing up the above, we know that we need a CAA record with the issue property and the same accounturi as caddy. It looks like this in the Bind format:

IN CAA   0 issue "letsencrypt.org;accounturi=https://acme-v02.api.letsencrypt.org/acme/acct/1968958296"

This record is placed on olivine.joly.eu.org, where a A (or AAAA) record contains the IP address of the Olivine server. In turn, the RSS reader host is a CNAME pointing to that A record.

Testing

To test that the new setup works, we can try to issue a certificate for a new domain. Just add it to the Caddy configuration as usual, for instance like this:

test-caa.cj.rs {
  respond "Hello, world!"
}

Caddy logs should show that the certificate was successfully issued:

{
  "level":"info",
  // ...
  "msg":"certificate obtained successfully",
  "identifier":"test-caa.cj.rs"
}

You can similarly test that changing the accounturi in the CAA. After the various caches expire, attempts to issue a certificate fail:

{
  "level":"error",
  // ...
  "msg":"could not get certificate from issuer",
  "error":"HTTP 0 urn:ietf:params:acme:error:caa - During secondary validation: While processing CAA for test-caa.cj.rs: CAA record for olivine.joly.eu.org prevents issuance"
}

Just a Mitigation in the End

We have limited the certificates that can be issued to a particular CA and to a particular account. In turn, that account is tied to a private key stored on the server.

However, as Hugo Landau explains, this is only a mitigation. A well resourced attacker could still succeed. For instance, they could alter the CAA record read by the CA.

Thus, complementary defenses include monitoring certificate transparency to spot suspicious issuance. Cloudflare offers such a service. So does crt.sh. While CAs sometimes emit certificates without logging them to Certificate Transparency⁴, those certificates should be rejected by most browsers. In our case, with web services, that would be enough. Other clients communicating over TLS might not reject them though, like XMPP clients of the jabber.ru example.

Despite the limitations of this CAA-based approach, it was interesting to learn more about it and see how the various pieces fit together. I hope you enjoyed reading!

⚡ TL;DR

Open cargo info in Vim or neovim for the package under the cursor using these 4 lines of Lua.

Cargo info

Rust 1.82 was released a couple of days ago. It’s packed with improvements, but one in particular caught my eye. Cargo now has a info sub-command. It displays details about a package in the registry from the comfort of your terminal. Here is an example:

$ cargo info lazy_static
lazy_static #macro #lazy #static
A macro for declaring lazily evaluated statics in Rust.
version: 1.5.0
license: MIT OR Apache-2.0
rust-version: unknown
documentation: https://docs.rs/lazy_static
repository: https://github.com/rust-lang-nursery/lazy-static.rs
crates.io: https://crates.io/crates/lazy_static/1.5.0
features:
  spin        = [dep:spin]
  spin_no_std = [spin]
note: to see how you depend on lazy_static, run `cargo tree --invert --package lazy_static@1.5.0`

Vim and neovim generally composes well with other terminal tools. So how can we easily integrate cargo info and neovim?

Demo

The goal is to press a key and display the cargo info for the crate under the cursor in a Cargo.toml file. Like this:

Introducing keywordprg

In the above demonstration, we press K over a crates name. Then neovim executes the program set in keywordprg, appending the work under the cursor. So if the cursor is on lazy_static

[dev-dependencies]
iai = "0.1"
insta = "1.40.0"
lazy_static = "1.5.0"
#   ^ cursor is here when we press K
mktemp = "0.5"

and keywordprg is set to cargo info, the command cargo info lazy_static is executed in a new terminal.

We can further improve this. Since we want a fast answer (without checking the remote repository), let’s use cargo info --offline. And we would like pretty colors, let’s force that using --color=always.

We only want to alter keywordprg when a Cargo.toml file is edited, because it makes no sense to call cargo info in a Python file. We can use a ftplugin for toml files for that¹, so that the plugin is only loaded and executed when a toml file is executed. Furthermore we can change the setting only when opening a file named “Cargo.toml”, instead of changing it for every toml file. We can put the necessary configuration in ~/.config/nvim/ftplugin/toml.lua², like so:

if vim.endswith(vim.fn.bufname(), "Cargo.toml") then
  vim.opt_local.keywordprg = "cargo info --color=always --offline"
end

-- Add chars that are often part of keys, especially in rust crates
-- (https://toml.io/en/v1.0.0#keys)
vim.opt_local.iskeyword:append "-"

The Lua code above also changes the iskeyword preference. This is so that tokio-test is considered as one word (a “keyword” in Vim parlance), instead of two (tokio and test). With this set, doing K on tokio-test will behave properly, because tokio-test is passed to the command, instead of just tokio.

One final note: the above configuration was the only thing sourced in the demo. All the rest is default, vanilla, neovim 0.10.2. And this does not even require recent neovim features, it has been supported in Vim for many years.

Learnings

Obviously this could be further refined. Other options could be passed to cargo info, like -v to make it more verbose and list dependencies. Some more scripting could add useful features to further improve the integration. For this particular problem, there are even a full plugin with many more features. But the point is to showcase a simple and robust integration³, applicable in a wide variety of contexts. In particular when the use case is too niche, a full plugin is too costly to write and maintain for the benefits it provides.

Let’s highlight the takeaways from this post that are applicable in a wide variety of contexts.

First, a number of Neovim commands can do something with the keyword under the cursor. Adjusting what counts as a keyword boundary is often, but not always, done in the syntax file of a particular language. Or you may just want to make a different trade off and set your own iskeyword value.

Second, the K mapping is often used to look up a word in a documentation (it is set to open man pages by default). It even works in visual mode, looking up the selected text. And arbitrary Vim or shell commands can be used, instead of the default :Man.

Please do read the corresponding parts of the Vim manual linked in this section to learn more. Next time, you may come up with your own quick integration between Vim and another tool. Happy hacking!

⚠️ DISCLAIMER

This is not a comment on Reddit’s future profitability or any reflection on the stock performance. This is a technical discussion on how a relatively simple link-aggreggator can be hosted, with the trade-offs.

Reddit went public today. Its IPO document states¹ that they use AWS and GCP. But I cannot find a break-down of infrastructure costs separately.

That makes me wonder: what does the infrastructure costs of a pure link-aggreggator look like?

I’ll go over two sites aggregating links related to computing, Lobsters and Hacker News. They are relatively high-traffic websites. Sites popular on Hacker News in particular often go down due to the sudden popularity². It’s a little less common with Lobsters, but it happens there as well.

Lobste.rs

Lobsters is very transparent on its infrastructure:

Using public prices, those servers cost $78 ($48+$24+$6) a month to run. Obviously, there are other costs for monitoring, backups, managed DNS³…

I’ve requested some numbers on the load the site is facing and infrastructure utilization. I’ll update that blog post with the results.

Hacker News

It’s harder to find details on Hacker News’s infrastructure, but the moderator of the site answered questions about the infra in this thread⁴:

So a CPU from 2016 handles the load for a very popular site.

No CDN either, requests are made to news.ycombinator, with DNS records pointing to M5 Hosting IPs (2606:7100:1:67::26 and 209.216.230.207)⁵.

Conclusions

Those two popular link-aggregator offer to submit and comment on links, two historical features of Reddit. Of course now Reddit also has chat, image hosting and presumably a bunch of other features. It also has “73.1 million daily active uniques (“DAUq”), around the world”, probably orders of magnitude more than Hacker News. Finally, Reddit might⁶ theoretically have higher uptime, because Hacker News and Lobsters sometimes have their single-point-of-failure fail, for instance during an upgrade.

After those incidents though, users tend to post understanding comments, pointing out that they thought their Internet connection was faulty, before thinking that the aggregator could be down. This sort of simple hosting might be the right trade-off for a non-profit link-aggregator: simple, so rarely down due to an operational mistake and with base components simple enough that users are understanding when an outage happens.

Security note: all the releases are signed. You can even check that the files JS and CSS file in the plugin match the ones uploaded in release pages. Thanks to @ku1ik for providing these files directly!

Updating in Hugo

As usual to get the latest version the player on your website, you can run:

hugo mod get -u cj.rs/gohugo-asciinema

Changelogs

Here is the changelogs in the upstream Asciinema player for the updated versions:

• 3.7.0
• 3.6.4
• 3.6.3
• 3.6.2
• 3.6.1
• 3.5.0
• 3.4.0
• 3.3.0

Delay Updating the CSS and JS Files

I apologize for the delay in updating those versions in my Hugo module. I do have alerts set up when a new version comes in but I’m still testing manually to make sure that the updated version works well. This Hugo module being a side-project, I did not get the chance to do the testing any earlier. I’ve tested all those versions against various pages of my website, and it all worked well. But please feel free to report any problem you may encounter.

Moving forward, I’ll rely on a script to automate the bulk of the update, that should help updating only a few days after the release. That script is not perfect and will be improved incrementally.

Since this is a release related to Asciinema, here is an asciicast of that new script downloading the latest CSS and JS, making a commit and tagging it.

Please feel free to use this script and open a PR if you spot an update before I do in the future 😁.

I’m now signing my git commit and tags with an SSH key. Details of the fingerprint can be found in the security document. It says that commit after 2024-01-01 are going to be signed, because I’m starting now on one machine and I will propagate the configuration over the next few days to other machines.

Why

Why bother with cryptographic signatures?
Anyone can pretend to be me. They just need to write my name and email in the author fields of a commit message. However¹, I’m the only one able to produce signatures with that particular public key. This will help to check that I’m actually the author of the commits and tags you rely on when using my code.

I’m doing it only now because GPG can be quite hard to use, especially with multiple machines. So I had to wait for the SSH signing scheme in Git to be supported more widely.

Verification

Code Hosting Platforms

GitHub will check this, adding a “verified” badge like this:

Codeberg shows similar badges.

Locally

Signatures can of course also be verified locally. This blog post explains in details how to do it.
TL;DR: populate a file (allowed_signers) with trusted keys, configures git to use it and then commands like git log --show-signature will check that signatures are valid for each commit.

Caveats

I wrote earlier that I was the only one able to generate this signature for this particular public key. This is true only as long as the corresponding secret key remains secret. I’m using a strong password to encrypt that key on my disk and that password is accessible only by physically touching a Yubikey. This goes a long way towards preventing the private key leaking. However, it is still possible for an attacker taking full control of my machine for extended periods of time to intercept that password and to decipher the secret key with it. Then, they will be able to produce signatures as if they were me. It would be a bit harder for the attacker if the SSH key was on the Yubikey, but then it becomes tricky to work on multiple machines. So that’s the setup for now, a compromise between potentially higher level of security and usability so that I actually use it.

Currently, I’m not signing release artifacts, but I might in the future. GPG is more common for this, but it’s very hard to maintain long term GPG keys. With this allowed_signers SSH file, rotating keys seems easier, so I’m more likely to do it more often and limit risks. I’ll try with only commit and tag signing first. Then I’ll apply the learnings to sign release artefacts.

EDIT(2023-12-31): This blog post describes the various options to sign commits very well, in particular talking about revocation scenarios. It was written shortly after I wrote this piece. I mostly agree with the author, but I think that SSH is a good middle ground: revocation works very well locally, it can be paired with Yubikey verification and it’s already supported by code hosting providers, without pesky OpenID verifications like gitsign.

⚡ TL;DR

A context-aware snippet for Go error handling code, returning the right types, with the default values.

• Demo
• Snippet

Introduction

Golang’s error handling is notoriously verbose. It was also the top pain point in the Go Developer Survey Q2 2022. Numerous proposals to simplify error handling have been written, but at the time of writing, none have been accepted.

So we have to live with the current state of things. It’s ok, Neovim with Treesitter has a good “understanding” of code, so we can use it to generate the error handling code.

Treesitter

One of Neovim 0.5 most exciting features was the introduction of Treesitter. This new parsing system gives the editor a basic understanding of the code at hand: the editor “knows” what the function name is, where a variable is defined… It enables great plugins, showing the context around some code, improved selection, better renaming and navigation and more! In this post, we will look into how Treesitter can be combined with LuaSnip for smarter snippets.

Error Handling in Go

In Golang, errors are handled by returning a value of type error, sometimes along the types returned when everything goes well. For instance, the time.Parse function has the following signature:

func Parse(layout, value string) (Time, error)

time.Parse will try to parse a string with the given format. If it’s successful, it’ll return a value of type Time and the error will be nil. If a date can’t be parsed, then the error won’t be nil. And so it’s common in Go to check for errors like so:

timeStr := "Aug 6, 2022 @ 7:54pm"
t, err := time.Parse("Jan 2, 2006 @ 3:04pm", timeStr)
if err != nil {
    log.Fatal(err)
}
// Now, we can safely use t

In the example above, we just log the error and panic. But if we are in a function, we will likely return an error to the caller, wrapping the error returned by time.Parse. For instance:

1func parseLong(timeStr string) (Time, error) {
2    t, err := time.Parse("Jan 2, 2006 @ 3:04pm", timeStr)
3    if err != nil {
4        return t, fmt.Errorf("couldn’t parse %v: %v", timeStr, err)
5    }
6    return t, nil
7}

The key takeaway here is that the return line in the error case (line 4 above) depends on the return type of the function and so can vary significantly.

This is covered in more details in this post on the Go blog.

Clever Snippets with LuaSnip and Treesitter

As we have seen, error handling code varies depending on the number and type of the values returned by the parent function. So we are very often writing variations of the following in our Go code:

func f(arg1 string, arg2 string) (…, error) {
    val, err := someFunction(arg1, arg2)
    if err != nil {
        return …, err
    }
    …
}

Or with 3 return values:

func f(arg string) (…, …, error) {
    val, err := someFunction(arg)
    if err != nil {
        return …, …, err
    }
    …
}

It would be handy to have a snippet that would adjust the return’s shape depending on the return type of the function that contains it.

Demo

Here is a short asciicast demonstration¹ of the snippet we are going to build. It is called smart_err:

The snippet inserted (line 11 in the asciicast) is

myVal, myErr := anotherFunc(arg1, arg2)
if myErr != nil {
	return false, fmt.Errorf("anotherFunc: %v", myErr)
}

when the return type of the function was (bool, error).

And

val, err := f()
if err != nil {
	return false, fmt.Errorf("f: %v", err), 0, ""
}

when the return type was (bool, error, int, string).

And it even works in nested functions:

func myFunc() (*MyStruct, error) {
    // ...
	a := func() (bool, error, int, string) {
		// The snippet detected that we need 4 values here,
		// and not 2 as in the outer function
		return v, nil, 0, ""
	}

Also note that we can choose to wrap the containing error, or not, with a keystroke (bound to require("luasnip").change_choice(1)). The snippet switches between the following 3 forms (line 18 in the asciicast):

1. return false, err
   

2. return false, fmt.Errorf("f: %v", err)
   

3. return false, errors.Wrap(err, "f")
   

That’s pretty cool! Let’s see how to add it to your Neovim configuration.

Code

The code to define this snippet was initially written by TJ DeVries². I’ve only slightly adapted it so that it fits in one file. I believe the author welcomes this³:

Copy this alongside your other LuaSnip snippets, for instance in ~/.config/nvim/snippets/go.lua:

  1local ls = require "luasnip"
  2local f = ls.function_node
  3local s = ls.s
  4local i = ls.insert_node
  5local t = ls.text_node
  6local d = ls.dynamic_node
  7local c = ls.choice_node
  8local snippet_from_nodes = ls.sn
  9
 10local ts_locals = require "nvim-treesitter.locals"
 11local ts_utils = require "nvim-treesitter.ts_utils"
 12local get_node_text = vim.treesitter.get_node_text
 13
 14-- Adapted from https://github.com/tjdevries/config_manager/blob/1a93f03dfe254b5332b176ae8ec926e69a5d9805/xdg_config/nvim/lua/tj/snips/ft/go.lua
 15local function same(index)
 16  return f(function(args)
 17    return args[1]
 18  end, { index })
 19end
 20
 21-- Adapted from https://github.com/tjdevries/config_manager/blob/1a93f03dfe254b5332b176ae8ec926e69a5d9805/xdg_config/nvim/lua/tj/snips/ft/go.lua
 22vim.treesitter.query.set(
 23  "go",
 24  "LuaSnip_Result",
 25  [[ [
 26    (method_declaration result: (_) @id)
 27    (function_declaration result: (_) @id)
 28    (func_literal result: (_) @id)
 29  ] ]]
 30)
 31
 32-- Adapted from https://github.com/tjdevries/config_manager/blob/1a93f03dfe254b5332b176ae8ec926e69a5d9805/xdg_config/nvim/lua/tj/snips/ft/go.lua
 33local transform = function(text, info)
 34  if text == "int" then
 35    return t "0"
 36  elseif text == "error" then
 37    if info then
 38      info.index = info.index + 1
 39
 40      return c(info.index, {
 41        t(string.format('fmt.Errorf("%s: %%v", %s)', info.func_name, info.err_name)),
 42        t(info.err_name),
 43        -- Be cautious with wrapping, it makes the error part of the API of the
 44        -- function, see https://go.dev/blog/go1.13-errors#whether-to-wrap
 45        t(string.format('fmt.Errorf("%s: %%w", %s)', info.func_name, info.err_name)),
 46        -- Old style (pre 1.13, see https://go.dev/blog/go1.13-errors), using
 47        -- https://github.com/pkg/errors
 48        t(string.format('errors.Wrap(%s, "%s")', info.err_name, info.func_name)),
 49      })
 50    else
 51      return t "err"
 52    end
 53  elseif text == "bool" then
 54    return t "false"
 55  elseif text == "string" then
 56    return t '""'
 57  elseif string.find(text, "*", 1, true) then
 58    return t "nil"
 59  end
 60
 61  return t(text)
 62end
 63
 64local handlers = {
 65  ["parameter_list"] = function(node, info)
 66    local result = {}
 67
 68    local count = node:named_child_count()
 69    for idx = 0, count - 1 do
 70      table.insert(result, transform(get_node_text(node:named_child(idx), 0), info))
 71      if idx ~= count - 1 then
 72        table.insert(result, t { ", " })
 73      end
 74    end
 75
 76    return result
 77  end,
 78
 79  ["type_identifier"] = function(node, info)
 80    local text = get_node_text(node, 0)
 81    return { transform(text, info) }
 82  end,
 83}
 84
 85-- Adapted from https://github.com/tjdevries/config_manager/blob/1a93f03dfe254b5332b176ae8ec926e69a5d9805/xdg_config/nvim/lua/tj/snips/ft/go.lua
 86local function go_result_type(info)
 87  local cursor_node = ts_utils.get_node_at_cursor()
 88  local scope = ts_locals.get_scope_tree(cursor_node, 0)
 89
 90  local function_node
 91  for _, v in ipairs(scope) do
 92    if
 93      v:type() == "function_declaration"
 94      or v:type() == "method_declaration"
 95      or v:type() == "func_literal"
 96    then
 97      function_node = v
 98      break
 99    end
100  end
101
102  local query = vim.treesitter.query.get("go", "LuaSnip_Result")
103  for _, node in query:iter_captures(function_node, 0) do
104    if handlers[node:type()] then
105      return handlers[node:type()](node, info)
106    end
107  end
108
109  return { t "nil" }
110end
111
112-- Adapted from https://github.com/tjdevries/config_manager/blob/1a93f03dfe254b5332b176ae8ec926e69a5d9805/xdg_config/nvim/lua/tj/snips/ft/go.lua
113local go_ret_vals = function(args)
114  return snippet_from_nodes(
115    nil,
116    go_result_type {
117      index = 0,
118      err_name = args[1][1],
119      func_name = args[2][1],
120    }
121  )
122end
123
124return {
125  -- Adapted from https://github.com/tjdevries/config_manager/blob/1a93f03dfe254b5332b176ae8ec926e69a5d9805/xdg_config/nvim/lua/tj/snips/ft/go.lua
126  s("smart_err", {
127    i(1, { "val" }),
128    t ", ",
129    i(2, { "err" }),
130    t " := ",
131    i(3, { "f" }),
132    t "(",
133    i(4),
134    t ")",
135    t { "", "if " },
136    same(2),
137    t { " != nil {", "\treturn " },
138    d(5, go_ret_vals, { 2, 3 }),
139    t { "", "}" },
140    i(0),
141  }),
142}

We start by importing and aliasing some LuaSnip functions. The snippet itself is defined at the end (line 126). Most of those functions were covered in the previous post, see there for calls to s, i and t. The clever part of this snippet is the dynamic node d on line 138. There, we call a function that queries (line 22 and 102) Treesitter to find the return types of the current function in Go code. Depending on the types found, default values for the type are used (line 33).

Treesitter Query

The Treesitter query is where the magic happens, so let’s take a deeper look.

Treesitter parses code and builds a tree with node for various parts of the code. Each node can have children, like a function definition has a name, a body… This tree can be queried in a Lisp-like language (s-expressions). For instance, in our snippet definition, we use this query:

1[
2    (method_declaration result: (_) @id)
3    (function_declaration result: (_) @id)
4    (func_literal result: (_) @id)
5]

The bracket syntax returns the nodes that match any of the patterns inside the brackets. So it’s effectively the union of all the nodes matched by any of the 3 patterns (on lines 2 to 4). Let’s take a concrete example and put the Go code in the Treesitter playground, with a trimmed down version of the query:

The tree built by Treesitter is at the bottom, with the corresponding code at the top and the query in the middle. The function_declaration node is highlighted in gray in the tree, like the corresponding part of the code (from line 13 to line 22). This node has a number of children, with the function name, the parameters of the function, a result node for the return type and body for the body of the function between curly braces.

Taking the first pattern on the playground, slightly modified:

    (function_declaration result: (_) @id2)

The part it captures is highlighted in blue in the Go code: (*MyStruct, error). It makes sense, since the function_declaration (the part on the gray background) has a result child, that’s captured by @id.

Now, on to the second pattern:

    (func_literal result: (_) @id3)

Similarly, further down in the body of that function, the return type (bool, error, int, string) of the anonymous function is matched.

Conclusion

I started writing this post over a year ago. The snippet has been very stable and kept working as without changes, which is a testament to the stability of the LuaSnip (and Treesitter) APIs. I’m still using it when I write Go code, even if some corner cases (like custom structs) aren’t handled. The snippet is quite complicated and niche, so I’m not sure that I would have invested the time to write it myself. I probably don’t write enough Go code to justify the investment, but finding it shared by someone buried in their config file made it worth it. I hope you have found this port useful, either to copy this particular snippet or as an example of advanced snippets.

This is the last post in a series on LuaSnip.

Resources

If you want to dig deeper:

• Working with Errors in Go 1.13
• LuaSnip documentation
• Explore the Treesitter tree for any Neovim buffer where it is active with :lua vim.treesitter.inspect_tree()
• Treesitter’s Pattern Matching with Queries

• How we built Network Analytics v2

Archived copies:

• WaybackMachine
• Archive.is
• Perma.cc

⚡ TL;DR

As a small start-up time optimization, you can pick the best suited compression algorithm for the initial ramdisk.

The Initial Ramdisk

When a Linux system boots, it needs to mount the root filesystem /. This may be relatively complicated, as it may be on a software RAID, on LVM, encrypted… To keep things manageable, an initial ramdisk can be used to get a small environment that has all the required modules and configuration to load the root filesystem. On Arch Linux, this initial ramdisk is generated using mkinitcpio. It takes multiple parameters to tune various aspects of the system and of the generated ramdisk.

Compression

One such parameter is COMPRESSION. It compresses the ramdisk to make the resulting image smaller. The manpage reads:

Another reason to compress the image is that it may reduce the start-up time. To understand why, imagine that the image is 100 MiB in size and only 20 MiB after compression. Let’s say that the disk reads 10 MiB¹ per second and that the CPU can decompress the full image in 1 second. If we keep the image uncompressed, the disk will need 10 seconds to read the uncompressed image, while it needs only 2 seconds to read the compressed image. Adding the decompression time, the compressed version require only 3 seconds.

Trade-offs

The above example is quite simple, but it illustrates the trade-off between a bigger image that the disk will take longer to read and a smaller image that may take longer to decompress. It is thus more of a spectrum, where more CPU-intensive compression (and decompression) methods could result in a smaller image and less read from the disk but more CPU time:

Then, the question is: is it worth compressing an image more (or at all), to get a faster start-up time?

Protocol

To answer this question on a particular machine², let’s compare the time required to read and decompress various initial ramdisks.

I’m using the linux package in version 5.18.15-arch1-2 from the Arch Linux repository. Then, I generate (sudo mkinitcpio -p linux) various images with the following parameters in /etc/mkinitcpio.conf:

• COMPRESSION="cat"
• COMPRESSION="lz4"
• COMPRESSION="zstd"

Each image is copied in a directory and renamed according to the compression used: cp /boot/initramfs-linux.img initramfs-linux.img.zstd. The result is as follows:

$ file *.img*
initramfs-linux.img:      ASCII cpio archive (SVR4 with no CRC)
initramfs-linux.img.lz4:  LZ4 compressed data (v0.1-v0.9)
initramfs-linux.img.zstd: Zstandard compressed data (v0.8+), Dictionary ID: None

The images are compressing quite well too:

We could compare more algorithms and compression level, but compression levels would need to be passed through COMPRESSION_OPTIONS, which the manpage discourages, as it can result in an unbootable image.

Results

Let’s run some decompression commands and compare their run-time with hyperfine. On a quiet computer:

$ hyperfine \
    --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' \
    'lz4 -d <./initramfs-linux.img.lz4' \
    'zstd -d <initramfs-linux.img.zstd' \
    'cat <initramfs-linux.img'

Note that the command has a --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' argument. This empties the OS file system caches to be closer to start-up conditions: when the computer starts, everything has to be read from the disk as the RAM is basically empty. Without this --prepare argument, we get much shorter times, e.g. 45ms for lz4.

Here are the results:

Lz4 is slightly faster, followed by zstd and no compression at all with cat. If we go back to the sizes table, the trade-off between a smaller image but a slower decompression is clear. Despite a ~30% smaller file size, zstd is still a bit slower to decompress than lz4, while no compression at all is even worse.

Conclusion

The above results are based on runs on a particular machine. As mentioned different machines will yield different results, depending on the relative performance of the disk and the CPU. It’s also a pretty small improvement in the grand scheme of things: only a few tens of milliseconds on a process that takes a couple seconds. But I found it to be a nice example of how compression can make things faster, compared to no compression at all, because CPU nowadays are so fast.

Appendix: Recording of the hyperfine Run

This was done on a different run from the table above, as running the benchmark through Asciinema is sometimes a bit less stable):

⚡ TL;DR

When you have many variations of the same snippet, one option is to generate those with Lua code. The complete example is at the end.

I’ve recently moved to LuaSnip as my snippets plugin for Neovim. When I first started, I sticked to the simplest features of LuaSnip, in particular the SnipMate-like syntax for snippets. But I have now started to explore the more distinctive features of LuaSnip, like Lua-defined snippets. It turns out that generating snippets with code can save tedious repetition.

Markdown Journaling

I tend to take notes in Markdown documents. I usually set up one text file for each recurring themes or project. Then, there is a “h1” title per day, a bit like the Bullet Journal daily log. The resulting file looks like this:

# 2022-07-31 Sun

* Point 1
* Point 2
* Point 3

I have a snippet to generate the # 2022-07-31 Sun line, based on the current date and day of the week, as I explained last time. For reference, here is the snippet from back then:

# “Bullet Journal”-styled date for today
snippet bjtoday
  # ${CURRENT_YEAR}-${CURRENT_MONTH}-${CURRENT_DATE} $CURRENT_DAY_NAME_SHORT

Lua-Generated Snippets

But then, I also sometimes want to put notes for the next session. I usually know when that will be, because the meeting happens, say, every Thursday or every month. So it would be great to have snippets for those dates as well. Let’s build them step by step.

1. Computing the Date of “Next Friday”

The first step is to calculate the date and weekday from human concepts like “next Friday” or “next week”. Thankfully GNU date supports this, and it’s preinstalled on most GNU/Linux systems:

$ date -d 'next Friday' +'%F %a'
2022-08-05 Fri

2. Putting Those Dates in LuaSnip

The second step is to use the output of the date command in LuaSnip.

We could define our own variables like ${NEXT_MONDAY_DATE}. That would be similar to the ${CURRENT_DATE} we used before. But we would need many such variables, for the date and weekday of the next Monday, Tuesday, …, next week or next month. Plus I use those dates only in Markdown snippets, so it feels a bit wasteful to define those variables everywhere.

In the end, I chose to use function nodes to call the date command on the fly, when the snippet gets expanded. This node is inserted in a particular snippet definition and is thus contained to that particular snippet. This way, we avoid creating values everywhere, and we keep the global namespace clean. It looks like this:

f(function(args, snip, user_arg_1)
    return vim.fn.trim(vim.fn.system([[date -d ']] .. target_date .. [[' +'%F %a']]))
end, {}),

3. Generating Snippets Variants

The last step is to generate the variants of the base snippet: bj_today, bj_next_monday, …, bj_next_week and bj_next_month.

In LuaSnip, snippets defined using Lua are just a table calling some agreed-upon functions. Here is an example from the documentation:

ls.add_snippets("all", {
	s("ternary", {
		-- equivalent to "${1:cond} ? ${2:then} : ${3:else}"
		i(1, "cond"), t(" ? "), i(2, "then"), t(" : "), i(3, "else")
	})
})

This creates a snippet (the s function) for all file types. This snippet will expand ternary into cond ? then : else, prompting the user to change the cond, then and else bits (function i) while ? and : are “static text” (function t).

We want to write a function that builds a table of date snippet definitions. That will be returned when the Lua module is loaded.

The Final Code

The final snippet looks like this:

 1-- Bullet Journal styled dates in the future
 2local function gen_date_snippets()
 3  local snippets = {}
 4  local target_dates = {
 5    "today",
 6    "tomorrow",
 7    "next monday",
 8    "next tuesday",
 9    "next wednesday",
10    "next thursday",
11    "next friday",
12    "next week",
13    "next month",
14  }
15  for _, target_date in pairs(target_dates) do
16    table.insert(
17      snippets,
18      s("bj_" .. target_date:gsub(" ", "_"), {
19        t "# ",
20        f(function(args, snip, user_arg_1)
21          return vim.fn.trim(vim.fn.system([[date -d ']] .. target_date .. [[' +'%F %a']]))
22        end, {}),
23      })
24    )
25  end
26
27  return snippets
28end
29
30return gen_date_snippets()

This generates the snippets name (line 18), for instance bj_today or bj_next_tuesday from the arguments passed to the date command. The body of the snippet is made of text (t "# ") and our function node from earlier (line 20-22). All those snippets get collected in a table that is returned at the end of the Lua module (line 30). It’s then usable by LuaSnip.

Conclusion

Generating a bunch of similar snippets like this turned out to be an unanticipated benefit of using Lua to define snippets. Before using LuaSnip, I only had bj_today, bj_tomorrow and bj_next_monday. If I had wanted to expand this set further, I would have to do some more copying and pasting, which quickly becomes unmaintainable.

Of course for simpler or less repetitive snippets, it’s probably best to use the less verbose SnipMate-like syntax presented in the previous post.

⚡ TL;DR

The Default trait can enhance the maintainability of your code. Default values for common types are listed at the end.

A PR Review

Recently, while reviewing a PR¹, I noticed that part of the patch was introducing a new field to a struct:

 1diff --git a/src/lib.rs b/src/lib.rs
 2index eba9a3a..8619e06 100644
 3--- a/src/lib.rs
 4+++ b/src/lib.rs
 5@@ -106,8 +108,9 @@ use std::{
 6 #[derive(Debug, PartialEq, Clone)]
 7 pub struct M<'u> {
 8     up: &'u str,
 9     down: Option<&'u str>,
10+    foreign_key_check: bool,
11 }
12 
13 impl<'u> M<'u> {
14@@ -137,8 +140,9 @@ impl<'u> M<'u> {
15     pub const fn up(sql: &'u str) -> Self {
16         Self {
17             up: sql,
18             down: None,
19+            foreign_key_check: false,
20         }
21     }

That prompted me to reflect on the code I had initially written. Prior to the patch, it looked roughly² like this:

 1#[derive(Debug, PartialEq, Clone)]
 2pub struct M<'u> {
 3    up: &'u str,
 4    down: Option<&'u str>,
 5}
 6
 7impl<'u> M<'u> {
 8    pub const fn up(sql: &'u str) -> Self {
 9        Self {
10            up: sql,
11            down: None,
12        }
13    }
14}

The Default Trait

What if I had used the Default trait here? The code could have looked like this:

 1#[derive(Debug, Default, PartialEq, Clone)]
 2pub struct M<'u> {
 3    up: &'u str,
 4    down: Option<&'u str>,
 5}
 6
 7impl<'u> M<'u> {
 8    pub const fn up(sql: &'u str) -> Self {
 9        Self {
10            up: sql,
11            ..Default::default()
12        }
13    }
14}

On the first line, the #[derive(Default)] attribute makes the structure M implement the Default trait. Thanks to this trait, a call to M::default() will create a struct with default values for its fields: M { up: "", down: None }. Note that when a structure M is expected, Default::default() is equivalent to M::default().

We then need to initialize the two fields of that structure, overriding some defaults:

• up is defined directly as before. That’s the value we want to override.
• down is set by the Default trait. This is done by ..Default::default() on line 11. Default::default() provides the values. Then the .. syntax fills out the fields that were not directly set. down is thus set to the same value as before, None.

The code is just as long as before, when we were not using the Default trait. But then, line 19 of the above patch would have been unnecessary: false is the default for a bool, so the new foreign_key_check field would have been covered by the ..Default::default().

This results in a shorter patch:

 1diff --git a/src/lib.rs b/src/lib.rs
 2index eba9a3a..8619e06 100644
 3--- a/src/lib.rs
 4+++ b/src/lib.rs
 5@@ -106,8 +108,9 @@ use std::{
 6 #[derive(Debug, PartialEq, Clone)]
 7 pub struct M<'u> {
 8     up: &'u str,
 9     down: Option<&'u str>,
10+    foreign_key_check: bool,
11 }
12 
13 impl<'u> M<'u> {

Conclusion

In the example of this post, we are doing only one instantiation of that particular struct, and it has very few fields anyway. But if there were many instantiations of that struct, we would have had to change all of those. Then, using Default would have been quite beneficial.

This pattern were the return type is built with a call to Default::default() seems relatively common in the Rust organization. It can enhance maintainability, much more than I initially thought.

Of course this is a balancing act. For instance, this pattern could be abused by defining custom default values on primitive types for a particular structure. That would lead to Default::default() filling surprising values and the code would be less predictable.

Appendix: Defaults for Some Common Types With derive

Why did I not use the Default trait initially? Part of it might be the fear of introducing incorrect code. It was slightly unclear to me what derive uses as a default value for common primitive types. And you don’t want a field set explicitly to false becoming a true once you use Default, right?

Let’s take a closer look by running the following program:

#[derive(Debug, Default)]
struct D<'a> {
    b: bool,
    c: char,
    o: Option<usize>,

    string: String,
    str: &'a str,

    v: Vec<usize>,
    a: [usize; 10],
    s: &'a [u32],

    f: f64,
    u: (usize, u32, u64)
}

fn main() {
    println!("{:#?}", D::default());
}

Output:

D {
    b: false,
    c: '\0',
    o: None,
    string: "",
    str: "",
    v: [],
    a: [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
    ],
    s: [],
    f: 0.0,
    u: (
        0,
        0,
        0,
    ),
}

It turns out that in general, the default for common types in the standard library is a 0 byte in the underlying data structure. Note that arrays are fixed size in rust and thus the default is an array of the right size, filled with defaults for the inner type. That’s quite similar to go.

Quick Reference

Here is a table for future reference:

⚡ TL;DR

LuaSnip is fast and doesn’t have to be complicated. Give it a try!

Introduction

Snippets are a convenient feature of some text editors to insert and adapt reusable pieces of code. For instance, snippets for for loops are common, to get the tedious bits of the syntax out of the way.

To get this feature in Vim back in the days, I started using UltiSnips. There are default snippets sets for it, and it’s easy to write custom snippets. These custom snippet can call Bash or Python scripts if you need more dynamic replacements. UltiSnips has been very powerful and has served me quite well over the past decade or so, and I have kept it when I migrated to NeoVim a few years ago.

Startup time

Every once in a while though, I run the excellent vim-startuptime command to assess the impact of various configurations and plugins on the startup time of Neovim. With UltiSnips and the corresponding completion plugins in my configuration, the first few lines are:

Extra options: []
Measured: 10 times

Total Average: 104.218300 msec
Total Max:     109.719000 msec
Total Min:     99.533000 msec

  AVERAGE       MAX       MIN
------------------------------
64.218600 70.419000 61.066000: ~/.config/nvim/init.lua
 6.255900  7.238000  5.764000: loading packages
 3.939000  5.338000  3.533000: ~/.local/share/nvim/site/pack/paqs/start/onedark.nvim/colors/onedark.lua
 3.651100  4.003000  3.381000: loading rtp plugins
 2.761600  3.957000  2.474000: expanding arguments
 2.683900  3.035000  2.566000: reading ShaDa
 2.418700  3.610000  2.131000: sourcing vimrc file(s)
 2.389600  2.751000  2.228000: /usr/share/nvim/runtime/filetype.lua
 1.954900  2.293000  1.702000: loading after plugins
 1.842500  2.015000  1.763000: BufEnter autocommands
 1.583350  3.532000  0.018000: ~/.local/share/nvim/site/pack/paqs/start/ultisnips/plugin/UltiSnips.vim
 1.027700  1.170000  0.831000: ~/.local/share/nvim/site/pack/paqs/start/nvim-treesitter/plugin/nvim-treesitter.lua
 0.968200  1.071000  0.860000: ~/.local/share/nvim/site/pack/paqs/start/cmp-nvim-ultisnips/after/plugin/cmp_nvim_ultisnips.lua
 0.859000  1.014000  0.702000: ~/.local/share/nvim/site/pack/paqs/start/nvim-treesitter-textobjects/plugin/nvim-treesitter-textobjects.vim
 0.590700  0.674000  0.524000: ~/.local/share/nvim/site/pack/paqs/start/indent-blankline.nvim/plugin/indent_blankline.vim
 0.557200  0.817000  0.493000: init highlight
 0.553200  0.898000  0.322000: opening buffers

The init.lua line covers a lot of different plugins and mappings set up in that file. Besides the onedark.nvim colorscheme¹, the next biggest contributor is ~/…/ultisnips/plugin/UltiSnips.vim. The script to integrate UltiSnips with cmp, ~/…/cmp-nvim-ultisnips/after/plugin/cmp_nvim_ultisnips.lua, is not far behind. In total, we spend nearly 4 ms of the startup time for UltiSnips-related files, on top of the setup done in init.lua. That feels suboptimal, so I’ve been looking for a possible snippet plugin alternative.

Installing LuaSnip

LuaSnip aims to be a faster² snippet engine, with support of treesitter in the snippets. It can also understand the LSP snippet “format”. Finally, it’s a pure Lua plugin, without any Python requirement, contrary to UltiSnips. That makes installation on various system much easier.

Its drawbacks for me are that it does not support the same snippet definition as UltiSnips and even in its own SnipMate-like syntax, backticks to execute code are not supported. As a result, I’ll have to migrate my (admittedly very small) snippet collection. Conversely, if I want to do more advanced things, I’ll have to learn the relatively complex “VS Studio Code” snippets in JSON or even the pure Lua snippets.

That’s said, I believe I can get the LuaSnip benefits without writing Lua snippets or JSON ones, at least to start. So let’s try and do that, using only the SnipMate-like syntax!

Overview of the Configuration Changes

I’ve made the following changes.

1. Remove UltiSnips and its associated plugins, namely for completion with cmp and the telescope integration.
2. Remove the UltiSnips mappings.
3. Add LuaSnip (following the instructions in the Readme), its cmp integration, a set of snippets and a telescope integration along with some mappings (requires nvim 0.7+³):

   vim.keymap.set({ "i", "s" }, "<C-i>", function() require'luasnip'.jump(1) end, { desc = "LuaSnip forward jump" })
   vim.keymap.se({ "i", "s" }, "<M-i>", function() require'luasnip'.jump(-1) end, { desc = "LuaSnip backward jump" })
   

4. Migrate my existing snippets, see below.
5. Add code to load the snippet set and my own snippet collection:

   require("luasnip.loaders.from_vscode").lazy_load()
   require("luasnip.loaders.from_snipmate").lazy_load({ paths = {"./snippets"} })
   

After

Let’s run vim-startuptime again. Here are the new top contributors:

Extra options: []
Measured: 10 times

Total Average: 98.251400 msec
Total Max:     100.159000 msec
Total Min:     96.519000 msec

  AVERAGE       MAX       MIN
------------------------------
60.469100 62.341000 59.619000: ~/.config/nvim/init.lua
 5.991100  6.219000  5.865000: loading packages
 4.088400  4.162000  3.942000: loading after plugins
 3.635700  3.752000  3.415000: loading rtp plugins
 3.561800  3.936000  3.220000: ~/.local/share/nvim/site/pack/paqs/start/onedark.nvim/colors/onedark.lua
 2.550500  2.628000  2.492000: reading ShaDa
 2.454200  2.525000  2.401000: expanding arguments
 2.363000  2.420000  2.270000: /usr/share/nvim/runtime/filetype.lua
 2.271100  2.337000  2.146000: sourcing vimrc file(s)
 1.701000  1.767000  1.635000: BufEnter autocommands
 1.676700  2.049000  1.366000: opening buffers
 1.034100  1.280000  0.782000: ~/.local/share/nvim/site/pack/paqs/start/nvim-treesitter/plugin/nvim-treesitter.lua
 0.901000  1.021000  0.799000: ~/.local/share/nvim/site/pack/paqs/start/nvim-treesitter-textobjects/plugin/nvim-treesitter-textobjects.vim
 0.590700  0.674000  0.524000: ~/.local/share/nvim/site/pack/paqs/start/indent-blankline.nvim/plugin/indent_blankline.vim
 0.549500  0.789000  0.514000: init highlight

The snippet infrastructure is now absent from that list! More importantly, the overall startup time is down, and we can be confident that the new calls to load the snippets in init.lua are not costlier than UltiSnips settings before, because the init.lua line is down as well.

As a bonus, the snippet expansion feels slightly snappier with LuaSnip, although it might be an illusion and I don’t have hard numbers to back this claim.

Migrate my Snippet Collection

Back to the topic of migrating existing UltiSnips snippets: LuaSnip will loudly complain when given UltiSnips snippets but it’s relatively easy to rewrite those snippets to the SnipMate-like format that LuaSnip understands.

Two Syntaxes

On the one hand, UltiSnips snippets roughly follow this syntax

snippet trigger "Comment" option
snippet content
endsnippet

And so, my markdown snippets would look like this:

priority 10

snippet bjtoday "“Bullet Journal”-styled date for today" b
# `date +'%F %A'`
endsnippet

On the other hand, LuaSnip uses a simplified version of SnipMate snippets:

# Comment
snippet toggle
  snippet content

Simple Snippets

Since I heavily rely on snippet sets, I have only about 30 snippets defined in my own snippet collection. Most of them are really simple, with only a few lines and almost no interactive text. So for most of those, the process has been to simply remove the priority … lines and then a simple substitution command⁴ did the trick.

Advanced Snippets With Environment Variables

However, there is also the case of the snippets calling external commands, like date in the example below:

snippet bjtoday "“Bullet Journal”-styled date for today" b
# `date +'%F %A'`
endsnippet

The problem is, to call arbitrary commands, one needs to define the snippets in Lua.

It turns out though that nearly all my snippets calling external command were actually inserting a date. And luckily LuaSnip defines “environment variables” holding just the values I need like $CURRENT_MONTH. So that snippet becomes:

# “Bullet Journal”-styled date for today
snippet bjtoday
  # ${CURRENT_YEAR}-${CURRENT_MONTH}-${CURRENT_DATE} $CURRENT_DAY_NAME_SHORT

and we get to keep the simple SnipMate-like syntax!

You can find more of those environment variables in the sources.

What’s Next

I now have a slightly faster NeoVim and the snippet syntax that I use the most is simpler. However, there is more to explore!

So far, I’ve steered clear of writing more complicated JSON and Lua snippets. The latter would be necessary to unlock smart snippets using treesitter context. I’ll look into that next, in particular to generate go error handling code.

This is also a very simple configuration, the impact on startup time of LuaSnip might go up slightly as we use more advanced feature, but during my tests, even using all available features, it was much lighter than UltiSnips.

Resources

• Many more details are covered in LuaSnip documentation.
• The SnipMate help contains the SnipMate snippet syntax, if you are unfamiliar with it.
• The LSP snippet documentation is also helpful.

⚡ TL;DR

In the Linux Git repository:

hyperfine --export-markdown /tmp/tldr.md --warmup 10 'git ls-files' 'find' 'fd --no-ignore'

git ls-files is more than 5 times faster than both fd --no-ignore and find!

Introduction

In my editor I changed my mapping to open files from fd¹ to git ls-files² and I noticed it felt faster after the change. But that’s intriguing, given fd’s goal to be very fast. Git on the other hand is primarily a source code management system (SCM), it’s main business³ is not to help you list your files! Let’s run some benchmarks to make sure.

Benchmarks

Is git ls-files actually faster than fd or is that just an illusion? In our benchmark, we will use:

• fd 8.2.1
• git 2.33.0
• find 4.8.0
• hyperfine 1.11.0

We run the benchmarks with disk-cache filled, we are not measuring the cold cache case. That’s because in your editor, you may use the commands mentioned multiple times and would benefit from cache. The results are similar for an in memory repo, which confirms cache filling.

Also, you work on those files, so they should be in cache to a degree. We also make sure to be on a quiet PC, with CPU power-saving deactivated. Furthermore, the CPU has 8 cores with hyper-threading, so fd uses 8 threads. Last but not least, unless otherwise noted, the files in the repo are only the ones committed, for instance, no build artifacts are present.

A Test Git Repository

We first need a Git repository. I’ve chosen to clone⁴ the Linux kernel repo because it is a fairly big one and a reference for Git performance measurements. This is important to ensure searches take a non-trivial amount of time: as hyperfine rightfully points out, short run times (less than 5 ms) are more difficult to accurately compare.

git clone --depth 1 --recursive ssh://git@github.com/torvalds/linux.git ~/ghq/github.com/torvalds/linux
cd ~/ghq/github.com/torvalds/linux

Choosing the commands

We want to evaluate git ls-files versus fd and find. However, getting exactly the same list of file is not a trivial task:

After some more tries, it turns out that this command gives exactly⁵ the same output as git ls-files:

fd --no-ignore --hidden --exclude .git --type file --type symlink

It is a fairly complicated command, with various criteria on the files to print and that could translate to an unfair advantage to git ls-files. Consequently, we will also use the simpler examples in the table above.

Hyperfine

Hyperfine is a great tool to compare various commands: it has a colored and markdown output, attempts to detect outliers, tunes the number of run… Here is an asciinema⁶ showing its output⁷:

First Results

For our first benchmark, on an SSD with btrfs, with commit ad347abe4a… checked out, we run:

hyperfine --export-markdown /tmp/1.md --warmup 10 'git ls-files' \
    'find' 'fd --no-ignore' 'fd --no-ignore --hidden' 'fd' \
    'fd --no-ignore --hidden --exclude .git --type file --type symlink'

This yields the following results:

As mentioned in the TL;DR, git ls-files is at least 5 times faster than its closest competitor! Let’s find out why that is.

How Does Git Store Files in a Repository

To try to understand where this performance advantage of git ls-files comes from, let’s look into how files are stored in a repository. This is a quick overview, you can find more details about Git’s storage internals in this section of the Pro Git book.

Git Objects

Git builds its own internal representation of the file system tree in the repository:

In the figure above, each tree object contains a list of folder or names and references to these (among other things). This representation is then stored by its hash in the .git folder, like so:

.git/objects
├── 65
│  └── 107a3367b67e7a50788f575f73f70a1e61c1df
├── e6
│  └── 9de29bb2d1d6434b8b29ae775ad8c2e48c5391
├── f0
│  └── f1a67ce36d6d87e09ea711c62e88b135b60411
├── info
└── pack

As a result, to list the content of a folder, it seems Git has to access the corresponding tree object, stored in a file contained in a folder with the beginning of the hash. But doing that for the currently checked out files all the time would be slow, especially for frequently used commands like git status. Fortunately, git also maintains an index for files in the current working directory.

Git Index

This index, lists (among other things) each file in the repository with file-system metadata like last modification time. More details and examples are provided here.

So, it seems that the index has everything ls-files requires. Let’s check it is used by ls-files

Strace

Let’s ensure that ls-files uses only the index, without scanning many files in the repo or the .git folder. That would explain its performance advantage, as reading a file is cheaper than traversing many folders. To this end, we’ll use strace⁸ like so:

strace -e !write git ls-files>/dev/null 2>/tmp/a

It turns out the .git/index is read:

openat(AT_FDCWD, ".git/index", O_RDONLY) = 3

And we are not reading objects in the .git folder or files in the repository. A quick check of Git’s source code confirms this. We now have an explanation for the speed git ls-files displays in our benchmarks!

Other Scenarios

However, listing file in a fully committed repository is not the most common case when you work on your code: as you make changes, a larger portion of the files are changed or added. How does git ls-files compare in these other scenarios?

With Changes

When there are changes to some files, we shouldn’t see any significant performance difference: the index is still usable directly to get the names of the files in the repository, we don’t really care about whether their content changed.

To check this, let’s change all the C files in the kernel sources (using some fish shell scripting):

for f in (fd -e c)
  echo 1 >> $f
end

git status | wc -l
28350

hyperfine --export-markdown /tmp/2.md --warmup 10 'git ls-files' 'find' 'fd --no-ignore' \
  'fd --no-ignore --hidden --exclude .git --type file --type symlink'

We see the same numbers as before and it is again consistent with ls-files source code.

Run git checkout -f @ after this to remove the changes made to the files.

With New Files and -o

With yet uncommitted files, there are two subcases:

• files were created and added (with git add): then the files are in index and reading the index is enough for ls-files, like above,
• files were created but not added: these files are not present in the index, but without the -o flag, ls-files won’t output them either, so it can still use the index, as before.

So the only case that needs further investigations is the use of -o. Since we don’t have baseline results yet for -o, let’s first see how it compares without any unadded new files.

Without any Unadded New Files (Baseline)

When we haven’t added any new files in the repository:

hyperfine --export-markdown /tmp/3.md --warmup 10 'git ls-files' 'git ls-files -o' 'find' \
  'fd --no-ignore' 'fd --no-ignore --hidden --exclude .git --type file --type symlink'

That suggests that git ls-files -o is performing some more work besides “just” reading the index. With strace, we see lines like:

strace -e !write git ls-files -o>/dev/null 2>/tmp/a
…
openat(AT_FDCWD, "Documentation/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
newfstatat(4, "", {st_mode=S_IFDIR|0755, st_size=1446, ...}, AT_EMPTY_PATH) = 0
getdents64(4, 0x55df0a6e6890 /* 99 entries */, 32768) = 3032
…

With Unadded New Files

Let’s add some files now:

for f in (seq 1 1000)
  touch $f
end

And compare with our baseline:

hyperfine --export-markdown /tmp/4.md --warmup 10 'git ls-files' 'git ls-files -o' 'find' \
  'fd --no-ignore' 'fd --no-ignore --hidden --exclude .git --type file --type symlink'

There is little to no statically significant difference to our baseline, which highlights that much of the time is spent on things relatively independent of the number of files processed. It’s also worth noting that there is relatively little speed difference between git ls-files -o and fd --no-ignore --hidden --exclude .git --type file --type symlink.

Using strace, we can establish that all commands but git ls-files were reading all files in the repository. By comparing the strace outputs of git ls-files -o and fd --no-ignore --hidden --exclude .git --type file --type symlink (the two commands that print the same file list), we can see that they make similar system calls for each file in the repository. How to explain the (small) time difference between the two? I haven’t found convincing reasons in git source code for this case. It might be that the use of the index gives ls-files a head start.

Conclusions

I’m now using git ls-files in my keyboard driven text editor instead of fd or find. It is faster, although the perceived difference described in the Introduction is probably due to spikes in latency on a cold cache. The selection of files is also narrowed down with ls-files to the ones I care about. That’s said, I’ve still kept the fd-based file listing as a fallback, as sometimes I’m not in a Git repository.

After all, Git is already building an index, why not use it to speed up your jumping from file to file!

⚡ TL;DR

I’m happy to introduce RISS (README In Static Site), a tool that transforms a README like this one and insert it on your website, like so.

A README is a Project’s Cover

Your GitHub README is what your visitors on GitHub will encounter first. Thus, it needs to efficiently describe your project. There are a lot of advice online on how to craft a great README.

So you then go on and create a great README. Wonderful! 🎉

But I want the README on my Website

But then, you want to include your README on your static website, for instance in a portfolio or just to have a GitHub-independent homepage. You also want more control on the layout and theme than GitHub has to offer, and perhaps even privacy-preserving analytics. This is a common question and there are options based on ajax or on one GitHub Pages website per repository.

However, dynamically loading the README on the client side sounds like taking a performance hit for what is actually static content. Furthermore, maintaining one full-blown GitHub Pages static website for each small project, might not be worth it.

What if you could just change your README a bit so that it is ready for inclusion in your Eleventy or Hugo personal website, just like any other markdown page?

Enter RISS!

I wrote RISS (README In Static Site) for this very purpose.

With RISS, your project’s README is the source of truth. You just add simple comments for the small bits that need to differ between GitHub and your website. You then give the GitHub README to RISS as an input and the output is a file suitable for inclusion in your static site sources. As your README change and evolves, updating it is easy, you just rerun the script.

Let’s go over some examples!

Post Metadata

Hugo (and other static-site generators) need some metadata at the start of your post, like so:

---
title: "README In Static Site (RISS)"
date: 2021-08-21T08:15:54
description: "💎 Transform and insert your GitHub README in your static site"
---

Of course, we don’t want to have this on GitHub. We thus put it in a comment, so that it remains hidden on GitHub:

<!-- insert
---
title: "README In Static Site (RISS)"
date: 2021-08-21T08:15:54
description: "💎 Transform and insert your GitHub readme in your static site"
---
end_insert -->

given the above comment as input, RISS outputs:

---
title: "README In Static Site (RISS)"
date: 2021-08-21T08:15:54
description: "💎 Transform and insert your GitHub README in your static site"
---

and the resulting file is usable by Hugo!

More

GitHub prohibits embedding arbitrary scripts for security reason. But what if you want to embed an asciinema player on your project homepage, so that users can play the asciicast as they would a standard video? You can have a placeholder with the link on GitHub as asciinema documentation advises and then use RISS to replace it with the full player on your website. You will find how to do that in the documentation. Plus, here is an example README this one doing this (and the corresponding website, look for the asciicast that autoplays).

See also how to automate the update of all these READMEs with, for instance, GitHub Action.

Thank you for reading!

Weaponizing Middleboxes for TCP Reflected Amplification

https://www.usenix.org/system/files/sec21-bock.pdf

This paper uses the fact that some middleboxes are non-compliant TCP stacks. In particular, middleboxes sometimes see only one side of a connection and as a result, it’s possible that they answer spoofed packets. After an initial training (with a genetic algorithm) on a subset of known censoring networks to optimize amplification factors, the authors present the results of applying this to the whole IPv4 internet.

Xor Filters: Faster and Smaller Than Bloom and Cuckoo Filters

https://arxiv.org/abs/1912.08258

A bit like bloom filters, but read-only, more compact and faster. The companion blog post is a good introduction: https://lemire.me/blog/2019/12/19/xor-filters-faster-and-smaller-than-bloom-filters/.

To be continued

I’ll update this page through 2021 if I want to share other papers. In the meantime, here are other posts listing papers:

• https://ordep.dev/posts/my-favorite-papers
• https://github.com/papers-we-love/papers-we-love
• https://github.com/facundoolano/software-papers

⚡ TL;DR

When Opening the DB

PRAGMA journal_mode = wal; -- different implementation of the atomicity properties
PRAGMA synchronous = normal; -- synchronise less often to the filesystem
PRAGMA foreign_keys = on; -- check foreign key reference, slightly worst performance

And check user_version to apply any migrations, for instance with this Rust library.

When Closing the DB

PRAGMA analysis_limit=400; -- make sure pragma optimize does not take too long
PRAGMA optimize; -- gather statistics to improve query optimization

Introduction

SQL pragma are statements (like SELECT … or CREATE TABLE …) that change the database behaviors or call a special functions. This post is a short list of SQLite pragma I use in my projects built on SQLite, to get better performance and more consistency.

Performance

File system Interactions

The following pragma statements set the way SQLite deals with the file system.

journal_mode wal

By default, when applying changes, SQLite uses a rollback journal, which is basically a copy of the data before the changes. Changing the journal mode to “Write-Ahead Log” is known to bring significantly better performance in most cases. It also allows concurrent readers with one writer. To activate it, run:

PRAGMA journal_mode = wal;

Some less common use cases are incompatible with this journal mode, for instance having a database on a network file system. The full list of drawbacks is listed in the documentation¹.

synchronous normal

To ensure integrity of the database, one of the mechanism SQLite uses is the file system synchronization operations. However, these synchronizations are quite costly. With WAL journal mode enabled, your database will still be consistent while synchronizing less often:

PRAGMA synchronous = normal;

Compared to the default synchronous = full, committed transactions could be rolled back if there is a power loss (although not if the application crashes).

Optimize

To execute statement as efficiently as possible, SQLite has a query planner, which tries to read the tables to provide good performance (for instance by evaluating the WHERE clauses that select the fewest rows first).

This query planner sometimes needs to know whether a column has many values or only a few, repeated values (like a boolean column would). To know this, it can’t read the whole table, as that may prove as costly as running the part of the query that is being optimized, so it uses some statistics collected in internal tables.

Using optimize right before closing the database connexion collects the statistics for some table columns. These columns are chosen mainly based on the queries executed during the connexion: if a query had benefited from more accurate statistics, the corresponding columns are analyzed.

For instance:

PRAGMA analysis_limit=400;
PRAGMA optimize;

In the above example, pragma analysis_limit ensures that optimize won’t run for too long, by limiting the number of rows read.

Allow Using More Memory

These two pragma could result in better performance, depending on your hardware and software configuration.

• Keep temporary storage in memory: PRAGMA temp_store = 2. However, setting this pragma does not guarantee that the temporary storage will be held in memory. Please refer to the documentation for other parameters that may change the final outcome.
• Keep more of the database pages in memory: for instance, use 32 MiB of memory for this purpose with PRAGMA cache_size = -32000. Note that the OS cache is already keeping parts of the database file in RAM, so this might end up wasting memory.

Consistency

user_version

The database’s schema can evolve over time. Your application could start with a car table to represent a car but later on, you realize you want to represent bicycles, so you add a bicycle table.

To keep track of the different versions of the schema, some libraries maintain an internal table with a single row with a version number. It then performs migrations as needed. In our example, version 1 would have only the car table and version 2, also the bicycle table. The migration from version 1 to version 2 add the bicycle table.

SQLite offers the user_version pragma, to keep track of these versions. It is an integer at a fixed offset in the database file. It is simpler and more efficient than maintaining a table with versions, in particular because the table has to be found in the database file while the integer is available right away.

The drawback is that this is not portable between database engines. If you only use SQLite though, it is almost always the best option. To use it, you can write some code to check the value of the user_version pragma value right after opening the database. If it is lower than expected, then atomically 1) run the necessary migrations and 2) increment the user_version pragma value. I wrote a library to ease this task in rust and here is an example in python.

foreign_keys on

This may come as a surprise for folks with experience with other database system, but SQLite does not enforce foreign key constraints by default. Consequently, a foreign key can point to a row that does not exist.

This can be fixed with pragma foreign_keys:

PRAGMA foreign_keys = ON;

This comes at a performance cost however, because more checks are performed when inserting values with foreign keys. The cost is usually negligible but your mileage may vary.

Note: pragma foreign_key_check can be used to check a particular table for violated foreign key constraint. This can be useful before enabling foreign_keys on a database with existing data.

STRICT

Another peculiarity of SQLite is dynamic typing through type affinity. When you define a column of type INTEGER in most other database engines, a value of type TEXT inserted in that column will be converted or return an error. But with SQLite, type affinities mean that if the value is not of the expected type, it will be converted if possible or stored with a different type if necessary. That’s handy when prototyping, but you may want a stricter behavior for consistency with other major SQL databases or when working with strictly typed languages.

To this end, SQLite supports the STRICT keyword at table creation. For instance, if a table t is created like so:

CREATE TABLE t(a INTEGER) STRICT;

then

INSERT INTO t VALUES(1);
INSERT INTO t VALUES('1');

are successful but

INSERT INTO t VALUES('f');

is not and returns Runtime error: cannot store TEXT value in INTEGER column t.a (19).

Note: Without the STRICT keyword at table creation, this last INSERT INTO … statement would have been successful and would have just returned a string.

Go deeper

The full list of supported pragma with detailed descriptions is available in the documentation. Some more options can be tweaked for the specific scenario of running a SQLite for a server.