Compare two versions of a dataset. Get the bump they deserve, and the changelog entry that explains it.
A dropped column, a phone number that turned into a string, a distribution that quietly shifted. All of them break the people downstream. All of them usually ship as "updated the dataset".
DataSemver makes that impact explicit and reviewable, so releasing a dataset can be discussed the way releasing a library is.
Every difference is matched against a rule, and every rule carries a severity. The strongest severity found becomes the bump, which is why the order below is the whole argument rather than a way to group three cards.
Something consumers already depend on is gone, or means something else now.
There is more than there was, and nothing that was there has moved.
The dataset says what it said before, with fewer gaps or less noise.
Severity belongs to your rules file, not to the change itself. The bundled defaults are one opinion; strict and lenient profiles ship beside them, and the rule catalogue lists every rule, metric and threshold. Anything no rule covers is reported as unclassified and never inflates the result.
This export separates its fields with semicolons while every amount also contains a comma. The comma never reaches the header, so the semicolon wins and the file loads as five columns instead of one.
Comma, semicolon, tab and pipe are recognised. Set
DATASEMVER_CSV_DELIMITER to skip the detection entirely.
The dashboard is a client of the library rather than a fork of it. It calls the same function and renders the same report, so the two agree by construction.
Upload two files, or pick two versions from a directory it watches.
A mean is one point of a distribution, and a column can be rebuilt around it without moving it. Three changes that break every consumer downstream, and that a comparison of schemas and averages reports as nothing at all:
The spread grows fortyfold and the mean does not move. Measured with a Kolmogorov-Smirnov statistic over a stored quantile grid.
std 0.998 → 39.96
A label goes from even to one-in-a-hundred with both values still present, so the category set is unchanged. Measured with the Population Stability Index.
'ok' 50.3% → 99.0%
A date column is compared on when it actually sits, so an export that slid forward or now covers half the period is a change rather than a silence.
2020-01-01 → 2026-01-01
Both are weighed against what the sample can support. A KS statistic has no fixed
reading: on four rows against five, appending a single row moves the distribution by a
fifth, so a shift has to clear the critical value for those sample sizes as well as the
configured threshold. On a handful of rows nothing is reported, because there is nothing
to report. And with a --key, rows are matched rather than summarised: a
version where a third of them were rewritten with values drawn the same way has the same
profile as the one before it, and is a different dataset to anyone joining against it.
A comparison reads a profile: the columns, their types, their null ratios, a quantile grid and the counts per category. It is small enough to commit beside the dataset, and then the previous version never has to be fetched — or exist.
# write it once, next to the data datasemver profile customers_v3.parquet # compare against it later, with the file long gone datasemver diff customers_v3.profile.json customers_v4.parquet
Built on typer and rich. Add --json when a script needs to read the answer instead of a person.
datasemver diff old.csv new.csv
A connection URL with the table after #. SQLite needs no driver; PostgreSQL and MySQL use the sql extra.
sqlite:///snapshots.db#customers
Two paths, or two profiles you already hold in memory when the data came from a warehouse query.
from datasemver import analyze
A local tool with no authentication and no rate limiting. Keep it on the loopback interface.
pip install "datasemver[web]"
Analyses the datasets a pull request touches and posts the suggested bump, rewriting the same comment on each push. Set a threshold and it refuses the merge — after commenting, so the refusal arrives with its reason.
scripts/run_datasemver_on_pr.py
# read the bump from a release script BUMP=$(datasemver diff old.csv new.csv --json | jq -r '.bump') # prepend the changelog entry to a file you keep datasemver diff old.csv new.csv --current-version "$(cat VERSION)" --output CHANGELOG.md # refuse a breaking change: 0 ran clean, 1 was refused, 2 could not run datasemver diff old.csv new.csv --fail-on major # which rows changed, not only whether the shape did datasemver diff old.csv new.csv --key id
Python 3.10 or newer, typed, no runtime network access. Releases are published from CI through Trusted Publishing with signed provenance, so no long-lived token exists.