Compare two versions of a dataset. Get the bump they deserve, and the changelog entry that explains it.
A dropped column, a phone number that turned into a string, a distribution that quietly shifted. All of them break the people downstream. All of them usually ship as "updated the dataset".
DataSemver makes that impact explicit and reviewable, so releasing a dataset can be discussed the way releasing a library is.
Every difference is matched against a rule, and every rule carries a severity. The strongest severity found becomes the bump, which is why the order below is the whole argument rather than a way to group three cards.
Something consumers already depend on is gone, or means something else now.
There is more than there was, and nothing that was there has moved.
The dataset says what it said before, with fewer gaps or less noise.
Severity belongs to your rules file, not to the change itself. The bundled defaults are one opinion; strict and lenient profiles ship beside them, and the rule catalogue lists every rule, metric and threshold. Anything no rule covers is reported as unclassified and never inflates the result.
This export separates its fields with semicolons while every amount also contains a comma. The comma never reaches the header, so the semicolon wins and the file loads as five columns instead of one.
Comma, semicolon, tab and pipe are recognised. Set
DATASEMVER_CSV_DELIMITER to skip the detection entirely.
One command whatever the dataset arrived as, because the format is rarely the thing being compared. A stored profile counts as a dataset on either side, so a comparison can run with the previous version long gone.
CSV and TSV, gzipped or not, with the delimiter detected. JSON as an array of objects, and JSON Lines.
.csv .tsv .csv.gz .tsv.gz .json .jsonl .ndjson
Parquet and Feather state the type of every column, so nothing is inferred: a postcode stays a string instead of becoming an integer with the leading zero gone.
.parquet .pq .feather .arrow
A sheet after #, or the first one when none is named. Needs the excel extra.
quarterly.xlsx#Q3
A connection URL with the table after #. The password never reaches the report, the logs or a file name.
sqlite:///snapshots.db#customers
The dashboard is a client of the library rather than a fork of it. It calls the same function and renders the same report, so the two agree by construction.
Upload two files, or pick two versions from a directory it watches.
A mean is one point of a distribution, and a column can be rebuilt around it without moving it. Three changes that break every consumer downstream, and that a comparison of schemas and averages reports as nothing at all:
The spread grows fortyfold and the mean does not move. Measured with a Kolmogorov-Smirnov statistic over a stored quantile grid.
std 0.998 → 39.96
A label goes from even to one-in-a-hundred with both values still present, so the category set is unchanged. Measured with the Population Stability Index.
'ok' 50.3% → 99.0%
A date column is compared on when it actually sits, so an export that slid forward or now covers half the period is a change rather than a silence.
2020-01-01 → 2026-01-01
Both are weighed against what the sample can support. A KS statistic has no fixed
reading: on four rows against five, appending a single row moves the distribution by a
fifth, so a shift has to clear the critical value for those sample sizes as well as the
configured threshold. On a handful of rows nothing is reported, because there is nothing
to report. And with a --key, rows are matched rather than summarised: a
version where a third of them were rewritten with values drawn the same way has the same
profile as the one before it, and is a different dataset to anyone joining against it.
The default engine loads the dataset into a dataframe, which is the fastest thing to do
until the dataset stops fitting. --engine duckdb aggregates over the file
instead, and duckdb-sketch estimates the quantile grid rather than computing
it — the one number that costs a full pass.
| Engine | Parquet, 380 MB | CSV, 1.21 GB | Answers |
|---|---|---|---|
| pandas | 21.4 s · 3190 MB | 49.9 s · 3311 MB | the reference |
| duckdb | 16.1 s · 1736 MB | 20.9 s · 2856 MB | identical, to float rounding |
| duckdb-sketch | 3.1 s · 1246 MB | 7.9 s · 1952 MB | quantile grid estimated |
Measured on 16 million rows and seven columns, at peak resident memory of the process.
The sketch is a named engine rather than an automatic shortcut, because it stops
interpolating between values on small inputs: on eight rows it answers 1 where the other
path answers 1.35, so it diverges most exactly where the result is easiest to check by
hand. Set DATASEMVER_DUCKDB_MEMORY_LIMIT and the ceiling is yours; what does
not fit spills to disk instead of failing.
# half the memory, same answers datasemver diff old.parquet new.parquet --engine duckdb # a third again, with the quantile grid estimated datasemver profile huge.csv --engine duckdb-sketch
A comparison reads a profile: the columns, their types, their null ratios, a quantile grid and the counts per category. It is small enough to commit beside the dataset, and then the previous version never has to be fetched — or exist.
# write it once, next to the data datasemver profile customers_v3.parquet # compare against it later, with the file long gone datasemver diff customers_v3.profile.json customers_v4.parquet # from the Parquet footer: a seek instead of a scan datasemver profile customers_v3.parquet --schema-only
A Parquet footer already holds the schema and, per column, the nulls and the smallest and largest value — every input a schema-level comparison reads. What it cannot give is the quantile grid and the category counts, so a profile written from it records that it came from there, and any comparison against it says no distribution was checked. Without that, the report would read exactly like one where nothing moved.
Built on typer and rich. Add --json when a script needs to read the answer instead of a person.
datasemver diff old.csv new.csv
A connection URL with the table after #. SQLite needs no driver; PostgreSQL and MySQL use the sql extra.
sqlite:///snapshots.db#customers
Two paths, or two profiles you already hold in memory when the data came from a warehouse query.
from datasemver import analyze
A local tool with no authentication and no rate limiting. Keep it on the loopback interface.
pip install "datasemver[web]"
Analyses the datasets a pull request touches and posts the suggested bump, rewriting the same comment on each push. Set a threshold and it refuses the merge — after commenting, so the refusal arrives with its reason.
uses: IzanVil/datasemver@v0.8.2
Compares the datasets DVC reports as changed between two revisions, continuing each from the version recorded beside it rather than restarting at zero.
datasemver dvc --rev HEAD~1
One executable per platform that brings its own Python, for machines where installing one is not an option. Database and workbook support are built in.
linux · windows · macos arm64 & x86_64
# read the bump from a release script BUMP=$(datasemver diff old.csv new.csv --json | jq -r '.bump') # prepend the changelog entry to a file you keep datasemver diff old.csv new.csv --current-version "$(cat VERSION)" --output CHANGELOG.md # refuse a breaking change: 0 ran clean, 1 was refused, 2 could not run datasemver diff old.csv new.csv --fail-on major # which rows changed, not only whether the shape did datasemver diff old.csv new.csv --key id # record the version it computed, instead of retyping it datasemver diff old.csv new.csv --current-version 1.4.2 --write-version
Python 3.10 or newer, typed, no runtime network access. Releases are published from CI
through Trusted Publishing with signed provenance, so no long-lived token exists. Extras
are opt-in: sql for databases, excel for workbooks,
duckdb for the engines, web for the dashboard.
No Python on the machine? Every release also carries a single executable per platform that brings its own, with database and workbook support built in — on the latest release.