Version data alongside schemas
Stage data with its schema when a commit must retain both. Data migration is a separate operation over a commit range.
Prerequisites
A panproto repository with at least one schema and a corresponding data instance.
The task
Data is staged together with its schema via schema add --data:
schema add user.json --data records/
schema commit -m "v1 schema and seed data"
# Evolve the schema and re-stage with the same data directory.
schema add user-v2.json --data records/
schema commit -m "v2 schema"
# Sync the working directory across parent..HEAD.
schema data sync records/
schema add --data <DATA> stages each immediate JSON file in the directory. Each file’s records are parsed and checked against the schema being staged, so a file that is not JSON, or one whose records do not fit the schema, fails at add rather than at whatever later operation first tries to read it. Staging is all or nothing for the data files, so a failure leaves none of them staged. --skip-verify stages without the check and leaves the result pending, which a default commit then refuses; it does not skip reading the file, since a data set records which schema its data belongs to and bytes that cannot be read as records of that schema cannot be recorded under it. schema data sync compares a target commit with its first parent, generates a lens, and rewrites records it can migrate; failed records are reported as skipped.
Preview the default parent..HEAD range and run coverage without writing:
schema data migrate records/ --dry-run --coverage
--dry-run prints the selected plan without attempting the conversions. The coverage pass then tries each immediate JSON file and reports successes and failures. Use --range old..new to select another pair.
schema status --data records/
schema data status records/
Checkout changes only the repository ref. Pass --migrate to request a corresponding on-disk data migration:
schema checkout <commit> --migrate records/
Verification
schema data status records/
prints the number of immediate JSON files, the HEAD schema ID, and the number of data sets tracked by that commit. It does not parse the files or prove conformance.
schema data migrate records/ --dry-run --coverage is the non-writing per-record check.
Common mistakes
- Editing data inside the store directly. Like schemas, data objects are content-addressed.
- Skipping data when committing schema changes. If the v1 commit has no staged data, the later commit-range operation has no stored v1 records to migrate. The v2 commit may thus have no data even when records exist in the working directory.
- Assuming rebase or amend migrates working data. Those commands take no data-directory option. Run an explicit data migration after rewriting history.