Data readiness assessment: checklist, scoring and how to run one
Contents
A data readiness assessment answers one question: does the data behind a specific use case exist, can it be used legally and technically, and is it good enough? Readiness is a property of the pairing between data and use case, so the same warehouse can be ready for a dashboard and unfit for a RAG assistant.
The useful output is a scored matrix, a remediation backlog with effort, and a go, fix-first or stop decision.
What is a data readiness assessment?
A data readiness assessment checks whether the data a specific use case needs exists, can be accessed legally and technically, and is good enough to support it. It ends in a go, fix-first or stop decision, with a remediation backlog listing each gap and its effort.
We recommend scoping to one or two named use cases, because data profiling only means something against a defined requirement. It is the data pillar of a broader AI readiness assessment, and its thresholds shift with the risk of the use case.
Why data readiness depends on the use case
Data readiness depends on the use case because the same dataset can clear every check for one model and fail several for another: each dimension is scored against what the named use case consumes.
Take one CRM export: 120,000 accounts, 24 months of history, structured fields plus a free-text support-notes column. Every figure below is an illustrative assumption, not a benchmark.
The same export, scored for two use cases:
| Dimension | Churn prediction (tabular) | Support RAG assistant |
|---|---|---|
| Volume and class balance | 4,800 churners (4%): pass, handled with class weights and a time-based split | Not applicable |
| Edge-case coverage | Pass: all plan tiers and regions present | Fail: notes cover 3 of 12 product areas |
| Completeness | Pass: 97% of structured fields populated | Fail: notes empty on 40% of tickets |
| Access permissions | Pass: one role reads the whole table | Fail: account-level ACLs are lost at retrieval |
| PII | Pass: identifiers pseudonymized | Fail: names and emails sit in the notes |
| Metadata and data catalog | Pass: catalog entries exist for tenure, plan and MRR | Fix first: no catalog entry for note author or origin |
| Timeliness | Pass: monthly snapshot is enough | Fix first: notes predate two product changes |
The churn model reads only structured fields, so its main risk is class imbalance, and 4% positives is workable. The RAG assistant reads the notes column, which the churn model ignores, so the export fails on edge-case coverage, permissions and PII.
A dimension that passes for one use case can fail for another on identical source data. A readiness score for "the CRM" therefore has no meaning until a use case is named. For the retrieval-specific checks, see RAG chatbot accuracy and data preparation.
How to run a data readiness assessment, step by step
- Scope the use case. Name one or two, the decision each supports, and the fields each consumes.
- Inventory sources. Record owner, system, access path, and refresh cadence per source. Read lineage from OpenLineage events where they exist. Otherwise trace by hand from the consuming table back to the system of record.
- Profile samples. Run data profiling with ydata-profiling or Great Expectations on a random 5,000-row sample per table, stratified by month if history matters.
- Score each dimension against your rubric thresholds, using the use case's fields only.
- Size the remediation backlog. Each finding becomes an item with an owner, an effort estimate in days, and a flag for whether it blocks a go. Cleaning methods live in the data preparation guide. Pipeline fixes usually need data engineering capacity.
- Decide. Write the go, fix-first, or stop call, with the blocking items listed, for the use-case sponsor.
A 5,000-row sample estimates a rate such as a null share to within roughly 1 to 2 percentage points at 95% confidence.
A sample cannot size rare classes: a class at 0.5% yields roughly 25 rows. It also underestimates duplicates, because a pair only counts when both rows land in the sample. Check duplicates and referential integrity with key-based SQL or dbt tests on the full table.
A source with no named owner cannot score above fix-first for production use. Record "owner: unassigned" as a finding, name the team that administers the system as interim contact, and ask the sponsor to assign an owner within the assessment window. A prototype can proceed on such a source, but the go decision cannot. Ownership as a dimension is covered in the data governance guide.
Four roles cover most assessments:
- Assessment lead: runs the steps and owns the matrix.
- Data engineer: profiling and lineage.
- Domain expert: field definitions and label review.
- Security or legal contact: legal basis and access.
Data readiness checklist by dimension (copyable template)
A data readiness checklist works when each checklist item, grouped by dimension, ends in a number or a pass/fail result and names the evidence a reviewer can open. Score every row against the named use case, not the whole company.
The seven dimensions below are the copyable core. Thresholds are adaptable defaults, so set them from the cost of a wrong prediction in your use case.
| Dimension | Item | How measured | Evidence |
|---|---|---|---|
| Availability and access | Every field the use case consumes has a named owner and a working read path | Fields with owner and tested access ÷ fields required; read test run with the production service account | Source inventory, access test log |
| Quality | Completeness, duplicates, consistency, timeliness (ISO/IEC 25012 characteristics) | Null rate per required field; duplicate rate on the business key; dbt tests (not_null, unique, relationships, accepted_values) or Great Expectations suites on a sample; schema drift events in the last 90 days | ydata-profiling report, test run history |
| Volume and representativeness | Enough rows per class and coverage of known edge cases | Rows per class and minority-class share; edge-case list with a count per case; train/test leakage check: no shared entity IDs or duplicate rows across splits, no feature computed after the label date | Class distribution table, leakage script output |
| Labels and ground truth | Labels are consistent between annotators | Cohen's kappa on a double-labeled sample (a few hundred items is a workable default); share of labels that reviewers overturn | Annotation guideline, kappa calculation |
| Metadata, lineage, documentation | Required tables are findable and traceable | Share of required tables in the data catalog with description and owner; share of feeding pipelines that emit OpenLineage events; one Datasheets for Datasets record per dataset (Gebru et al., 2021) | Catalog export, lineage graph, completed datasheets |
| Governance, privacy, legal basis | The intended use is permitted and access is enforced | Legal basis recorded for this purpose; share of fields with a PII classification tag; test that a restricted user cannot retrieve restricted records | Processing record, access review, ACL test result |
| Pipelines and infrastructure | Data arrives fresh and can be rebuilt | Lag between source update and availability vs. the use case's freshness requirement; rebuild a training snapshot from raw twice and compare row counts and checksums | Pipeline run history, snapshot hashes |
Two rows change by use case. The leakage and kappa checks apply to supervised learning on tabular data. For RAG over documents, replace them with document coverage against a list of questions users ask, duplicate and superseded versions in the index, and permissions that survive chunking into retrieval.
A row you cannot test belongs in the remediation backlog as its own item, with an effort estimate. An unmeasurable row with no backlog entry is the most common gap in checklists that stop at questions.
How do you measure data quality?
Measure data quality by mapping each ISO/IEC 25012 characteristic to one computed metric and one automated test, run on the data your use case needs. ISO/IEC 25012 names the characteristics, and the ISO/IEC 5259 series (Part 2 covers quality measures) defines how to measure them for analytics and ML.
Start with a ydata-profiling report on a representative sample to see missing values, duplicate rows and distributions. Then encode the rules you care about as Great Expectations expectations or dbt tests, so the same checks run on every refresh.
The mapping we use, side by side:
| Metric | ISO/IEC 25012 characteristic | What to compute | Tool |
|---|---|---|---|
| Completeness | Completeness | Share of non-null values in fields the use case reads, per column and per segment | ydata-profiling missing-values section; Great Expectations expect_column_values_to_not_be_null with mostly; dbt not_null |
| Accuracy | Accuracy | Share of sampled records that match a verified reference; format and range validity | Great Expectations range and regex expectations; manual audit of a sample against source documents |
| Consistency | Consistency | Referential integrity and agreement on the same entity across systems | dbt relationships and accepted_values; Great Expectations cross-table checks |
| Duplicates | Consistency (our filing; 25012 has no separate uniqueness characteristic) | Share of duplicate keys and near-duplicate rows | dbt unique; Great Expectations expect_compound_columns_to_be_unique; ydata-profiling duplicate rows |
| Timeliness | Currentness | Age of the newest record against the refresh the use case requires | dbt source freshness with warn_after and error_after; Great Expectations max-timestamp expectation |
Compute every metric per segment as well as overall. In an illustrative case, a feature that is 98% complete across the table can be 60% complete for the minority class the model exists to detect.
Accuracy is the only metric on the list that needs a human or a trusted reference, so budget reviewer time for it. Set pass thresholds from the cost of a wrong prediction: a null rate that a gradient-boosted model tolerates can break a regulatory report.
The ISO/IEC 25012 model targets structured data. Document corpora need different checks, covered in the generative AI and RAG section.
Are your labels good enough?
Labels are good enough when independent annotators agree on each class you need to predict, at a Cohen's kappa threshold set by what a wrong label costs. Raw agreement alone will mislead you on imbalanced data.
The failure mode is class imbalance. Assume two annotators double-label 1,000 support tickets, and 5% should be escalations. Each flags 50, but only 25 overlap. Raw agreement is 95%, yet kappa is about 0.47 and positive-class F1 between the annotators is 0.50. The model's target class is labeled by coin flip.
Design the audit sample around that risk:
- Stratify by class. A random draw of 400 tickets holds roughly 20 escalations, too few to estimate anything.
- Double-label at least 50-100 items per class, blind, then adjudicate disagreements and record the rules that resolve them.
- Report per-class agreement (one-vs-rest kappa or inter-annotator F1) next to overall kappa. Use Fleiss' kappa for three or more annotators.
- Check provenance. Labels inferred from proxies, such as an agent's closing tag, need a separate accuracy check against a gold subset.
Threshold defaults depend on the use case, so adapt them. For ticket routing, we would treat per-class kappa of 0.6 or higher as workable, 0.4-0.6 as fix-first, and below 0.4 as a stop until the guidelines are rewritten. For medical, compliance, or other high-stakes labels, set the ready bar at 0.8 or higher.
Write the guidelines and agreement results into a datasheet, following Gebru et al., 'Datasheets for Datasets' (Communications of the ACM, 2021), so the next team inherits them.
Governance, privacy and legal basis for the intended use
Two governance gaps stop a data initiative before build: no named owner who can approve the use, and no legal basis for the intended purpose. Every other gap on this dimension belongs in the remediation backlog.
Owners and legal bases are decisions that people outside the project make, and engineering time cannot buy them. Missing documentation can be written during the build.
A model trained on data with no lawful basis may have to be retrained, which is why the gate sits before build.
We recommend applying the following split per source, for each named use case:
| Gap | Verdict | Why |
|---|---|---|
| No accountable owner who can approve this use | Stop | Nobody can grant access or accept the risk |
| No GDPR lawful basis for the purpose, or a purpose incompatible with the original collection purpose | Stop | Needs a legal decision, not a pipeline fix |
| Special-category data with no Article 9 condition | Stop | Same, with a higher penalty tier |
| Licence or contract does not cover training or retrieval use | Stop | Rights sit with a third party |
| Source missing from the data catalog (no owner, purpose or retention fields) | Backlog | Documentation work, sized in days |
| No datasheet or recorded collection origin | Backlog | Datasheets for Datasets gives the template |
| Stale access review, or DPIA not started | Backlog, with a date before launch | Must close before go-live, not before build |
For high-risk systems, Article 10 of the EU AI Act requires documented data governance: collection origin, the original purpose of personal data, preparation steps such as annotation and cleaning, and bias examination. Each backlog row above produces evidence Article 10 asks for.
The NIST AI RMF 1.0 Map function records intended purpose and legal context, which is where owner and basis come from. The Measure function tests quality and representativeness, which feeds the quality dimensions of this assessment.
Article 10 binds only high-risk systems (our EU AI Act guide covers the classification), so treat it as a template elsewhere. Whether a lawful basis exists is a call for your DPO or counsel. The assessment only checks that someone has made it. For the program behind these checks, see our data governance guide.
Data readiness assessment for RAG vs classic ML
Data readiness checks for RAG differ from classic ML: tabular projects test class imbalance, leakage, and labels, while RAG projects test document coverage, versioning, and retrieval-time access control. The same dimensions apply, but the failing check changes with the data type.
The side-by-side comparison:
| Dimension | Classic ML on tabular data | RAG on unstructured documents |
|---|---|---|
| Representativeness | Class imbalance per target, edge-case coverage by segment | Document coverage: share of expected questions whose answer exists in the corpus |
| Duplicates | Duplicate rows that cause train/test leakage | Near-duplicate files and superseded versions competing for top-k slots |
| Ground truth | Inter-annotator agreement (Cohen's kappa) on labels | Golden question set with reference answers and source passages, reviewed by two domain experts |
| Access control | Column and row permissions enforced in the warehouse | Retrieval-time access control: source ACLs carried into the index |
| Sensitive data | PII in known columns, found by profiling | PII in free text, found by entity detection before chunking |
| Tooling | Great Expectations, ydata-profiling, dbt tests | OpenLineage events for ingestion, document metadata audits |
Thresholds are use-case dependent. As adaptable defaults, we suggest a kappa of 0.6 or higher before training on labels and a golden set that spans every business unit the assistant will serve.
RAG failure modes to test before a pilot
Pilots most often fail when source permissions never reach the vector index. SharePoint, Confluence, and Google Drive enforce per-document ACLs, but an ingestion job typically runs under a service account that can read everything. Unless each chunk carries its ACL as metadata and retrieval filters on the caller's identity, a junior employee can surface board-level documents.
The test is simple: query as a low-privilege user for content you know they cannot open. Any hit is a stop-level finding, because fixing it can mean re-ingesting the entire corpus.
Two further checks belong in the same assessment:
- Versioning: if v1 and v3 of a policy are both indexed with no effective-date metadata, the retriever will quote either. Check that each document has a version, an owner, and a supersession link.
- Chunk-level PII: profile a sample of chunks with an entity detector, since chunking can separate a name from the context that made it safe to retain.
Remediation for these gaps goes into the backlog with effort estimates. Accuracy tuning after that point is covered in RAG chatbot accuracy and data preparation.
How to score readiness: Rubric with go, fix-first and stop thresholds
A data readiness rubric scores each dimension as ready, fix first or stop for one named use case, and a short decision rule turns the scores into an outcome. The same table can pass a dashboard and fail a classifier.
The thresholds below are adaptable defaults. Dimension names follow the characteristics in ISO/IEC 25012, and the starred rows depend heavily on the use case.
| Dimension | Check | Ready (2) | Fix first (1) | Stop (0) |
|---|---|---|---|---|
| Access and legal basis (gate) | Service-account access; documented basis for the intended use | Both in place | Access pending, basis drafted | No basis for the intended use |
| Completeness* | Non-null share of fields the model needs | 95% or more | 80-95% | Under 80% |
| Duplicates | Exact and near-duplicate share of records or documents | Under 1% | 1-5% | Over 5% |
| Timeliness* | Data age against the use case's freshness need | Within it | Within 2x | Beyond 2x |
| Representativeness* | Smallest class or segment, edge-case coverage | Enough examples per class to split and test | Gaps fillable by collection | Target class essentially absent |
| Label quality (gate, supervised) | Per-class Cohen's kappa between annotators | 0.60 or more (0.80 for high-stakes labels) | 0.40-0.60 | Under 0.40 |
| Lineage and reproducibility | Source-to-feature lineage; rerun gives same output | Both | Partial | Neither |
Score each dimension 2, 1 or 0. We recommend this decision rule:
- Go: no zeros, and no more than two dimensions at 1.
- Fix first: no gate at 0, and every 1 can be remediated within the project budget.
- Stop: any gate at 0, or remediation that costs more than the use case returns.
Two red flags force a stop whatever the total: no legal basis for the intended use, and no ground truth for a supervised target. Both are decisions or acquisitions, not engineering tasks.
Every fix-first score becomes one row in the remediation backlog:
- The failing check and its measured value.
- An owner.
- An effort size, for example S (up to 3 days), M (1-2 weeks) or L (3 weeks or more), as illustrative defaults.
- Whether it blocks the go decision.
- The test that proves it fixed, such as a Great Expectations expectation or a dbt test.
Sort blockers first. Cleaning steps for each row are covered in our data preparation guide, and teams without capacity for L items can scope them with data engineering support.
Worked example: Scoring a support-ticket RAG assistant
A support-ticket RAG assistant scores fix-first on the rubric, because retrieval-time access control and document versioning fall short while the rest of the data is usable. All figures below are illustrative, invented for this example.
Assume an assistant that answers agent questions from 4,000 resolved tickets and 600 knowledge-base articles. The assessment samples 300 tickets and 100 articles.
| Dimension | Illustrative measurement | Score |
|---|---|---|
| Availability and access | 95% of articles exportable through the helpdesk API | Ready |
| Coverage | 82% of the 40 most frequent ticket categories have a current article | Ready |
| Duplicates and versioning | 14% near-duplicate articles, no effective-date field | Fix first |
| Access permissions | Source ACLs exist on 85% of documents but the retriever ignores them | Fix first |
| PII in text | 3% of sampled tickets contain emails or phone numbers | Fix first |
Three dimensions score 1, more than the go rule allows, so the use case is fix first, not go. Nothing scores stop, because every gap has a known remediation path.
The remediation backlog, with effort ranges that are also illustrative:
- Retrieval-time access control: copy source ACLs into chunk metadata and filter on them at query time. 2-3 weeks.
- Deduplication and versioning: cluster near-duplicates, add effective-date and supersedes fields, and re-index. 1-2 weeks.
- PII redaction and ingestion checks: mask contact details in tickets, then add a Great Expectations suite that fails the load on null ACL fields, pattern-matched PII, or stale source exports. 1-2 weeks.
Items 2 and 3 can run in parallel, so the illustrative total is 4-6 weeks before a re-score. The mechanics of the first two items are covered in our RAG data preparation guide.
Retrieval-time access control sits first because it is the only gap that can leak data to the wrong audience on day one. The other two degrade answer quality, which a pilot can tolerate.
Cost, timeline and whether to self-assess or hire help
A use-case-scoped data readiness assessment costs more and takes longer as source count, label audit size, and legal review depth grow. Three drivers explain most of the variation between a quick internal check and a multi-month engagement.
- Source count: Each source needs an access request, a profiling sample, and a lineage check. An existing data catalog shortens the inventory step. Without one, the team builds the inventory through interviews and schema reads.
- Label audit: Computing Cohen's kappa needs two independent annotators on a sample. Cost follows annotator expertise, so clinical or legal labels cost far more than ticket categories.
- Legal review: Personal data in free text, cross-border transfers, or a likely high-risk classification under the EU AI Act bring in counsel and a data protection officer. In our judgement, this usually sets the calendar, because profiling runs in days once credentials exist and legal queues do not.
The criteria for choosing between doing it yourself and hiring help:
| Factor | Self-assess when | Bring in outside help when |
|---|---|---|
| Sources | One or two, owned by your team | Many, spread across business units or vendors |
| Labels | Your staff can audit them | They need domain experts or an independent reviewer |
| Legal exposure | No personal data, low-risk use | Personal data in text, regulated or high-risk use |
| Decision stakes | Internal pilot | The go / fix-first / stop call gates a large budget |
Self-assessment fails when nobody on the team has written profiling checks before. Outside help adds independence and speed, not a different method. Either way, require the same output: a remediation backlog with effort estimates that feeds your AI budget.
FAQ
What does data readiness mean?
What is the difference between data readiness and AI readiness?
How to conduct a readiness assessment?
What should a data readiness checklist include?
How much does a data readiness assessment cost?
How to choose a data readiness assessment partner?
Turn your assessment into a go or fix-first decision
The verdict applies to one use case only: a fix-first result for ticket classification says nothing about a RAG assistant. Order the remediation backlog by blocking impact and effort, route cleanup items to the data preparation guide, and feed the data findings into your AI readiness assessment before budgeting.
When the verdict is go, the next step is scoping the build around that use case, and our AI consulting team can help with either step. If you want a second pair of eyes on your scored matrix, send us your use case and we will tell you what we would check first.
