There is no clean line in the sand when it comes to FDA’s acceptance of real-world evidence (RWE). Regulators have spent the past decade building a framework that encourages fit-for-purpose registries, but their confidence is not won by sophisticated statistical methods alone. The most ambitious RWE submissions still collapse under basic data-integrity failures. If your RWE registry failed FDA scrutiny, the root cause is rarely the algorithm. More often, it is the silent architecture behind the data. This article walks through three common pitfalls — unverifiable source data, missing-data denial, and version-control blindness — and shows precisely how to fix them before you submit, using a composite case study drawn from real regulatory setbacks.
Pitfall #1: Source Data That Cannot Be Verified
Imagine spending months preparing a registry analysis only to receive an information request from FDA asking: “How do you know the data in your electronic case report forms (eCRFs) match the underlying patient records?” This is not a hypothetical. In one recent pre-submission meeting, a sponsor presented a prospective registry of patients with a rare autoimmune condition. The registry captured disease activity scores, laboratory results, and medication changes from multiple sites. The analysis was elegant — but the source data verification trail was not.
The registry pulled directly from each site’s electronic medical record (EMR) via automated extraction tools. However, the mapping logic between the EMR native codes and the registry variables was never documented at the measurement level. When FDA asked for line-level source records for 50 random patients, the sponsor could not provide a clean pathway from the recorded value back to the original clinical note or laboratory system. The submission was withdrawn before review even began.
The fix is to treat source data verification (SDV) as a pre-submission deliverable, not a site-monitoring chore. Whether you use a risk-based SDV model or a full SDV approach, the key is documenting the provenance of every variable. The following steps can prevent this failure:
- Create a detailed source data matrix that maps each registry variable to the original source system and field, including code translations and unit conversions.
- Perform an independent verification exercise on a representative sample of patients before locking the dataset, and retain both the query and resolution logs.
- Maintain a versioned appendices package showing any changes to the source data definitions or extraction scripts.
- Ensure that site-level EMR access is available for remote or in-person verification during the FDA review window.
Without a verifiable chain of custody, your RWE registry is just a collection of numbers with no civilian identity. FDA needs to see the same data you saw, and if you cannot show them the path, they have no reason to trust the conclusion.
Pitfall #2: Missing Data Treated as an Afterthought
Missing data is a stubborn companion in almost every real-world registry, but its effect becomes lethal when it is addressed only at the analysis stage. The second case in our review involves an oncology registry built to evaluate progression-free survival (PFS) from longitudinal follow-up. The registry had strong enrollment and accurate baseline data, but by month twelve, nearly 40% of patients lacked a documented progression assessment. The sponsor used a complete-case analysis, dropping all patients without a recorded event or a confirmed disease-free visit. This dramatically shortened the estimated PFS curve and produced results that contradicted the clinical trial literature.
FDA’s exchange with this sponsor was not about statistical software; it was about the assumptions behind the missingness. The failure lay in the registry’s operational design: no protocol had been written to minimize dropout, no tracking of the reason for missing visits, and no pre-specified method for handling intermittent missingness. When the sponsor presented a multiple imputation model during their late-cycle meeting, FDA questioned whether the imputation could compensate for data whose absence was directly related to disease severity. That fundamental concern could have been solved with a tiny bit of forethought.
To fix this pitfall, missing data must be addressed in the study design phase, not in the analysis plan appendix. Consider adopting these practices:
- Define key variables and the minimal acceptable level of completeness for each endpoint in the statistical analysis plan (SAP).
- Pre-specify the missingness mechanism, including which covariates are likely to drive dropout, and justify your chosen imputation method based on that mechanism.
- Collect the primary reason for missing data at the visit level — patient refusal, adverse event, disease progression, or administrative failure.
- Run at least two sensitivity analyses under different assumptions (e.g., informative missingness and worst-case scenario) to show that your results are robust.
Real-world registries are not clinic trials, but that does not mean missing data is an acceptable norm. FDA expects evidence that you understand why you are missing data and that your conclusions are not an artifact of what your registry did not see.
Pitfall #3: Frozen Databases and Version-Control Blindness
The third and perhaps most avoidable pitfall is a product of database governance. One medical device registry we observed in regulatory consultation had a robust data pipeline, a strong statistical analysis, and all the right endpoints. But the sponsor’s team was still asked to answer a series of questions about the exact contents of the dataset at the moment the analysis was run. Did you include the 14 patients added after the initial extraction? Did you update the adverse event coding after the final data cut? What was the exact date the database was locked?
Those questions triggered the third participant’s plunge into regulatory limbo. The sponsor had relied on a research database backup from one day, but then a site re-open query changed a subset of lab values and a physician added a missing follow-up visit. After those edits, a new snapshot was created, or so the team thought. In reality, they had generated three different versions of the database in a matter of weeks, and nobody could articulate which one was authoritative for the submission.
FDA does not require a single static database for an RWE submission, but they do require a precise, reliable audit trail that allows the agency to follow every change. A frozen dataset is a misnomer if it is not strictly controlled. The practical remedy involves a professional-level data governance process:
- Create a dataset freeze procedure that prevents any edits after the analysis start date, unless approved through a formal change control process.
- Use a data versioning system that assigns a unique identifier to every analysis-ready dataset, including each run of data management, cleaning, and derivations.
- Maintain an audit log with timestamps, user identification, before-and-after values, and a description of the reason for each change.
- Archive all data repositories, logs, and software scripts used to generate the final analysis in a manner that can be independently reproduced by FDA reviewers.
- Before submission, run a reproducibility check by asking an independent analyst to recreate the primary results from the exact final dataset version.
When reviewers see a clean versioning trail, they can spend their time evaluating your science rather than questioning your data governance. If you cannot establish a single source of truth, your registry analysis is not ready for prime time.
A Pre-Submission Playbook for Data-Integrity Readiness
These three pitfalls are not isolated events; they often appear together in the same submission. The solution is to build a pre-submission playbook that puts data integrity at the center of your regulatory pathway, not as a compliance add-on. Begin with a mock submission readiness audit that simulates the FDA review experience. Have an external team query your registry for source data verification on a sample of patients, ask you to explain missingness assumptions, and demand an inventory of database versions.
You should also create a “reviewer packet” in parallel to your statistical analysis. That packet should contain the source data matrix, a missing data roadmap, and a complete data dictionary with version history. Make it easy for FDA to say yes. The less effort they need to untangle your data, the faster they can focus on clinical meaningfulness and external validity.
Technology is not the answer to every data-integrity issue, but modern RWE platforms can help with automated audit logs and traceable data transformations. At minimum, every registry should have an immutable metadata store that records field-level lineage. While you are at it, do not forget to train your investigators on the importance of documentation at the site level. A good analysis cannot rescue a registry that was not designed for verification.
Conclusion
Your RWE registry did not fail FDA scrutiny because you lacked enough patients or because your confidence intervals were too wide. It failed because the foundation of evidence — trustworthy data with clear provenance, a justifiable approach to missingness, and a controlled path to the final analysis — was not in place. The good news is that all three pitfalls are preventable. Source data verification, deliberate missing-data planning, and disciplined version control may seem like unglamorous tasks, but they are the difference between a submission that invites dialogue and one that invites rejection. Make pre-submission data integrity the central act of your RWE story, not the last item you remember.
