Can the questions actually be estimated?
The four pilot estimands were screened against real data structures. The result is intentionally uneven: one review can proceed, one question needs redesign, and two absolute episode-risk estimates are blocked.
Alcohol episode
Outcome systems and drinking surveys do not provide a compatible exact 60 g episode numerator and denominator.
Tobacco strategy
Repeated exposure and 10-year adjudicated outcomes are split across sources; the current combined estimand is too ambitious.
Illicit opioid episode
Fatal-overdose and supply data exist, but no representative intranasal episode denominator exists.
Controlled psilocybin
A controlled-trial evidence map is feasible, although serious-event estimates will remain imprecise.
The data are allowed to say “not estimable.”
The project now has explicit go, hold, and block rules. A nearby dataset no longer counts as evidence that the exact research question can be answered.
Psilocybin
Launch a systematic evidence map for controlled 25 mg administration in MDD. Keep direct MDD/placebo evidence separate from treatment-resistant or minimal-dose-control evidence.
- Extract study-level risks and risk differences.
- Pool only if at least two studies are clinically and methodologically compatible.
- Allow “insufficient precision for rare serious events” as the final conclusion.
Tobacco
Split mortality from incident morbidity or preregister a data-fusion model. PATH is excellent for repeated tobacco behavior, but it does not by itself identify every 10-year outcome in the present estimand.
Alcohol and opioids
Do not divide national event counts by mismatched survey proxies and call the result per-episode risk. These remain denominator-methods projects.
What each source can—and cannot—contribute
| Question | Strongest source type found | Useful contribution | Release-blocking limitation |
|---|---|---|---|
| Alcohol 60 g episode | Population alcohol surveys + NHAMCS/other outcome systems | Exposure-frequency proxies and acute-event numerators | No source links exact 60 g/2-hour episodes to all 24-hour outcomes in the same population. |
| Smoking continuation vs cessation | PATH + NCHS linked outcomes | Repeated use/cessation and mortality/health outcomes in complementary sources | No single source directly observes both sustained strategies and all adjudicated 10-year outcomes. |
| Unverified opioid episode | SUDORS + DEA/laboratory data | Fatal events, circumstances, and market composition signals | No representative count of intranasal use episodes; supply samples are selection-biased and unlinked. |
| Controlled 25 mg psilocybin | Randomized controlled trials and registries | Arm-level safety counts under defined support conditions | Few direct trials, heterogeneous controls/support, and inadequate power for rare serious events. |
The protocol has now been tested against real reports.
The single-author pilot extraction exposed reporting incompatibilities; the completed review will follow PRISMA-P/PRISMA-S and use result-level RoB 2 assessment. The protocol forbids a pooled number when studies are not compatible.
MEDLINE, Embase, PsycINFO, CENTRAL, ClinicalTrials.gov, WHO ICTRP and citation searching.
Two independent reviewers with retained full-text exclusion reasons.
Dose, formulation, support, comparator, denominators, exact event definitions and windows.
RoB 2 by result, plus registry/publication discrepancies and missing evidence.
Study-level risk first; pooling only after directness and compatibility checks.
The package is ready for independent content review—not passed.
Reviewers should judge the meaning and operationalization of the four questions, not score substances. Original ratings, minority views, conflicts and author responses remain public.