> For the complete documentation index, see [llms.txt](https://docs.nested-knowledge.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.nested-knowledge.com/ai-transparency/model-cards/critical-appraisal.md).
# Model Cards: Critical Appraisal
AI tools at this stage draft first-pass risk-of-bias and quality answers against a nest's selected appraisal instrument, for a reviewer to confirm or revise.
| Tool | What it does | Model card |
| ----------------------------------------------------- | ----------------------------------------- | ---------- |
| [Smart Critical Appraisal](#smart-critical-appraisal) | Risk of bias and study quality assessment | Live |
Smart Critical Appraisal
#### **Model Name**: Smart Critical Appraisal (SCA)
**Version: 1.0**
***
### **Overview**
**Smart Critical Appraisal (SCA)** is a large language model assisted appraisal system designed to support structured quality assessment of included studies within systematic review workflows.
Integrated directly into Nested Knowledge’s AutoLit platform, SCA helps reviewers evaluate methodological quality, risk of bias, and study reliability using established appraisal frameworks. The system reviews full-text study PDFs against checklist-based critical appraisal questions and generates **recommended responses with supporting annotations** drawn from the source text.
Rather than replacing expert review, SCA is designed to reduce the manual burden of appraisal while preserving reviewer control. AI-generated answers are presented within the structured appraisal workflow, can be reviewed and edited by humans, and never overwrite existing human decisions.
SCA is currently compatible with selected checklist systems and operates only on **included studies with uploaded full texts**, ensuring that appraisal recommendations are grounded in the full study record.
The system is intended for evidence synthesis teams seeking to accelerate critical appraisal while maintaining methodological transparency, auditability, and human oversight.
***
### **Intended Use**
#### **Primary Purpose**
To assist reviewers in applying structured critical appraisal frameworks to full-text studies by generating evidence-linked recommended responses to checklist questions.
#### **Intended Users**
* Systematic review teams
* Clinical researchers
* Medical affairs teams
* HEOR teams
* Regulatory and market access teams
* Evidence synthesis organizations conducting structured literature reviews
#### **Limitations**
* Outputs depend on the completeness and clarity of reporting in the full-text article.
* Some appraisal questions require interpretive methodological judgment that may not be fully recoverable from text alone.
* Performance varies by appraisal framework and study type.
* SCA is intended to support, not replace, expert appraisal.
***
### **Training Data**
* **Dataset**:\
SCA relies on a general-purpose large language model provided by OpenAI (GPT 5.6 Luna). The underlying model is trained on a mixture of licensed data, data created by human trainers, and publicly available text.
* **Validation Dataset**:\
Internal benchmarking datasets based on expert-reviewed studies appraised using supported critical appraisal systems.
* **Inference Inputs**: For each study, SCA evaluates:
* The full-text PDF
* Extracted article text
* The selected appraisal framework and checklist questions
* Optional configured appraisal scope, where applicable
* **Language**:\
Primarily English.
SCA does not perform fine-tuning on customer data. User documents are processed transiently for inference within the appraisal workflow.
***
### **Evaluation**
#### **Performance Metrics**
Internal validation compares SCA recommendations to expert reviewer judgments using inter-rater reliability (IRR).
| Appraisal Framework | IRR with Experts |
| ------------------- | ---------------- |
| ROB2 | 0.60 |
| JBI | 0.52 |
| SIGN | 0.30 |
Overall performance varies depending on:
* Appraisal framework
* Study reporting quality
* The degree to which checklist criteria are explicitly stated in text
* The methodological complexity of the study under review
#### **Performance Interpretation**
Agreement is strongest in frameworks where appraisal questions align more directly with explicit textual evidence. Lower agreement may occur in frameworks that rely more heavily on nuanced methodological interpretation or implicit reporting signals.
#### **Known Issues**
* Some checklist items require contextual or domain-specific judgment beyond explicit statements in the article.
* PDF extraction quality may affect the model’s ability to identify relevant supporting evidence.
* Performance may vary across domains, study designs, and reporting standards.
* Some frameworks are more readily supported by current prompting and evidence extraction workflows than others.
***
### **Ethical Considerations**
#### **Human-In-The-Loop Review**
Smart Critical Appraisal is designed as a **decision-support system** for methodological review.
The system recommends appraisal answers and highlights the passages that appear most relevant to each judgment, but final appraisal decisions remain under reviewer control. Human users may accept, revise, or clear AI-generated responses at any time.
This is particularly important because critical appraisal plays a central role in determining how much confidence reviewers place in a body of evidence. SCA is therefore built to support transparency and efficiency without removing human authority from the process.
Best practice is to review AI-generated answers alongside the annotations before finalizing appraisal records.
***
### **Limitations**
* Performance depends on the reporting quality of the full-text article.
* Poorly structured PDFs or extraction artifacts may reduce evidence quality.
* Some appraisal criteria are inherently interpretive and may not map cleanly to textual evidence alone.
* Current Smart CA compatibility is limited to NK-integrated appraisal frameworks.
* SCA only operates on included studies with full-text uploads available.
***
### **Planned Improvements**
* Improved LLM as new models are released
* Broader internal benchmarking across study designs and review types
* Enhanced reviewer tooling for auditing, revising, and validating AI-generated appraisal outputs
***
#### **Additional Notes On Compliance**
SCA securely shares necessary data with OpenAI to process requests. However, OpenAI does not store this data, and it is not used to train OpenAI's models, as outlined in our Data Processing Addendum.
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://docs.nested-knowledge.com/ai-transparency/model-cards/critical-appraisal.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.