> For the complete documentation index, see [llms.txt](https://docs.nested-knowledge.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.nested-knowledge.com/ai-transparency/model-cards/screening.md).
# Model Cards: Screening
AI tools at this stage answer or draft screening criteria, fill a reviewer seat in dual screening, and highlight relevant terms in abstracts. None finalizes an inclusion or exclusion decision without a human reviewer.
| Tool | What it does | Model card |
| ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | ---------- |
| [Screening Model/Robot Screener](#screening-model-robot-screener) | Predicts likelihood of inclusion in a review; Generates reviewer decisions in dual screening | Live |
| [Smart Screener](#smart-screener) | Answers screening criteria; generates reviewer decisions in dual screening | Live |
| [Smart Config](#smart-config-screening) | Drafts screening criteria from a research question | Live |
Screening Model/Robot Screener
#### **Model Name**: Screening Model
**Version: 1.0**
***
## Overview
The **Screening Model** is a supervised machine learning system designed to assist systematic review teams in prioritizing and screening literature records based on their likelihood of inclusion in a review.
The model learns from screening decisions made within a specific nest, identifying patterns associated with included, excluded, or advanced records. It then generates inclusion or advancement probabilities for unscreened records. The model itself is a gradient boosted decision tree, leveraging cross validation–based hyper-parameter tuning and SMOTE oversampling to address class imbalance. Training begins once a bare minimum of 50 screened records and 10 included or advanced records are generated in the review.
Predicted probabilities from the Screening Model can be used to:
* Assist manual screening workflows by prioritizing records for review based on predicted relevance, or
* Power the **Robot Screener**, an automated reviewer that can act as a second reviewer in dual screening workflows. It excludes and includes records using a cross-validation determined threshold on inclusion probabilities
The system is designed to accelerate the screening stage of evidence synthesis while maintaining a human-in-the-loop workflow and maximizing includable evidence recall.
***
## Intended Use
#### **Primary Purpose**
To support systematic review workflows by predicting the probability that a study should be included or advanced during screening, enabling reviewers to prioritize high-relevance records and reduce manual screening workload.
#### **Intended Users**
* Systematic review teams
* Clinical researchers
* Evidence synthesis professionals
* Health technology assessment groups
* Enterprise research organizations conducting literature reviews
#### **Limitations**
* Model performance depends on the quantity and quality of prior screening decisions in the nest.
* Early in a review, limited training data may reduce model reliability.
* Screening decisions rely on human-defined inclusion criteria, which may be complex or difficult for the model to learn.
* The model should not replace expert review for final screening decisions unless used within controlled workflows (e.g., Robot Screener in dual-review mode).
***
## Training Data
#### **Dataset**
The Screening Model is trained dynamically using **screening decisions within a specific nest**, including:
* Included records
* Excluded records
* Advanced records (in two-pass screening)
Training data is generated from reviewer decisions made during the screening process.
#### **Input Features**
The model uses multiple features derived from bibliographic metadata and textual information, including:
* Bibliographic metadata
* Publication age
* Page count
* Keywords and descriptors
* Abstract content
* Text n-grams
* Text embeddings (OpenAI embedding model)
* Citation metrics from Scite
* Number of citing publications
* Supporting and contrasting citation statements
Missing features are imputed using distributions derived from other records within the nest.
#### **Language**
Primarily English.
***
## Evaluation
#### **Performance Metrics**
Internal testing across several hundred systematic review projects produced the following representative performance metrics.
**Standard Screening**
| Metric | Value |
| ----------------------- | ----- |
| AUC | 0.88 |
| Classification Accuracy | 0.92 |
| Recall | 0.76 |
| Precision | 0.40 |
| F1 Score | 0.51 |
**Two-Pass Screening**
| Metric | Value |
| ----------------------- | ----- |
| AUC | 0.88 |
| Classification Accuracy | 0.93 |
| Recall | 0.81 |
| Precision | 0.44 |
| F1 Score | 0.56 |
#### **Performance Characteristics**
* **High recall by design**, minimizing the risk of excluding relevant studies.
* Precision is lower due to class imbalance and the model’s preference for conservative exclusion.
* Accuracy is typically high due to the large proportion of excluded records in screening datasets.
#### **Known Issues**
* Performance varies based on the size of the screened dataset used for training.
* Class imbalance (many more exclusions than inclusions) can reduce precision.
* Some records may lack sufficient metadata or abstract text for reliable prediction.
* Predictions early in the screening process may be unstable due to limited training examples.
***
## Ethical Considerations
#### **Human-In-The-Loop Limitations**
The Screening Model is intended to **augment human screening workflows**, not replace expert judgment.
When used solely to generate inclusion probabilities, the model provides **decision support** that reviewers may use to prioritize records.
When used as **Robot Screener**, the model may act as a second reviewer in dual-review workflows. While this can significantly accelerate screening, it increases reliance on automated predictions.
Best practice is to:
* Monitor model performance metrics
* Validate screening outcomes
* Use human adjudication where disagreements occur
***
## Limitations
* Model accuracy depends on the number of screened records available for training.
* Early-stage reviews may not provide sufficient training data for strong performance.
* Abstract-only information may not fully capture eligibility criteria and a full text screening phase remains unsupported by the Screening Model.
* Missing metadata may reduce prediction reliability.
* Precision is intentionally lower than recall, meaning the model may suggest inclusion of some irrelevant records.
***
## Additional Notes On Compliance
The Screening Model trains and operates on user-provided records within a nest and does not retain content beyond the review workflow. Nested Knowledge will never use the training data or the model itself for any purpose beyond assisting screening within the nest it was trained in.
Smart Screener
### Overview
Smart Screener is an LLM-based automation of the screening process of a literature review. Operating at both the Abstract and Full Text screening levels, it answers preconfigured screening criteria questions that support inclusion or exclusion in the review. To convert Smart Screener's criteria answers into screening decisions:
* In single reviewer modes: the user may leverage either thresholded screening, thresholding on the number of criteria satisfied, or they may review records one by one, assessing each record using the prefilled criteria answers.
* In dual reviewer modes: In combination with a second (human) reviewer level screening decision, an administrator may leverage auto-adjudication or review records one by one to generate final screening decisions.
Smart Screener is driven by GPT 5.6 Luna. It leverages pdfminer for PDF text extraction and Datalab in high fidelity mode.
### Intended Use
* **Primary Purpose**: Automate or accelerate the screening process in systematic reviews.
* **Intended Users**: Researchers, healthcare analysts, and systematic review teams.
### Training Data
As an unsupervised model backed by GPT 5.6, no conventional training data was leveraged by Nested Knowledge in the development of Smart Screener.
Its development (used in tuning the approach) & evaluation datasets were internally developed at Nested Knowledge.
### Evaluation
Criteria-level Recall & Precision statistics are given against an internally developed set of 10 reviews across diverse topics. To generate meaningful binary decision data, any NR responses were bucketed to Yes (AB) and No (FT) based on the screening level.
* Abstract
* Recall: .94
* Precision: .88
* Full Text
* Recall: .95
* Precision: .96
High recall statistics indicate that Smart Screener will exclude relevant evidence from reviews infrequently.
### Ethical Considerations
* Smart Screener Evaluation is made available to reviewers for the assessment and transparent sharing and publication of Smart Screener performance in their reviews.
* By default, Smart Screener does not generate screening decisions on top of criteria answers. Correspondingly, human-in-the-loop review workflows & quality assurance are reinforced.
* Smart Screener offers non-english language support
### Limitations
* Smart Screener can only imply screening decisions on the basis of configured criteria. If configured criteria have quality issues or gaps, performance will suffer.
* In full text screening, Smart Screener does not generate criteria using Supplemental Materials.
### Planned Improvements
* Nested Knowledge development consistently monitors and tests for core LLM improvement
* Multi-annotation support for improved interpretability
### Additional Notes On Compliance
The Smart Screener securely shares necessary data with OpenAI to process requests. However, OpenAI does not store this data, and it is not used to train OpenAI's models, as outlined in our Data Processing Addendum.
Smart Config: Screening
### Overview
Smart Config: Screening is an LLM-based automation of the screening configuration process of a literature review. Operating at both the Abstract and Full Text screening levels, it configures criteria that effectively identify and discriminate to studies that will address the research question at hand. Smart Screener is driven by GPT 5.4.
Smart Config is intended to solve the "blank page problem", demonstrating how to establish screening criteria for a Nested Knowledge review and indicating key concepts to screen on. Reviewers are expected to find enhanced screening performance by curating Smart Config's results, and the configuration interface is designed to accept reviewer edits.
### Intended Use
* **Primary Purpose**: Assist reviewers in setting up the automated (Smart Screener) or manual screening process in their review
* **Intended Users**: Researchers, healthcare analysts, and systematic review teams.
### Training Data
As an unsupervised model backed by GPT 5.4, no conventional training data was leveraged by Nested Knowledge in the development of Smart Config.
Its development (used in tuning the approach) & evaluation datasets were internally developed at Nested Knowledge.
### Evaluation
Internally developed benchmarks leverage datasets of:
* a research question
* search results (examples to be screened)
* Human screening decisions (labels)
The combined dataset contains 3 internally developed reviews of clinical interventions in different therapeutic areas. Evaluation proceeds by generating screening criteria for the research question, then running Smart Screener on the criteria. Performance is only recorded on the abstract screening level.
* Precision: .54
* Recall: .86
As a fully automated workflow of a critical review function, low performance is to be expected. Note that recall is relatively high, indicating the preference of Smart Config and Smart Screener to advance records to full text screening & avoid the exclusion of relevant evidence.
### Ethical Considerations
* Smart Config makes informed assumptions on which types of information are conventionally reported in abstracts or only in the full text report. Authors, articles, and journals not conforming to these publishing norms may be incorrectly excluded from a review.
### Limitations
* A typical research question does not contain full specification of review purpose and intent. Reviewers are strongly encouraged to review criteria prior to actioning them, to avoid systemtically generating gaps in the included evidence in reviews.
### Planned Improvements
* Nested Knowledge development consistently monitors and tests for core LLM improvement
* Users will be given the ability to specify inputs to Smart Config through "Instructions", going beyond limited information in the Research Question.
### Additional Notes On Compliance
The Smart Config securely shares necessary data with OpenAI to process requests. However, OpenAI does not store this data, and it is not used to train OpenAI's models, as outlined in our Data Processing Addendum.
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://docs.nested-knowledge.com/ai-transparency/model-cards/screening.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.