> For the complete documentation index, see [llms.txt](https://docs.nested-knowledge.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.nested-knowledge.com/ai-transparency/model-cards/literature-search.md).
# Model Cards: Literature Search
AI tools at this stage turn a research question into a workable search strategy or bring existing citations into a nest, cutting down manual query-building and re-keying references.
| Tool | What it does | Model card |
| ----------------------------- | -------------------------------------------------------------------------------------- | ---------- |
| [Smart Search](#smart-search) | Turns a research question into ready-to-edit search queries across supported databases | Live |
| [Bibliomine](#bibliomine) | Imports every citation from an uploaded PDF into a nest | Live |
Smart Search
#### **Model Name**: Smart Search
**Version: 1.1**
***
#### **Overview**
The **Smart Search Tool** leverages an agent-based large language model (LLM) system to generate boolean queries tailored to specific research questions and databases. The LLM employed is OpenAI GPT 5.6 Luna. It employs a **generator-critic loop**:
* **Generator**: Proposes a boolean search and retrieves results.
* **Critic**: Evaluates the query results based on their relevancy and proximity to a desired review size (e.g., 1000, 3000, or 5000) and provides feedback to refine the query.
This iterative process continues until the search meets predefined size and relevance criteria. Searches may be moved to Search Exploration (for manual curation and interpretability of query contents) or directly imported.
***
#### **Intended Use**
* **Primary Purpose**: Automate the generation and refinement of database searches to assist researchers in systematic review workflows.
* **Intended Users**: Researchers, healthcare analysts, and systematic review teams.
***
#### **Training Data**
* **Dataset**: The tool is not trained in the traditional sense but uses LLMs as the underlying agent for query generation and evaluation.
* **Validation Dataset**: The performance was validated on two data sets:
* **Nested Knowledge Dataset (NK)**: 20 reviews conducted by Nested Knowledge and its affiliates with corresponding research questions and included studies.
* **Cochrane Review Dataset (CR)**: Research questions and included studies from 10 randomly selected Cochrane reviews.
* **Data Limitations**:
* Heuristics were developed exclusively on English-language abstracts.
***
#### **Evaluation**
Two validation datasets were generated from research questions and included studies: internally curated reviews (NK) and Cochrane reviews. Recall and precision on the included records (at the full text screening level) were measured as:
* **NK Recall**: 0.87
* **NK Precision**: 0.02
* **Cochrane Recall**: 0.72
* **CR Precision**: 0.0050
Note that the Cochrane dataset includes searches over non-PubMed databases, so reported recall is a floor on expected performance in reviews that would incoporate other databases on top of Smart Search + PubMed.
***
#### **Ethical Considerations**
* **Human Oversight**: The tool is intended as a supplement, not a replacement, for expert-led search strategy development. Users are expected to refine and verify the generated queries.
* **Bias and Limitations of LLMs**: As the LLM is a general-purpose model, its biases or inaccuracies may propagate into the search strategy. These risks are mitigated by the critic module, which evaluates results numerically rather than semantically.
* **Automation Bias**: Users may over-rely on the tool without critically evaluating its outputs.
***
#### **Limitations**
1. **Precision**: The low precision highlights a propensity for over-inclusive search strategies. Although this low precision is typically seen in human derived search strategies as well; low precision is preferred to low recall and results in increased screening time instead of lost evidence
2. **Scope**: The system is currently limited to databases for which Nested Knowledge offers integrations and is optimized for English-language queries.
3. **Input**: Requires a clear input research question that scopes to relevant studies, in advance of screening.
4. **Language:** Restricted to English-language abstract databases
***
#### **Planned Improvements**
1. **Enhanced Precision**: Incorporate domain-specific heuristics or secondary validation models to improve the specificity of query results.
2. **Extended Databases**: Expand functionality to other databases as Nested Knowledge adds integrations
3. **Human Interpretability**: Upon building Advanced Search, which discretizes searches into individual lines with descriptions, Smart Search may be integrated
***
#### Additional Notes On Compliance
The Smart Search Tool securely shares necessary data with OpenAI to process requests. However, OpenAI does not store this data, and it is not used to train OpenAI's models, as outlined in our Data Processing Addendum.
Bibliomine
#### **Feature Name**: Bibliomine
**Version: 1.1**
***
#### **Overview**
The **Bibliomine** feature extracts cited references from an uploaded PDF for import into a nest & progression to screening. This is useful for updating an existing review (where prior inclusions are cited) or identifying related studies (e.g. from an introduction or discussion section) to increase the comprehensiveness of the evidence at hand, beyond database search.
Bibliomining offers two modes of operation:
* **Standard:** References are extracted from the study using CERMINE
* **Smart:** References are extracted from the study using text extraction (pdfminer) and an LLM (GPT-5.6 Luna)
Once extracted, references are identified using key identifiers (DOI, title) on external literature databases (PubMed, OpenAlex, CrossRef), and enriched bibliographic data are imported alongside the original references into import staging. A reviewer curates and approves the staged references for import to the review.
***
#### **Intended Use**
* **Primary Purpose**: Extend the coverage of evidence in a review and speed up manual processes of reference extraction and matching in external databases via augmentation
* **Intended Users**: Researchers, librarians, and systematic review teams.
***
#### **Training Data**
* **Dataset**: The tool is not trained in the traditional sense (instead, it was manually developed and tuned) but the constituent Cermine (publicly documented training methods) and GPT (proprietary) models were built on variably documented datasets.
* **Validation Dataset**: The performance was validated using a random sample of 100 PubMed Central open access articles.
* **Data Limitations**:
* Open Access-only: paywalled articles may structure their references differently than the validation set
* Small size: Journal articles show a diversity of citation formats, layout, and formatting choices. 100 articles do not capture the entirety of this diversity.
***
#### **Evaluation**
Ascertaining a match using edit-distance based similarity on titles, with metrics scored using the pooled set of cited studies within the 100 articles:
* **Standard:**
* **Recall: .**73
* **Precision:** .78
* **Smart:**
* **Recall:** .83
* **Precision:** .85
***
#### **Ethical Considerations**
* **Human Oversight**: Users are intended to validate and curate the results of bibliomining; the user interface makes the removal & addition of references possible.
* **Data Quality**: Only records hosted on external databases may be imported with high quality/complete bibliographic information (e.g. abstracts), since references themselves contain minimal information.
* **Automation Bias**: Users may over-rely on the tool without critically evaluating its outputs.
***
#### **Limitations**
1. **Input Sensitivity**: An article with unusual or poor formatting (including citation format, layout, or even the whole PDF being an image scan) could result in
2. **External Databases**: If a record is not found on an external database, its bibliographic data will be of low quality/completeness, which may hamper its inclusion in a review.
***
#### Additional Notes On Compliance
When running Smart mode, Bibliomining securely shares necessary data with OpenAI to process requests. However, OpenAI does not store this data, and it is not used to train OpenAI's models, as outlined in our Data Processing Addendum.
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.
Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:
```
GET https://docs.nested-knowledge.com/ai-transparency/model-cards/literature-search.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.