> For the complete documentation index, see [llms.txt](https://docs.nested-knowledge.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.nested-knowledge.com/ai-transparency/model-cards/tagging.md). # Model Cards: Tagging/Extraction AI tools at this stage build a tag hierarchy and extract supporting evidence from abstracts or full texts, producing a first-pass structure and values for review rather than final, unreviewed data. | Tool | What it does | Model card | | ---------------------------------------------- | ------------------------------------------------------------------ | ---------- | | [Core Smart Tags](#core-smart-tags) | Builds and extracts standard PICO + Study Type, Location, and Size | Live | | [Adaptive Smart Tags](#adaptive-smart-tags) | Extracts evidence from abstracts and full texts | Live | | [Smart Config: Tagging](#smart-config-tagging) | Drafts a tag hierarchy from a research question | N/A | ### Core Smart Tags
Research question refinement **Model Name: Research Question Refinement** **Version: 1.0** *** #### **Overview** The **Research Question Refiner** leverages LLMs (GPT 4o) to assist users in crafting research questions that meet the recommendations for systematic literature reviews (SLRs). The tool performs the following tasks: 1. Evaluates whether a research question adheres to established SLR criteria for clarity, focus, and relevance. 2. Provides actionable feedback and follow-up questions to guide refinement. 3. Suggests modifications to improve alignment with SLR best practices. This iterative process helps users create research questions optimized for systematic review workflows. *** #### **Intended Use** * **Primary Purpose**: Facilitate the creation of high-quality research questions for systematic reviews. * **Intended Users**: Researchers, healthcare analysts, and systematic review teams. *** #### **Evaluation** * **Qualitative Nature of Performance**: * The effectiveness of the tool’s refinements is subjective and depends on the user’s research context, precision of their research question, and expertise. * Performance is assessed qualitatively through user feedback, as there are no standard metrics to quantify research question quality. *** #### **Ethical Considerations** * **Human Oversight**: The tool complements, rather than replaces, expert judgment. Users are responsible for evaluating and adopting the tool’s suggestions. * **Bias in GPT-4o**: As the underlying model is an OpenAI LLM, its feedback may reflect the biases or limitations inherent to large language models. *** #### **Limitations** 1. **Subjectivity**: The quality and relevance of refinements depend on user interpretation and research context. 2. **Explainability**: While the tool provides textual explanations, the internal decision-making of LLMs is not fully transparent. 3. **Scope**: Currently optimized for English-language research questions and general SLR guidelines. 4. **Feedback**: Feedback is based on the LLM’s’s understanding of Cochrane guidelines and general SLR principles and may not fully account for specific domain nuances. *** #### **Planned Improvements** 1. **Enhanced Guidelines**: Extend support for additional domain-specific SLR criteria. 2. **Explainability Features**: Introduce user-facing justifications for each refinement to improve transparency. 3. **User Feedback Loop**: Incorporate mechanisms to learn from user interactions and tailor suggestions more effectively. #### Additional Notes On Compliance The Research Question Refiner securely shares necessary data with OpenAI to process requests. However, OpenAI does not store this data, and it is not used to train OpenAI's models, as outlined in our Data Processing Addendum.
PICO generation **Model Name: PICO Hierarchy Generator** **Version: 1.0** *** #### **Overview** The **Hierarchy Generator** creates concept hierarchies from multiple research abstracts by extracting PICO entities and organizing them into meaningful relationships. The workflow involves: 1. Running a PICO extractor on research abstracts to identify key entities. 2. Performing a frequency analysis to determine the most relevant entities 3. Using a LLM to analyze and organize these entities into hierarchical structures, with significant pre- and post-processing for consistency and relevance. These hierarchies help users explore study characteristics, enabling better comprehension and synthesis of research evidence. *** #### **Intended Use** * **Primary Purpose**: Assist researchers in organizing study features into structured hierarchies to facilitate evidence synthesis and systematic reviews. * **Intended Users**: Researchers, healthcare analysts, and systematic review teams. *** #### **Evaluation** * The tool’s output is assessed subjectively, based on whether the generated hierarchy is well-aligned to the research question and domain or aligns with user expectations. * Effectiveness depends on the quality of input abstracts and the relevance of extracted entities. *** #### **Ethical Considerations** * **Transparency**: Hierarchies are traceable to source text via abstract annotations, though the internal reasoning of LLMs in constructing the hierarchy is opaque. * **Bias**: Outputs may reflect biases in the LLM, the PICO extraction process, or the selection of input data. * **Human Oversight**: Users must validate and adapt hierarchies for their specific research contexts. *** #### **Limitations** 1. **Subjectivity**: No formal evaluation metrics are available; users assess hierarchy quality based on perceived relevance and utility. 2. **Explainability**: While PICO extraction is traceable, the LLM step is a black box method and thus the reasoning cannot be understood. 3. **Scope**: The tool is currently optimized for English-language abstracts and PICO frameworks. Outputs may be less useful for unstructured or poorly described studies. *** #### **Planned Improvements** 1. **User Feedback Loop**: Incorporate mechanisms to learn from user preferences and improve hierarchy generation over time. 2. **Explainability Features**: Enhance post-processing to provide justifications for hierarchical relationships. 3. **Multilingual Support**: Extend support to non-English abstracts. *** #### Additional Notes On Compliance Hierarchies are generated per session, ensuring that sensitive data remains private and secure. The Hierarchy Generator tool securely shares necessary data with OpenAI to process requests. Per DPA, OpenAI does not store this data, and it is not used to train OpenAI's models.
PICO extraction **Model Name: BioELECTRA-PICO** **Version: 1.0** *** #### **Overview** The **BioELECTRA-PICO** model () is a variant of the ELECTRA architecture, specifically pre-trained and fine-tuned for extracting PICO elements (Participants, Interventions, and Outcomes) from biomedical research abstracts. This tool is used in systematic reviews and evidence-based medicine workflows to streamline information extraction and synthesis. *** #### **Intended Use** * **Primary Purpose**: Extract PICO elements from research abstracts to assist systematic reviews and meta-analyses. * **Intended Users**: Researchers, healthcare analysts, and systematic review teams. *** #### **Training Data** * **Pretraining Data**: * **Source**: PubMed (22 million abstracts) and PubMed Central (3.2 million full-text articles). * **Total Text**: \~13.8 billion words. * **Fine-tuning Data**: * Dataset: EBM-NLP corpus, 4,993 abstracts annotated with PICO elements. * **Validation Datase**t: the EBM-NLP dataset, comprising of 4,993 abstracts annotated with (P)articipants, (I)nterventions, and (O)utcomes * **Data Limitations**: * Algorithm was trained solely on English-language abstracts. *** #### Evaluation * **Performance**: F1 score of 0.74 on the EBM-NLP dataset. *** #### **Ethical Considerations** * **Bias**: The model may inherit biases from its pretraining corpora or the EBM-NLP dataset. * **Human Oversight**: Users should validate extracted elements to ensure they align with their specific use cases. *** #### **Limitations** * Optimized for English biomedical texts, potentially less effective on texts in other languages or domains. * The model's performance is subject to the quality and clarity of input abstracts. *** #### **Planned Improvements** 1. Explore fine-tuning on additional datasets for broader applicability. 2. Investigate multilingual capabilities to extend support beyond English texts. 3. Implement calibrated confidence scores for extracted elements. *** #### Additional Notes On Compliance This algorithm was trained solely on publicly available PubMed abstracts and does not store, transmit, or utilize input data beyond the prediction process.
Study type #### **Model Name**: Smart Study Type (SST) **Version: 1.0** *** #### **Overview** **Smart Study Type (SST)** is a machine-learning–based classification system designed to automatically identify and categorize biomedical study designs from article titles and abstracts. SST is part of the Core Smart Tags suite and supports systematic literature reviews (SLRs) by assigning hierarchical study type labels and providing interpretability through highlighted evidence segments. The system predicts study type across multiple hierarchical levels (e.g., Clinical vs Pre-clinical, Primary vs Secondary, Observational vs Experimental, and granular subtypes such as Randomized Controlled Trial or Meta-analysis). In addition to classification, SST produces text annotations indicating which portions of the abstract most influenced the model’s decision. *** #### **Intended Use** * **Primary Purpose**:\ To accelerate and improve systematic literature reviews by automatically classifying study design and enabling filtering based on protocol-specific inclusion/exclusion criteria. * **Intended Users**:\ Evidence synthesis professionals, clinical researchers, health economists, and systematic review teams. * **Limitations**: * Designed for biomedical abstracts and titles only. * Performance depends on the completeness and clarity of abstracts. * Not intended to replace human review; outputs should be validated by domain experts. *** #### **Training Data** * **Dataset**:\ Approximately 40,000 publicly available biomedical abstracts sourced from PubMed. Labels were generated in batch using large language models and refined through targeted sampling to balance underrepresented study types. * **Validation Dataset**:\ 300 manually curated and QA-reviewed abstracts selected to represent complex and ambiguous cases across the full taxonomy. * **Language**:\ English. A custom study taxonomy was developed rather than relying on MeSH publication types, which matched internal labels in only \~65% of cases and lacked sufficient granularity for SLR workflows. *** #### **Evaluation** * **Performance Metrics**: At Level 3 (Primary/Secondary + Observational/Experimental), the model achieved: * **Accuracy**: 0.93 * **Weighted F1-score**: 0.93 * Strong performance across clinical, experimental, observational, pre-clinical, and secondary categories. At Level 4 (fine-grained subtypes): * Overall accuracy: 0.82 * High-performing classes include: * Protocol for clinical trials (F1 = 0.98) * Case reports (F1 = 0.95) * Randomized controlled trials (F1 = 0.93) * Lower-performing classes include: * Meta-analyses (F1 = 0.55) * Case-control studies (F1 = 0.57) * Secondary analysis of clinical trials (F1 = 0.40) * **Known Issues**: * Meta-analyses are frequently confused with systematic reviews. * Case-control studies are often mistaken for cross-sectional studies. * Some distinctions (e.g., prospective vs cross-sectional) rely on temporal details that are commonly omitted from abstracts. *** #### **Ethical Considerations** * **Human-in-the-Loop Limitations**: SST is intended as a decision-support tool. Misclassifications can occur, particularly in granular categories. Users are expected to review and validate predictions, especially for inclusion-critical study types. The system includes interpretability features (text annotations) to support transparency and responsible use. *** #### **Limitations** * Abstract-only classification; full-text signals are not used. * Reduced reliability for rare or ambiguously described study designs. * Catch-all categories (e.g., “Other”) exhibit lower recall by design. * Performance decreases as classification becomes more granular. *** #### **Planned Improvements** * Expansion of labeled data for low-support classes (e.g., meta-analyses and secondary analyses). * Continued refinement of hierarchical taxonomy. * Enhanced annotation methods beyond attention-based attribution. * Ongoing calibration using real-world user feedback. *** #### Additional Notes On Compliance This algorithm was trained solely on publicly available PubMed abstracts and does not store, transmit, or utilize input data beyond the prediction process. The model operates transparently to support ethical deployment and mitigate the impact of misclassifications.
Study size #### **Model Name**: Smart Study Size **Version: 1.0** *** #### **Overview** **Smart Study Size** is a heuristic-based algorithm designed to extract participant numbers from study abstracts. It extracts participant (experimental unit) counts using a three-phase approach: 1. Checks for numbers in the text. If there aren't any then; there are no sample size values. 2. Applies regular expression (regex) matching for common phrases indicating participant numbers. 3. Selects the largest number amongst population entities, with filters to exclude non-integers or irrelevant large values. The output is a single numeric value representing the estimated participant count. *** #### **Intended Use** * **Primary Purpose**: Automatically extract participant numbers from research abstracts for meta-analyses or systematic reviews. * **Intended Users**: Researchers, healthcare analysts, and systematic review teams. *** #### **Training Data** * **Dataset**: This algorithm was not trained in the traditional sense. Instead, it was developed heuristically by analyzing approximately 300 research papers. * **Validation Dataset**: the [**PICO Corpus dataset**](https://github.com/jind11/PubMed-PICO-Detection), comprising 1,011 abstracts. * **Data Limitations**: * Heuristics were developed exclusively on English-language abstracts. *** #### **Evaluation** * **Performance Metrics**: * Accuracy: 91% on the PICO Corpus dataset. * Error Rate: 23%, with errors arising entirely from the third method (largest number), which triggers in 40% of cases. *** #### **Ethical Considerations** * **Bias**: The algorithm does not utilize training data and does not store or process data in a manner that introduces bias. However, as the heuristics were manually created, this does represent a potential source of bias. *** #### **Limitations** * Works exclusively with English-language abstracts. * Regex patterns are fixed, potentially missing less common phrasing. * The fallback method of selecting the largest number may misidentify participant counts in some cases. * Due to lack of contextual understanding; conversion of numbers in text form (e.g., "four thousand and twenty five") to digits remains imperfect. *** #### **Planned Improvements** 1. Allow for contextual understanding in the text to digit conversion. 2. Extend capabilities to non-English abstracts to increase global applicability. *** #### Additional Notes On Compliance This algorithm is **not trained on the input data**. It does not learn, store, or transmit input data, ensuring privacy and avoiding potential biases or ethical concerns related to training data.
Study location **Model Name: Smart Study Location** **Version: 1.0** *** #### **Overview** The **Smart Study Location** tool identifies the geographical location of a study based on textual information in bibliographic data, including country and city mentions in abstracts and titles, and affiliation details. It is designed to extract the study's most relevant location Key features: * Extracts mentions of geopolitical entities, cities, and nationalities in various bibliographic fields using general purpose entity recognition models * Selects the most likely country of the study using context and population statistics For non-original research or research occurring in multiple locations, the model aims to provide the most contextually appropriate result, such as author affiliations for narrative reviews. *** #### **Intended Use** * **Primary Purpose**: Automate the extraction of study location data from research abstracts to support systematic reviews, meta-analyses, and research landscape analyses. * **Intended Users**: Researchers, healthcare analysts, and systematic review teams. *** #### **Evaluation** * **Performance Metrics**: * **Accuracy**: 78% on a random sample of PubMed records with NCT ID linkage. * **Recall:** 0.79 * **Precision**: 0.90 * **Most Reliable Use Case**: Clinical study types, such as RCTs and cohorts. *** #### **Ethical Considerations** * **Human Oversight**: Users should validate extracted locations, especially in cases where the study type may influence the definition of “study location.” * **Bias**: The model may favor certain study types (e.g., RCTs) due to training data distribution and may misinterpret different types of mentions (e.g. “Japanese population in America”). *** #### **Limitations** 1. **Study Type Sensitivity**: The definition of “study location” varies across research designs, leading to potential inaccuracies for less-structured study types (e.g., reviews). 2. **Ambiguity Handling**: In cases of unclear location information, the model provides the most sensible result based on context, but this may not always align with user expectations. 3. **Scope**: Outputs are restricted to country-level data, which may oversimplify studies conducted across multiple locations. *** #### **Planned Improvements** 1. **Study Type Awareness**: Improve the model’s contextual understanding of study designs to refine its definition of “study location”. 2. **Multilingual Support**: Extend capabilities to non-English abstracts to increase global applicability. *** #### Additional Notes On Compliance This algorithm is **not trained on the input data**. It does not learn, store, or transmit input data, ensuring privacy and avoiding potential biases or ethical concerns related to training data.
### Adaptive Smart Tags
Adaptive Smart Tags #### **Model Name**: Adaptive Smart Tags (AST) **Version: 1.0** *** #### **Overview** **Adaptive Smart Tags (AST)** is a large-language-model–powered information extraction and recommendation system designed to support systematic literature reviews by identifying custom, user-defined concepts within study abstracts and/or full texts. Unlike Core Smart Tags (which address predefined fields such as PICO or study type), AST enables reviewers to configure arbitrary tagging questions via a hierarchical tag structure. For each study and tag, AST can either: * **Recommend** extracted values for human review, or * **Apply** tags automatically for immediate downstream analysis and export. AST also provides in-text annotations highlighting evidence used to support each extracted concept, enabling transparent human validation. The system is intended for enterprise-level workflows requiring scalable, customizable data extraction across large evidence corpora. *** #### **Intended Use** * **Primary Purpose**:\ To automate or accelerate extraction of custom review variables from biomedical abstracts and full texts, supporting qualitative synthesis, structured outputs, and evidence inspection. * **Intended Users**:\ Systematic review teams, clinical researchers, health economists, and enterprise evidence synthesis groups. * **Limitations**: * Outputs depend heavily on the clarity and completeness of source documents. * Performance varies by tag formulation and domain. * Not designed to replace expert judgment; human confirmation is strongly recommended, especially for inclusion-critical variables. *** #### **Training Data** * **Dataset**:\ AST relies on a general-purpose large language model provided by OpenAI: GPT 5.6 Luna. The underlying model is trained on a mixture of licensed data, data created by human trainers, and publicly available text. * **Validation Dataset**:\ Internal benchmarking datasets derived from expert-tagged abstracts and full texts across multiple nests and tagging configurations (Form-Based Tagging and Standard Tagging). * **Language**:\ Primarily English. AST does not perform fine-tuning on customer data. User content is processed transiently for inference only. *** #### **Evaluation** * **Performance Metrics**\ Internal validation using expert tagging shows the following representative performance: | Dataset | Recall | Precision | Annotation Rate | | ------------------------------ | ------ | --------- | --------------- | | Abstract – Form-Based Tagging | 0.85 | 0.69 | 98% | | Full Text – Form-Based Tagging | 0.83 | 0.67 | 97% | Overall expected performance: * **Recall**: \~80–85% * **Precision**: \~67–70% Results vary substantially depending on: * Question design * Tag hierarchy structure * Document type (abstract vs full text) * Subject matter * **Known Issues**: * Precision may decrease when tag questions are underspecified. * Over-application of tags can occur when hierarchies lack sufficient contextual constraints. * Certain datasets were only partially QAed, meaning reported precision likely underestimates true model capability. * Full-text extraction is significantly slower than abstract-only processing. *** #### **Ethical Considerations** * **Human-in-the-Loop Limitations**: AST supports both recommendation-based workflows (with expert review) and fully automated extraction. While auto-apply enables rapid outputs, it increases risk of silent errors. Best practice is to use **Recommend mode** for critical variables and to validate extracted concepts using the provided annotations. *** #### **Limitations** * Performance depends on how well tag questions are written. * Ambiguous or implicit concepts in source text may be missed or misinterpreted. * Precision is lower for loosely defined tags. * Generation limits apply (per nest) for abstracts and full texts. * Full-text processing is slower and more resource-intensive. *** #### **Planned Improvements** * Improved guidance and tooling for tag question optimization. * Expanded internal QA datasets, especially for full-text workflows. * Enhanced evidence attribution and summarization options. * Continued performance tuning for precision on complex or abstract concepts. *** #### Additional Notes on Compliance Adaptive Smart Tags processes user content only for prediction and extraction. Data is not retained beyond the generation workflow, guaranteed via subprocessor DPA.
### Smart Config: Tagging
Smart Config: Tagging Please see [Smart Config: Screening](/ai-transparency/model-cards/screening.md#smart-config-screening) for model card details. No tagging-specific evaluation statistics are offered at this time.
--- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.nested-knowledge.com/ai-transparency/model-cards/tagging.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.