> For the complete documentation index, see [llms.txt](https://docs.nested-knowledge.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.nested-knowledge.com/review-workflow/screening/criteria-based-screening/smart-screener/smart-screener-evaluation.md). # Smart Screener Evaluation Learn how to use Smart Screener Evaluation. **Smart Screener Evaluation** shows how closely **Smart Screener**'s criteria answers match your team's on records you've already screened. It reports **Recall**, **Precision**, and **Accuracy** for each criterion, and lets you review every disagreement side by side. Use it for three things: 1. Debug your criteria: a criterion **Smart Screener** often gets wrong is usually a criterion whose wording is ambiguous 2. Decide whether and how to use **Smart Screener**: whether to use it at all, as a second reviewer in Dual Screening, or to answer criteria across your whole nest 3. Report its reliability in your methods, with figures from your own review rather than a general benchmark Run it after your team has answered criteria on a sample of records, and *before* you run **Smart Screener** across your nest. If you need to evaluate include/exclude decisions rather than criteria, compare offline instead (see [Limitations](#limitations)). {% hint style="info" %} Evaluation compares **criterion answers** only. It doesn't evaluate screening decisions (include, exclude, or advance). {% endhint %} ### Screen A Sample Evaluation needs answers your team has finalized to compare against. Answer every criterion on at least 10 records at the stage you want to evaluate; 50 or more gives more reliable figures. If a criterion is rarely answered Yes (for example, one record in 10), screen more records, so the sample contains enough Yes answers to measure **Recall** and **Precision** meaningfully. Answer every criterion on each sample record, even after one is answered No. A criterion you skip on a record doesn't count toward that criterion's figures, so stopping at the first No shrinks the sample for every later criterion. Where your answers come from depends on your screening mode. In Single Screening, your recorded answers are final. Only criterion answers a human records count as labels. If you run **Smart Screener** in Single Screening, its criterion answers stay AI answers, even after thresholding creates screening decisions from them. Evaluation doesn't compare **Smart Screener** with itself, so these records aren't labeled.if s In Dual Screening, only adjudicated answers count; reviewer-level answers awaiting adjudication don't. Screen your sample from the screening queue rather than **Study Inspector**, so records arrive in random order and each gets no more than two reviewers. ### Run The Evaluation {% stepper %} {% step %} ### Open Evaluate From **Configure Screening** **,** select **Evaluate.**

{% endstep %} {% step %} ### Select Abstract or Full Text Choose the stage to evaluate. If a criterion has fewer than 10 final answers at that stage, Nested Knowledge tells you instead of showing results. {% endstep %} {% step %} ### Generate The Evaluation Set If screened records don’t have a **Smart Screener** answer yet, a message tells you how many and offers **Generate Evaluation Set**. This runs **Smart Screener** on those records as a separate, reviewer-level set used *only for comparison.* It uses your nest’s screening generation budget. Full Text evaluation uses High Fidelity Mode. In Dual Screening with **Smart Screener** as a reviewer, the evaluation set is **Smart Screener**’s existing reviewer-level answers. Evaluation doesn’t generate new answers for those records. It doesn't change your final answers, your screening decisions, your queues, or your PRISMA diagram. Regenerate the set whenever you edit or delete a criterion, or after you label more records. You don’t need to regenerate after you change a final answer, because the figures update immediately.

{% endstep %} {% step %} ### Review The Results Table Each criterion shows **Evaluation Coverage**, **Recall**, **Precision**, and **Accuracy**, each with the numerator and denominator behind it. An **Overall** row summarizes all criteria.

{% endstep %} {% step %} ### Inspect Disagreements Select the magnifier on a criterion to compare your team's answer and **Smart Screener**'s answer record by record. Disagreements are listed first.

{% endstep %} {% step %} ### Open The Evaluation Set in Study Inspector Select **Evaluation Set** to open **Study Inspector** in a new window, filtered to the records used. {% endstep %} {% endstepper %} ### Read The Results Each figure treats a **Yes** answer as the positive result and your team's final answer as correct. Evaluation counts **NR (Not Reported)** as **No**, for your team’s answers and for **Smart Screener**’s answers. **Evaluation Coverage** is the number of labeled records **Smart Screener** has answered, out of all labeled records. * **Recall** is how often **Smart Screener** answers Yes when your team answered Yes: `Recall = records both answered Yes ÷ records your team answered Yes` Higher recall means fewer relevant records missed. * **Precision** is how often **Smart Screener**'s Yes answers match your team's: `Precision = records both answered Yes ÷ records Smart Screener answered Yes` Lower precision means more irrelevant records answered Yes, which costs reviewer time later. * **Accuracy** is how often the two answers match at all: `Accuracy = records where both answers match ÷ records compared` In most reviews, recall matters most: a missed Yes on an inclusion criterion can exclude a relevant record. Read accuracy alongside the other two, because it can look high when answers are imbalanced. If your team answers No on nine of 10 records, a criterion **Smart Screener** always answers No scores 90% accuracy and 0% recall. #### Tune And Iterate Use the disagreement view to find the pattern behind low figures. If **Smart Screener** keeps misreading the same concept, rewrite that criterion's question to be more specific, regenerate the evaluation set, and compare. When you're satisfied, run [**Smart Screener**](/review-workflow/screening/criteria-based-screening/smart-screener.md) across your nest. If you revise a criterion several times against the same sample, its figures can end up better than its performance on the rest of your records. Before you report figures, add newly labeled records and check them again. #### Report Smart Screener's Performance In your methods, report the stage evaluated, the number of labeled records, and **Recall**, **Precision**, and **Accuracy** for each criterion with their numerators and denominators. Note which version of your criteria the figures apply to; if you change a criterion afterward, the earlier figures no longer describe it. #### Limitations **Smart Screener Evaluation** evaluates criteria answers only. Evaluation of screening decisions isn't available yet. You can't import answers from outside Nested Knowledge to evaluate against. To compare **Smart Screener** with include/exclude decisions from a completed review (for example, for a pilot), import the records, run **Smart Screener**, export its results, and compare them with your existing decisions offline. ### FAQs

Do I have to answer criteria on 10 records before I can run Smart Screener?

No. The minimum applies only to **Smart Screener Evaluation**. You can run **Smart Screener** at any time. To evaluate it first, label a sample before running it across your nest; **Generate Evaluation Set** runs **Smart Screener** on the sample for you.

Can I evaluate Smart Screener in both Single and Dual Screening?

Yes. In Single Screening, your recorded answers are compared. In Dual Screening, only adjudicated answers are compared, so finish adjudicating your sample first.

Why don't my own calculations match the results table?

Check these differences first: * Evaluation compares criterion answers, not screening decisions. * Each criterion is calculated only on records where your team answered that criterion and **Smart Screener** has an answer, so each criterion can have a different denominator. * In Dual Screening, Evaluation uses adjudicated answers, not either reviewer's answers. * Evaluation uses the evaluation set’s **Smart Screener** answers. These can differ from a separate **Smart Screener** run, unless **Smart Screener** is a reviewer in Dual Screening. * The evaluation set may be out of date if you’ve edited criteria since you last generated it. If you’ve labeled more records, check **Evaluation Coverage**. * Evaluation counts NR as No. If you counted NR as Yes, or left it out, your figures will differ. To see exactly which records went into a figure, open **Evaluation Set** in **Study Inspector**, or select the magnifier on that criterion.

Can I evaluate include/exclude decisions from a review I've already completed?

Not yet. Evaluation needs criterion answers, and you can't import them. Compare offline instead; see [Limitations.](#limitations)

Does running Smart Screener Evaluation affect my nest's real screening decisions?

No. The evaluation set is a separate, reviewer-level set. It doesn't change any final answer, screening decision, queue, or your PRISMA diagram.

Can I use Smart Screener's answers to speed up labeling?

Yes, but use care. Run **Smart Screener**, then go through the records and confirm or correct each criterion answer. A criterion becomes a human label when you select Yes or No, even if you select the same answer **Smart Screener** gave. Seeing **Smart Screener**'s answer first can influence your own answer. This can make the evaluation figures higher than its real performance. For figures you will report, label records without **Smart Screener** answers visible.

### Next Steps


Smart Screener	Answer your screening criteria across your records.	Smart Screener
Criteria-Based Screening	Set up screening criteria for your review.	Criteria-Based Screening
Screen Records	Review records against your configured criteria.	Screen Records
Adjudicate Records	Resolve disagreements in Dual Screening.	Adjudicate Records

--- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.nested-knowledge.com/review-workflow/screening/criteria-based-screening/smart-screener/smart-screener-evaluation.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.