Colonoscopy video annotation involves labeling objects, findings and events across video footage so AI models can learn to detect and track them. When the goal is to train a model to detect polyps, however, marking a suspected polyp in a single frame is only part of the task. Annotators also need clear rules for defining its boundaries, handling partial visibility and keeping the same polyp consistently labeled as it appears across consecutive frames.
These details matter because the model learns from the labels in its training data. If annotators apply different rules to the same type of finding, the resulting dataset may teach the model to miss polyps, detect irrelevant structures or lose track of a polyp as the camera moves.
How these labels are created also depends on what the model is expected to do. A detection model may need bounding boxes around polyps, while a segmentation model requires more precise, pixel-level masks. If the goal is to track polyps throughout a procedure, the annotation guidelines must also explain how to maintain object identity from one frame to the next.
In this guide, we look at how AI teams can define polyp labels, choose the right annotation format and maintain consistent quality across colonoscopy video datasets.
What counts as a polyp label in colonoscopy video annotation?
A polyp label is an annotation that identifies a polyp according to the project’s agreed criteria. Depending on the task, it may record the object’s location, visible boundaries, class and identity across frames.
The annotation guideline should define what annotators are expected to label and what they should do when the visual evidence is unclear. Annotators should not independently turn an uncertain visual finding into a clinical diagnosis.
Before annotation begins, the team should agree on 5 elements:
Target class: Which objects count as polyps for this task, and which structures or artefacts should not receive that label?
Annotation format: Does the model require a bounding box, polygon, segmentation mask or another representation?
Visibility rules: How should annotators label objects that are partly outside the frame, obscured or blurred?
Frame-level rules: Should the polyp be labeled in every frame where it is visible, or only in selected frames?
Object identity: If the same polyp disappears briefly and reappears, should it retain its original ID?
These decisions should be documented before production. Otherwise, two annotators may interpret the same frame differently while both believe they are following the instructions.
It is also important to distinguish visual annotation from clinical characterization. A dataset intended for polyp detection does not automatically need a diagnosis, histology or risk classification for every visible object. Those labels should be included only when the task requires them and the relevant clinical ground truth is available.
For medical video projects, matching the task to appropriate clinical expertise is important. Doctors in the Loop’s medical annotation services use medical professionals matched to the relevant field, with specialist review.
Which annotation format does a polyp detection model need?
The annotation format should reflect what the model is expected to predict. Choosing a more detailed format does not automatically make a dataset better. It adds value when the extra detail supports the intended task.
| Annotation format | What it records | When it is useful |
|---|---|---|
| Bounding box | A rectangular region around the polyp | Object detection and localization |
| Polygon | The outline of an object | More detailed shape representation |
| Segmentation mask | The pixels belonging to the polyp | Pixel-level segmentation and boundary analysis |
| Frame-level label | Whether a frame contains a polyp or another target event | Frame classification or event detection |
| Object tracking ID | The identity of an object across frames | Tracking and temporal analysis |
A bounding box is often a practical starting point for a polyp detection model. It marks where the target appears without requiring annotators to trace every boundary.
On the other hand, segmentation is more appropriate when the model must identify the visible extent of a polyp at pixel level. However, segmentation requires more detailed guidelines, particularly when the polyp boundary is unclear or blends into surrounding tissue.
Tracking introduces a different requirement: the same physical polyp must be handled consistently across time. A sequence can contain correct individual bounding boxes but still have tracking errors if the object ID changes without a defined reason.
Teams evaluating medical image annotation approaches should consider the model’s intended output, the level of detail needed for evaluation and the resources available for annotation and review.
Before choosing a format, consider:
- What output should the model produce?
- Does the evaluation metric require precise boundaries or approximate localization?
- Must the model follow an object across multiple frames?
- Which edge cases will affect performance in the intended application?
The answers should guide the annotation specification, rather than the other way around.
How should annotators handle difficult polyp frames?
Colonoscopy footage includes camera movement, changes in distance, occlusion, blur, fluid, debris and changes in illumination. The annotation guideline must explain how these conditions affect labeling.
The following are practical cases that a project guideline should address. They are examples of annotation decisions, not universal clinical rules.
1. The polyp is only partly visible
A polyp may extend beyond the image boundary or be partly obscured by an instrument or another structure.
The guideline should specify whether annotators mark only the visible portion, draw a box around the visible evidence or flag the frame for review. For segmentation, the team must also define how to handle boundaries that cannot be seen clearly.
The rule should reflect the model’s task and be applied consistently.
2. The image is blurred or poorly illuminated
Motion blur, overexposure and poor focus can make a target difficult to distinguish.
Annotators need a defined approach for frames where the polyp is not clearly visible. The guideline should specify when to label, when to mark a frame as uncertain and when to request expert review. Unclear cases should not be resolved by guessing.
3. The polyp disappears and reappears
A polyp may leave the field of view, become obscured or disappear temporarily as the camera moves.
For a detection task, the guideline should specify which frames receive a label based on visibility. For a tracking task, it should also define whether the original object ID is retained when the polyp reappears.
An absent label in one frame does not necessarily mean the polyp has ceased to exist.
4. Several structures resemble a polyp
Folds, debris, bubbles, reflections and other visual features may resemble the target in individual frames. The guideline should define the evidence required for a positive label and establish an escalation process for ambiguous cases.
Colonoscopy footage also contains normal anatomical landmarks that annotators need to distinguish from the target class. Examples include the ileocecal valve, appendiceal orifice, terminal ileum, colonic folds and Valves of Houston.
Annotators also need to distinguish potential target objects from normal anatomical structures. Colonoscopy footage contains landmarks that can look different depending on the location and viewing angle. Examples include the ileocecal valve, appendiceal orifice, terminal ileum, colonic folds and Valves of Houston. Clear class definitions help ensure that these structures are not incorrectly labeled as polyps.
These anatomical examples provide context for annotation decisions. They are not examples of polyp labels or polyp bounding-box annotations.
5. Multiple polyps appear in the same sequence
When more than one target is visible, annotators need rules for assigning separate annotations and maintaining distinct object IDs.
The team should also define how to handle overlapping boxes, partial occlusion and objects that appear close together. These decisions should reflect the model’s intended use, not individual annotator preference.
For all these cases, document the rule, provide reviewed examples where possible and make the escalation path clear. If the team cannot agree on a rule, resolve it before applying the same uncertain decision across a large dataset.
How do you keep polyp labels consistent across video frames?
Temporal consistency means that labels follow the agreed rules as the video changes. It does not mean every frame must contain an annotation or that a polyp must have identical box dimensions throughout the sequence.
A practical workflow should cover four areas.
Object identity. Assign each polyp a stable identifier when tracking is required. Define how to handle temporary disappearance, reappearance and cases where identity cannot be established confidently.
Frame sampling. Decide whether every frame, selected frames or a defined sampling interval will be annotated. Sparse sampling can reduce annotation effort, but it may omit brief appearances that matter to detection performance.
Interpolation. Annotation tools may estimate object positions between manually labeled frames. Interpolation can speed up the process, but its output needs review. Camera movement, changes in visibility and changes in the object’s apparent size can make automatically propagated boxes inaccurate.
Disappearance and reappearance. Establish rules for when an object should have no label, when an annotation should be marked as uncertain and when a returning object can retain its previous identity.
These requirements are particularly relevant when working with long procedures. Splitting a video into short clips without preserving context can make it harder to maintain consistent interpretations across segments.
How do you measure annotation quality for colonoscopy AI?
Annotation quality is not a single score. It includes agreement between annotators, object localization, consistency across frames and the reliability of the review process.
Inter-annotator agreement
Inter-annotator agreement (IAA) measures how consistently different annotators apply the same guideline. The appropriate measure depends on the annotation task.
For bounding boxes, teams may compare overlap between annotations. For segmentation, they may use metrics such as intersection over union (IoU) or Dice similarity. For classification labels, agreement measures should reflect the categories and the possibility of chance agreement.
A high agreement score does not automatically prove that labels are clinically correct. Annotators can agree with one another and still apply an incorrect or incomplete rule. Expert review and appropriate ground truth remain important.
Localization and segmentation quality
For object detection, IoU measures the overlap between a predicted region and the reference annotation:
IoU = area of overlap ÷ area of union
The project should choose evaluation thresholds that fit the intended task. Segmentation projects need metrics suited to pixel-level boundaries, while tracking tasks need additional measures that assess identity and continuity over time.
These metrics serve different purposes. A single localization score is not a complete measure of annotation quality.
A practical QA checklist
A robust quality assurance process should define how samples are reviewed, which disagreements require escalation and who can approve changes to the guideline.
Use this checklist when reviewing a colonoscopy video dataset:
- Are target classes and exclusion rules clearly defined?
- Are partial visibility, blur and occlusion covered?
- Are bounding boxes or masks applied consistently?
- Are object IDs maintained according to the tracking rules?
- Are interpolated annotations reviewed?
- Are negative and uncertain frames handled explicitly?
- Are disagreements recorded and resolved?
- Are guideline changes communicated to all annotators?
- Are QA results documented so recurring issues can be traced?
When reviewers identify recurring errors, the team should determine whether the cause is individual execution, ambiguous instructions or a systematic issue with the annotation tool or workflow. Correcting the underlying rule can be more effective than repeatedly fixing isolated labels.
What can published colonoscopy datasets teach AI teams about annotation?
Published datasets offer useful examples of how annotation choices affect model development and evaluation. They should be treated as references for designing a project, not universal templates.
One relevant example is REAL-Colon, described by Biffi and colleagues in Scientific Data in 2024. The dataset contains approximately 2.7 million native-resolution frames from 60 full-resolution colonoscopy recordings and around 350,000 bounding-box annotations produced under expert gastroenterologist supervision.
The authors highlight a limitation of datasets built mainly from selected polyp-positive images or short clips: they may not reflect the mix of positive and negative frames found in complete procedures. In REAL-Colon, approximately 87.6% of frames had no bounding-box annotation.
This has a practical implication for dataset design. Teams should consider whether their training and evaluation data represent the conditions in which the model is expected to operate, including frames without visible polyps.
The paper also describes polyps that temporarily disappear from view, resulting in separate tracklets rather than one continuous sequence of visible annotations. This illustrates why object identity and visibility rules need explicit treatment in video annotation. Read the study: REAL-Colon: A dataset for developing real-world AI applications in colonoscopy.
The broader lesson is that dataset size alone does not establish quality. Annotation format, coverage of complete procedures, clinical supervision, treatment of difficult frames and transparency of the evaluation process all matter.
Doctors in the Loop provides medical data annotation services for AI teams working with colonoscopy and other medical specialties.
Our team includes medical professionals who understand the clinical context of the data. We work closely with clients and their technical teams to define annotation guidelines, apply consistent labels, and review the quality of the annotated datasets.
Have a colonoscopy video dataset to annotate? Get in touch with our team.
