gokayfem/ComfyUI_VLM_nodes

ComfyUI VLM Nodes

Production-oriented vision-language, structured prompting, audio, and utility nodes for ComfyUI. Version 3.4 supports ComfyUI's selected NVIDIA CUDA, AMD ROCm, Apple Metal, Intel XPU, and CPU device without replacing its PyTorch build. It removes startup installers and global accelerator cache flushes, adds real image/video batches and live token streaming, and uses ComfyUI model residency and offloading.

Modern model coverage

The Modern VLM node provides one stable interface with a deliberately small, 12-choice production picker:

• Qwen 3.5 0.8B and 4B
• Qwen 3 VL 2B, 4B, and 8B Instruct
• SmolVLM2 500M and 2.2B Video
• Liquid LFM2.5-VL 450M
• InternVL 3.5 1B
• Granite Vision 4.1 4B
• Gemma 3 4B IT
• a compatible custom Hugging Face image-to-text repository

The separate [Legacy] Modern VLM Compatibility node contains redundant, superseded, experimental, and very large tiers:

• Qwen 3.5 2B, 9B, 27B, and 35B-A3B
• Qwen 3.6 27B
• Qwen 3 VL 30B-A3B Instruct
• Qwen 2.5 VL 3B and 7B for existing workflows
• Gemma 3 12B and 27B IT
• SmolVLM2 256M Video
• Liquid LFM2.5-VL 1.6B
• InternVL 3.5 2B
• Granite Vision 3.3 2B

Previously saved ModernVLM workflows remain valid even when their selected model moved to Legacy. The server accepts every known catalog value for backward compatibility; only the visible new-workflow picker is curated. Dedicated Molmo, PaLI-Gemma, Qwen2-VL, MiniCPM-V, Kosmos-2, MC-LLaVA, UForm, and script-style MoonDream nodes are also collected under VLM Nodes/Legacy/Model Loaders. Maintained creator-facing Florence-2, Moondream2, JoyTag, llama.cpp/GGUF, detection, segmentation, tracking, API, and video-intelligence nodes stay in their functional categories.

Sixteen curated sub-4B/low-VRAM choices are marked internally as the small-and-fast tier. The default is Qwen 3 VL 2B: it is much quicker to load than larger checkpoints while retaining broad image and video understanding. The catalog intentionally uses official model repositories and maintained Transformers interfaces rather than unverified community quantizations. Curated models use native Transformers implementations; remote repository code is enabled only when the explicit custom-model option requires it. Florence-2 uses the Transformers-native converted checkpoints instead of Microsoft’s legacy repository code.

Live text output

Modern VLM streams decoded text through ComfyUI's native progress_text WebSocket channel by default. A connected ViewText node updates while tokens arrive, shows the final response after execution, and restores the last result when ComfyUI rehydrates workflow output history. Disable stream_output for API-only or headless runs that do not need incremental UI updates. Streaming is best-effort and never changes the final STRING output or makes inference fail.

Text workflow toolkit

The original SimpleText, JsonToText, and ViewText node IDs and their first STRING outputs remain stable for saved workflows. They now live in organized VLM Nodes/Text subcategories and expose descriptive names, search aliases, tooltips, appended metrics, and strict error messages:

| Node | Purpose | | --- | --- | | Text (SimpleText) | Multiline/dynamic prompt source with optional edge/newline normalization and character, word, and line outputs | | View Text (Streaming) | Read-only live output with counts, copy, UTF-8 download, line wrapping, stream following, reroute traversal, and history rehydration | | JSON to Text | Plain or fenced JSON parsing with readable, values-only, key/value, pretty, and compact render modes | | Text Join | Join up to eight prompt/context values with empty-value removal and stable deduplication | | Text Template | Safe named placeholders from a JSON object plus four convenient live text sockets, with explicit missing-key policy | | Text Clean | Unicode NFC/NFKC, newline/whitespace cleanup, enclosing Markdown-fence removal, line deduplication, and deterministic length caps | | Text Replace | Literal or regex substitution with case, count, and missing-pattern controls | | JSON Extract | JSONPath-lite ($.items[0]) and RFC 6901 JSON Pointer extraction from plain or fenced model responses | | Text Split / Batch | Lines, paragraphs, delimiters, regex, CSV, or JSON arrays converted to a real mapped Comfy STRING list | | Text Inspector | Pass-through text plus characters, UTF-8 bytes, words, lines, rough token budget, SHA-256, and JSON metadata |

The JSON utilities never evaluate code, follow references, access files, or make network requests. Template fields are direct names rather than Python attribute/index expressions. approx_tokens is deliberately labeled as a rough UTF-8 budget estimate; use the target model tokenizer when exact billing or context accounting matters.

Specialized nodes remain available where a generic chat node would discard useful model capabilities:

• Moondream 3.1 9B-A2B: official 2B-active Photon runtime with query,

caption, and high-throughput image/video detection and pointing.

• Moondream 3 Preview segment: native SVG segmentation through the same

isolated Photon loader. The SVG is preserved and also converted into antialiased MASK, black/white previews, foreground cutouts, overlays, polygons, canonical VLM_DETECTIONS, and core bounding boxes. Detection/pointing submit frames concurrently so Photon can dynamically batch them; every run reports measured worker FPS, end-to-end FPS, and real-time factor.

• Florence-2: captioning, OCR, detection, region captioning, and referring

expression segmentation, with structured JSON, mask, and overlay outputs.

• PaLI-Gemma: caption/VQA plus the official 16-token VQ-VAE segmentation

decoder; segmentation tokens are no longer misinterpreted as polygon points.

• Moondream2: pinned query API with explicit decoding controls. The official

checkpoint is loaded through its native safetensors state dict, avoiding the silent empty-output regression in Transformers 5 while retaining ComfyUI managed loading and unloading.

• Qwen2-VL: image batches and real video-frame batches.
• Legacy Molmo, Kosmos-2, UForm, MCLLaVA, and MiniCPM-V 2.6 GGUF, plus

maintained JoyTag.

• llama.cpp LLaVA/GGUF, structured prompt suggestions, OpenAI-compatible

prompting, and AudioLDM2.

Structured detection, segmentation, and tracking

The vision nodes use stable, typed sockets instead of passing model-specific lists between nodes:

| Socket | JSON schema | Purpose | | --- | --- | --- | | VLM_DETECTIONS | comfyui-vlm/detections, version 1 | Per-frame boxes, labels, scores, optional polygons/quads, and in-process masks | | VLM_TRACKS | comfyui-vlm/tracks, version 1 | Durable object IDs with ordered observations over time | | VLM_POINTS | comfyui-vlm/points, version 1 | Pixel-coordinate points, including detection centers | | VLM_EVENTS | comfyui-vlm/events, version 1 | Ordered temporal events for downstream video analysis | | VLM_VIDEO_SELECTION | comfyui-vlm/video-selection, version 1 | Exact mapping from sampled images to source frame indices and timestamps | | VLM_SCENE_STATE | comfyui-vlm/scene-state, version 1 | Compact persistent objects, motion, visibility, and validated events |

All spatial coordinates are source-image pixels. Bounding boxes are [x1, y1, x2, y2] with an exclusive right/bottom edge; polygons contain at least three points and quads exactly four. JSON roots contain schema, version, media dimensions/frame count/FPS, and their ordered records. Dense mask tensors remain in-process and are deliberately omitted from JSON so API results do not unexpectedly grow by hundreds of megabytes.

The utility layer converts without model-specific glue:

• VLMStructuredSpatialParser strictly parses pixel, normalized 0–1, or

normalized 0–1000 JSON from any VLM into VLM_DETECTIONS and VLM_POINTS. VLMSpatialPromptBuilder creates the matching constrained prompt.

• VLMDetectionsToBoundingBoxes, VLMDetectionsToPoints, and

VLMDetectionsToMasks emit Comfy core boxes, center points, combined and individual binary masks, inverse masks, ready-to-preview black-and-white images, and stable-color instance maps. Polygon/quad masks are rasterized when present, otherwise the bounding box is used. Existing output indexes remain stable; the creator-facing mask images and instance map are appended.

• VLMFilterDetections, VLMSelectDetection, VLMCropDetections, and

VLMRenderDetections provide label/score/area/frame selection, padded crops, and deterministic overlays.

• VLMMaskProcessor accepts any Comfy MASK, including SAM2/SAM3 masks, and

returns a feathered matte, strict binary mask, inverse mask, and black-and-white image. Its grow/shrink and Gaussian feathering run in Torch without OpenCV or SciPy.

• VLMMaskComposite applies still-image or video mask batches to a source and

returns the replacement composite, isolated foreground, original background-only plate, and black-and-white mask image. A single mask or background broadcasts safely across a video batch.

• VLMDetectionsFromJSON and VLMDetectionsToJSON are the explicit API and

persistence boundary for the versioned detection schema.

Universal VLM performance utilities

The performance nodes sit before any local or hosted VLM, so their savings do not depend on CUDA, ROCm, MPS, XPU, CPU, Transformers, llama.cpp, or Photon:

• VLM Performance Profile emits coherent max_frames, pixel budget,

longest-edge, batch-size, and unload_after values. Live / robotics, Fast video, Balanced, High detail, and Low VRAM handoff are explicit starting points rather than hidden global flags.

• VLM Adaptive Frame Sampler is the existing track-aware temporal gate. It

combines uniform coverage, scene changes, motion, and optional track changes while preserving source frame indices and timestamps.

• VLM Image Pixel Budget downsizes the selected analysis copy once, preserves

aspect ratio, never upscales, and can align dimensions to 14/28-pixel VLM patches or 32-pixel detector backbones. Fast area and antialiased bicubic modes are available.

The recommended order is Video Slice → VLM Adaptive Frame Sampler → VLM Image Pixel Budget → any VLM. A model's own official processor still performs its required normalization/crop; the pixel-budget node simply prevents every downstream model from repeatedly receiving unnecessary source pixels. Local torch models remain registered with ComfyUI's smart model manager, while external allocators reserve space before loading and close only the handle they own.

On the real vlm_api_people_birds.mp4 input in this repository's D-drive test environment, the utilities selected 10 of 60 1280×720 frames and resized them to 938×518 in about 0.44 seconds on a cold WSL run. That reduced the frame×pixel analysis workload by 11.38× before model inference. This is an input-work reduction measurement, not a claim that every model runs 11.38× faster; token generation and model-specific vision encoders still determine end-to-end speed.

Adaptive video intelligence

The video-intelligence layer keeps generative VLM inference out of the per-frame loop:

• VLMAdaptiveFrameSampler combines scene-change, motion, track-change, and

uniform-coverage signals. It always preserves the real source frame index and timestamp, enforces a frame budget, and returns selection/diagnostic JSON. Uniform coverage, motion, scene, and track-priority modes remain available for deterministic experiments.

• VLMVideoTemporalReasoner is the one-node path. It adaptively samples the

input, downsizes only the VLM analysis copy (448-pixel longest side by default), runs a recommended video-capable model, parses the result into validated VLM_EVENTS, and returns summary, events, selection, sampled previews, raw response, diagnostics, event JSON, and selection JSON.

• VLMVideoReasoningPrompt and VLMEventsFromVideoJSON expose the same strict

timestamp/evidence contract for custom local or hosted VLM workflows.

• VLMTrackAwareCrops chooses representative observations for each durable

track, adds configurable context, and letterboxes crops to one batch size. This lets a VLM label identities without rereading every full frame.

• VLMBuildSceneState converts tracks plus optional events into a compact

persistent world-state summary with first/last observation, current box, confidence, state, and pixel velocity.

Small VLMs commonly return evidence as positions in the supplied image batch even when asked for source indices. The parser accepts that form only when every value is an unambiguous valid supplied-image position, maps it back to the immutable source selection, and records the normalization mode. Arbitrary or unsupplied evidence frames, out-of-range timestamps, invalid confidence, duplicate evidence, malformed JSON, and non-finite values fail validation.

On the repository's real-data smoke test (RTX 3090, Qwen3-VL 2B, 157-frame 896x448 H.264 clip), hybrid sampling selected 12 frames in 0.30 seconds, reduced temporal inputs by 92.36%, reduced analysis pixels by 75%, used 4.24 GiB peak allocated VRAM in the standalone runner, and produced a valid timestamped result in 35.17 seconds. The equivalent live ComfyUI /prompt graph completed in 37.45 seconds. These are one-machine measurements, not portable performance guarantees.

Open-vocabulary image and video detection

VLMOpenVocabularyDetection exposes one interface for:

• Grounding DINO Tiny and Base
• OWLv2 Base Ensemble
• OmDet Turbo Swin Tiny

It accepts a still image or an IMAGE batch of video frames and processes the batch frame by frame. Outputs, in socket order, are detections, json, preview, box_mask, and Comfy core bounding_boxes. Connect the FPS output of GetVideoComponents when the input is video so every timestamp is correct. For tracking-by-detection, run detection over the complete bounded batch and connect it to VLMTrackDetections.

VLMTrackDetections uses a ByteTrack-style two-stage high/low-confidence association, motion prediction, label-aware matching, and time-based expiry. IDs are durable within the supplied sequence and survive short missed detections when emit_predictions is enabled. Independent Comfy queue runs or independently sliced chunks are separate tracking sessions; they do not silently reuse IDs.

SAM2.1 and Comfy core SAM3.1

VLMSAM2VideoSegmentation propagates first-frame detections, one core BOUNDING_BOX, or seed masks through an IMAGE batch using SAM2.1 Hiera Tiny, Small, Base+, or Large. It returns VLM_TRACKS, report JSON, per-frame union masks, frame-major individual object masks, and an overlay batch. The object IDs assigned at the seed frame remain stable for that video session.

VLMSAM3TrackAdapter is intentionally an adapter, not a second SAM3 loader. It validates ComfyUI core SAM3_TRACK_DATA, preserves the core bit-packed mask payload unchanged, and exposes lightweight VLM_TRACKS metadata with mask references. Connect its passthrough output to core SAM3_TrackPreview or SAM3_TrackToMask, and connect tracks to VLMTrackReport. This avoids duplicating dense masks in memory or JSON.

SAM3 weights use Meta's SAM License. The upstream facebook/sam3 repository requires accepting access terms and sharing the requested account information; the ComfyUI checkpoint is also marked sam-license. Review and accept the license before downloading. The example names ComfyUI's sam3.1_multiplex_fp16.safetensors; if it is unavailable, use the SAM2.1 workflow rather than substituting an unrelated checkpoint.

Florence-2 task coverage

Florence2 exposes all 15 supported task contracts:

| Task | Extra input | Structured result | | --- | --- | --- | | Caption | none | text | | Detailed caption | none | text | | More detailed caption | none | text | | OCR | none | text | | OCR with regions | none | text plus quadrilateral regions | | Object detection | none | labeled boxes | | Dense region caption | none | captions with boxes | | Caption to phrase grounding | text_input | phrase boxes | | Referring expression segmentation | text_input | polygons and mask | | Region to segmentation | one BOUNDING_BOX per image | polygons and mask | | Open vocabulary detection | text_input | model-provided spatial records | | Region to category | one BOUNDING_BOX per image | text | | Region to description | one BOUNDING_BOX per image | text | | Region to OCR | one BOUNDING_BOX per image | text | | Region proposals | none | boxes |

Every task returns text, structured_json, mask, and visualization. Tasks that do not produce a spatial result return an empty mask and the source image visualization. Region tasks reject ambiguous multi-box input; use VLMSelectDetection to isolate the record, then supply exactly one core BOUNDING_BOX with the same pixel coordinates.

Video memory strategy

• Trim long media with core Video Slice, then use GetVideoComponents.

Downscale the complete frame batch before detection or segmentation and keep every frame at identical dimensions.

• Grounding detection supports configurable micro-batches; keep batch_size=1

for minimum VRAM or increase it when memory allows. It returns both nested per-frame core BOUNDING_BOX values and flat metadata-rich BOUNDING_BOXES.

• SAM2.1 stores source video frames on CPU, keeps its inference state on CPU by

default, and limits the vision-feature cache to one frame. Union masks and previews return on CPU. Full per-object mask volumes are opt-in with mask_output=union_and_objects; disable render_preview to avoid another full-resolution overlay copy on long clips.

• Start with Grounding DINO Tiny plus SAM2.1 Hiera Tiny. Increase detector or

segmenter size only after the pipeline is correct. unload_after=false caches one model per node instance; use true when another large model must run immediately afterward.

• A Video Slice is an independent propagation session. For very long media,

use bounded slices, reseed each slice, and keep the overlap/output mapping in the caller. The pack does not pretend IDs are globally stable across separate queues.

• The SAM3 adapter never unpacks the complete mask volume for its report. Use

core SAM3_TrackToMask only when a dense selected mask is actually needed.

API-format examples are in examples/vision:

• grounding_dino_image_api.json
• moondream3_preview_svg_segment_api.json
• moondream31_video_detect_api.json
• sam2_video_tracking_api.json
• sam3_core_adapter_blueprint_api.json
• video_temporal_reasoning_api.json
• vlm_performance_preflight_api.json

The dependency-free text-toolkit example is examples/text_toolkit_api.json. Robotics policy, safety, and sidecar examples are in examples/robotics, including a complete universal HTTP policy graph.

Upload the named media to ComfyUI's input directory, adjust the filenames and labels, then submit the JSON object as the prompt value to /prompt. These are API graphs, not frontend workflow-export JSON.

Node reference

All 89 registered nodes, grouped by their menu category. The Node ID is the class_type written into workflow and API JSON — search for that string when you need to find a node you saw on a canvas.

Modern VLM

The main entry point for current vision-language models.

| Node | Node ID | Outputs | | --- | --- | --- | | Modern VLM (Qwen / SmolVLM2 / LFM / InternVL / Granite / Gemma) | ModernVLM | STRING | | Moondream 2 | Moondream2model | STRING |

Moondream 3

Moondream 3 / 3.1 in an isolated Photon runtime. Load once, then reuse the MOONDREAM31_MODEL output across the task nodes.

| Node | Node ID | Outputs | | --- | --- | --- | | Moondream 3 / 3.1 Loader (Isolated Photon) | Moondream31Loader | MOONDREAM31_MODEL, STRING | | Moondream 3 / 3.1 Caption | Moondream31Caption | STRING, STRING | | Moondream 3 / 3.1 Query | Moondream31Query | STRING, STRING, STRING | | Moondream 3 / 3.1 Detect (Image / Video) | Moondream31Detect | VLM_DETECTIONS, STRING, IMAGE, MASK, BOUNDING_BOX, BOUNDING_BOXES, STRING | | Moondream 3 / 3.1 Point (Image / Video) | Moondream31Point | VLM_POINTS, STRING, IMAGE, STRING | | Moondream 3 Preview SVG Segment (Image / Video) | Moondream31Segment | VLM_DETECTIONS, STRING, STRING, MASK, IMAGE, IMAGE, IMAGE, BOUNDING_BOX, BOUNDING_BOXES, STRING |

Florence-2

| Node | Node ID | Outputs | | --- | --- | --- | | Florence-2 Multitask Vision | Florence2 | STRING, STRING, MASK, IMAGE |

Vision: detection, segmentation, tracking

Open-vocabulary detection and video segmentation. These emit the structured VLM_DETECTIONS / VLM_POINTS / VLM_TRACKS types rather than loose strings.

| Node | Node ID | Outputs | | --- | --- | --- | | VLM Open-Vocabulary Detection | VLMOpenVocabularyDetection | VLM_DETECTIONS, STRING, IMAGE, MASK, BOUNDING_BOX, BOUNDING_BOXES | | VLM SAM2.1 Video Segmentation | VLMSAM2VideoSegmentation | VLM_TRACKS, STRING, MASK, MASK, IMAGE | | VLM SAM3 Track Adapter | VLMSAM3TrackAdapter | VLM_TRACKS, SAM3_TRACK_DATA | | VLM Track Detections | VLMTrackDetections | VLM_TRACKS | | VLM Track Report | VLMTrackReport | STRING, STRING | | JoyTag | Joytag | STRING |

Vision: spatial reasoning

| Node | Node ID | Outputs | | --- | --- | --- | | VLM Spatial Prompt Builder | VLMSpatialPromptBuilder | STRING | | VLM Structured Spatial Parser | VLMStructuredSpatialParser | VLM_DETECTIONS, VLM_POINTS, STRING |

Vision: detection utilities

Converters and filters between structured detections and ordinary Comfy types.

| Node | Node ID | Outputs | | --- | --- | --- | | Filter VLM Detections | VLMFilterDetections | VLM_DETECTIONS | | Select VLM Detection | VLMSelectDetection | VLM_DETECTIONS | | Crop VLM Detections | VLMCropDetections | IMAGE, STRING | | Render VLM Detections | VLMRenderDetections | IMAGE | | VLM Detection Centers | VLMDetectionsToPoints | VLM_POINTS, STRING | | VLM Detections from JSON | VLMDetectionsFromJSON | VLM_DETECTIONS | | VLM Detections to JSON | VLMDetectionsToJSON | STRING | | VLM Detections to Bounding Boxes | VLMDetectionsToBoundingBoxes | BOUNDING_BOXES, STRING | | VLM Detections to Masks | VLMDetectionsToMasks | MASK, MASK, STRING, MASK, IMAGE, IMAGE, IMAGE |

Vision: mask tools

| Node | Node ID | Outputs | | --- | --- | --- | | VLM Mask Processor | VLMMaskProcessor | MASK, MASK, MASK, IMAGE | | VLM Mask Composite | VLMMaskComposite | IMAGE, IMAGE, IMAGE, IMAGE |

Video intelligence

Adaptive frame selection and temporal reasoning for long videos.

| Node | Node ID | Outputs | | --- | --- | --- | | VLM Adaptive Frame Sampler | VLMAdaptiveFrameSampler | IMAGE, VLM_VIDEO_SELECTION, STRING, STRING | | VLM Video Reasoning Prompt | VLMVideoReasoningPrompt | STRING, STRING | | VLM Video Temporal Reasoner | VLMVideoTemporalReasoner | STRING, VLM_EVENTS, VLM_VIDEO_SELECTION, IMAGE, STRING, STRING, STRING, STRING | | VLM Temporal Events From JSON | VLMEventsFromVideoJSON | VLM_EVENTS, STRING, STRING | | VLM Persistent Scene State | `VLMBui