ComfyUI Node: Florence2Run

Florence2Run:

Florence2Run is a powerful node designed to process images and text inputs using the Florence2 model, a sophisticated AI model for various image and text-related tasks. This node allows you to perform tasks such as object detection, dense region captioning, region proposal, and more, by leveraging the capabilities of the Florence2 model. It is particularly useful for AI artists who want to generate detailed captions, segment images based on referring expressions, or perform OCR (Optical Character Recognition) on images. The node ensures that the model is efficiently loaded and offloaded to manage resources effectively, making it a versatile tool for complex image and text processing tasks.

Florence2Run Input Parameters:

image

The image parameter is the input image that you want to process using the Florence2 model. This image serves as the primary data on which various tasks like object detection, captioning, and segmentation will be performed. The quality and content of the image can significantly impact the results of the processing tasks.

text_input

The text_input parameter is used to provide additional textual information or prompts for specific tasks such as referring_expression_segmentation and caption_to_phrase_grounding. This input helps the model to focus on particular aspects of the image based on the provided text. If the task does not require text input, this parameter can be left empty.

florence2_model

The florence2_model parameter is the pre-loaded Florence2 model that will be used for processing the image and text inputs. This model includes both the neural network and the processor required for the tasks. It is essential to ensure that the correct model is loaded to achieve accurate results.

task

The task parameter specifies the type of processing you want to perform on the image. Options include region_caption, dense_region_caption, region_proposal, caption, detailed_caption, more_detailed_caption, caption_to_phrase_grounding, referring_expression_segmentation, ocr, and ocr_with_region. Each task has a specific purpose and will generate different types of outputs based on the input image and text.

fill_mask

The fill_mask parameter is a boolean flag that indicates whether to fill the mask for segmentation tasks. When set to True, the model will generate a filled mask for the segmented regions in the image. This is particularly useful for tasks like referring_expression_segmentation.

keep_model_loaded

The keep_model_loaded parameter is a boolean flag that determines whether the Florence2 model should remain loaded in memory after processing. Setting this to True can save time if you plan to run multiple tasks sequentially, but it will consume more memory. The default value is False.

num_beams

The num_beams parameter controls the number of beams used in beam search for generating captions. A higher number of beams can lead to more accurate and diverse captions but will increase the computational load. The default value is 3.

max_new_tokens

The max_new_tokens parameter specifies the maximum number of new tokens to generate for tasks involving text generation, such as captioning. This limits the length of the generated text and helps control the output size. The default value is 1024.

do_sample

The do_sample parameter is a boolean flag that determines whether to use sampling for text generation. When set to True, the model will generate text by sampling from the probability distribution, leading to more diverse outputs. The default value is True.

Florence2Run Output Parameters:

out_tensor

The out_tensor parameter is the primary output tensor containing the processed image data. This tensor includes the results of the specified task, such as detected objects, generated captions, or segmented regions. It is essential for further analysis or visualization of the results.

out_mask_tensor

The out_mask_tensor parameter is the output tensor containing the mask data for segmentation tasks. This tensor provides the segmented regions of the image, which can be used for detailed analysis or further processing. If no masks are generated, this tensor will contain a default value.

out_results

The out_results parameter is a list of additional results generated by the Florence2 model. This can include various metadata, such as confidence scores, bounding boxes, or textual descriptions, depending on the specified task. These results provide valuable insights into the processed image and text data.

Florence2Run Usage Tips:

• Ensure that the input image is of high quality and relevant to the task to achieve the best results.
• Use the text_input parameter effectively for tasks like referring_expression_segmentation to guide the model's focus.
• Experiment with the num_beams and max_new_tokens parameters to balance between computational load and output quality.
• Set keep_model_loaded to True if you plan to run multiple tasks sequentially to save loading time.

Florence2Run Common Errors and Solutions:

ValueError: Text input (prompt) is only supported for 'referring_expression_segmentation' and 'caption_to_phrase_grounding'

• Explanation: This error occurs when text_input is provided for a task that does not support it.
• Solution: Ensure that text_input is only provided for the supported tasks or leave it empty for other tasks.

Offloading model...

• Explanation: This message indicates that the model is being offloaded from memory to manage resources.
• Solution: This is an informational message and does not require any action. If you want to keep the model loaded, set keep_model_loaded to True.

Downloading Lumina model to: <model_path>

• Explanation: This message indicates that the model is being downloaded because it is not found locally.
• Solution: Wait for the download to complete. Ensure you have a stable internet connection and sufficient disk space.

using <attention> for attention

• Explanation: This message indicates the type of attention mechanism being used by the model.
• Solution: This is an informational message and does not require any action.