ComfyUI Node: 🪙 String Tokenizer

String Tokenizer [Dream]:

The String Tokenizer [Dream] node is designed to split a given string into smaller segments or tokens based on a specified separator. This node is particularly useful for breaking down large blocks of text into manageable pieces, allowing you to focus on specific parts of the text for further processing or analysis. By providing a simple and intuitive way to tokenize strings, this node helps streamline workflows that involve text manipulation, making it easier to handle and analyze textual data in your AI art projects.

String Tokenizer [Dream] Input Parameters:

text

The text parameter is the main input string that you want to tokenize. This parameter accepts a string value and can handle multiline text, making it versatile for various text processing tasks. The default value is an empty string (""), and you can input any text you need to split into tokens.

separator

The separator parameter defines the character or string used to split the text into tokens. By default, this is set to a comma (,), but you can specify any character or string that suits your needs. If the separator is not provided or is an empty string, the node will use a space (" ") as the default separator.

selected

The selected parameter determines which token to select from the list of tokens generated after splitting the text. This parameter accepts an integer value, with the default set to 0. The value should be within the range of the number of tokens generated. If the value is negative or exceeds the number of tokens, it will wrap around using the absolute value modulo the number of tokens.

String Tokenizer [Dream] Output Parameters:

token

The token output parameter provides the selected token from the list of tokens generated by splitting the text based on the specified separator. This output is a string representing the specific segment of the original text that you have chosen to focus on, making it easier to work with individual parts of the text in subsequent nodes or processes.

String Tokenizer [Dream] Usage Tips:

• To split a paragraph into sentences, use a period (".") as the separator.
• For CSV data, keep the default comma (,) separator to tokenize each field.
• Use the selected parameter to quickly access specific parts of the text, such as the first word or the last sentence.

String Tokenizer [Dream] Common Errors and Solutions:

"IndexError: list index out of range"

• Explanation: This error occurs when the selected parameter value exceeds the number of tokens generated from the text.
• Solution: Ensure that the selected value is within the range of the number of tokens. You can use a modulo operation to wrap around the index if needed.

"ValueError: empty separator"

• Explanation: This error occurs when the separator parameter is set to an empty string.
• Solution: Provide a valid separator character or string. If you want to use a space as the separator, explicitly set the separator parameter to " ".

"TypeError: text.split is not a function"

• Explanation: This error occurs if the text parameter is not a string.
• Solution: Ensure that the text parameter is a valid string. Convert any non-string input to a string before passing it to the node.