token_estimator

hiveadb.ai.token_estimator(string: str, encoding_name: str) → int

Estimate the number of tokens for a given string using a specified encoding.

This function uses the tiktoken library to encode the given string and then returns the number of tokens in the encoded string.

Parameters:

• string (str) – The string for which the token count needs to be estimated.

• encoding_name (str) – The name of the encoding to be used for token estimation.

Returns:

The number of tokens in the encoded string.

Return type:

int

Example

from hiveadb.ai import token_estimator

token_count = token_estimator("I love waking up and seeing text messages on my phone.", "gpt-3.5-turbo")