g ddlZddlZddlZddlZddlmZej rddlmZm Z dZ eje Z dedefdZdd dedefd Zdd ded edej"efd Zdd ded ej(edefdZdddedefdZddded edej"efdZddded ej(edefdZdedefdZy)N) Tokenizer) AsyncClientClient tokenizersmodelreturnctd|S)N:)TOKENIZER_CACHE_KEY)rs Z/var/www/openai/venv/lib/python3.12/site-packages/cohere/manually_maintained/tokenizers.pytokenizer_cache_keyr s! "!E7 ++corc|jt|}||S|jj|j}|st d| t |}tjd|d|dtj|}tj|j}|jt|||S#t$r"}tjd|Yd}~wd}~wwxYwz9Returns a HF tokenizer from a given tokenizer config URL.Nz!No tokenizer URL found for model z Downloading tokenizer for model z . Size is z MBs.z0Failed to get the size of the tokenizer config: ) _cache_getr modelsget tokenizer_url ValueError_get_tokenizer_config_sizeloggerinfo Exceptionwarnrequestsrfrom_strtext _cache_setrr tokenizerrsizeeresponses r get_hf_tokenizerr%s 1%89IIIMM%(66M >M >r)r5loggingrCrrr TYPE_CHECKING cohere.clientrrr getLogger__name__rstrr r%ListrBr-Sequencer3r8r:r<floatrrr rQsH   1"   8 $,s,s,#)0@x@@3@6;;s;K@ $$#$vs7K$PS$ ]392@=@@C@FKKX[L\@ $]$3$X[H\$ad$ ?c ?e ?r