g :ddlZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z m Z ddlmZddlmZmZddlmZddlmZddlmZdd lmZddlZdd lmZdd ldd lmZdd l m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1dZ2Gdde3eZ4GddZ5e6jndddgdfde e6de e3de e3d e e3d!e e e8f d"Z9e6jndddgdfde e6de e3de e3d e e3d!e e e8f d#Z:d$Z;y)%N)Enum)AnyDictListOptionalSetUnion)TranscriptionCreateParams)"CompletionCreateParamsNonStreamingCompletionCreateParamsStreaming)r )r )EmbeddingCreateParams) BaseModel)verbose_logger)*) RerankRequest)all_litellm_params) BaseCache) DiskCache) DualCache) InMemoryCache)QdrantSemanticCache) RedisCache)RedisSemanticCache)S3Cachec tj|tjr t |yy#t $rYywxYw)N)rdebuglitellm set_verboseprint Exception)print_statements L/var/www/openai/venv/lib/python3.12/site-packages/litellm/caching/caching.py print_verboser$0s= _-    / "     s04 AAceZdZdZdZy) CacheMode default_on default_offN)__name__ __module__ __qualname__r'r(r#r&r&9s JKr-r&c:JeZdZejej ddddddddgddddddddddddddddddddddf deedeedeed eed eed eed ee d ee dee dee dee e deedeedeedee dee e efdeedeedeedeedeedeedeedee deedeed eed!eef8d"Zd#efd$Zd%ed&ed#eefd'Zd&ed#efd(Zd)ed*eed#eefd+Zd&ed#efd,Zd#eefd-Zd.ed#dfd/Zd#eefd0Zd#eefd1Zd#eefd2Zd#eefd3Zd#eefd4Zd#eefd5Zd#eefd6Z e!d7ed#efd8Z"d9ed#efd:Z#d;Z$dZ%d?Z&d@Z'dAZ(dBZ)dCZ*dDZ+dEZ,dFZ-dGZ.dHZ/dIZ0d#e fdJZ1y)KCacheN completion acompletion embedding aembeddingatranscription transcriptionatext_completiontext_completionarerankrerankTFztext-embedding-ada-002typemodehostportpassword namespacettldefault_in_memory_ttldefault_in_redis_ttlsimilarity_thresholdsupported_call_typess3_bucket_names3_region_names3_api_version s3_use_ssl s3_verifys3_endpoint_urls3_aws_access_key_ids3_aws_secret_access_keys3_aws_session_token s3_configs3_pathredis_flush_sizeredis_startup_nodesqdrant_api_baseqdrant_api_keyqdrant_collection_nameqdrant_quantization_configc! |tjk(rtd|||||d|!|_n|tjk(rt d|||| ||d|!|_n|tj k(rt|||| || |_nw|tjk(rt|_nT|tjk(rtd| | |||||||||d |!|_n$|tjk(rt||_dtjvrtjj!ddtj"vrtj"j!ddtj$vrtj$j!d| |_||_||_||_||_|xst0j2|_|j(tjk(r |||_|j(tjk(s|j(tjk(r | | |_|j*7t7|jtr|j*|j_yyy) a Initializes the cache based on the given type. Args: type (str, optional): The type of cache to initialize. Can be "local", "redis", "redis-semantic", "qdrant-semantic", "s3" or "disk". Defaults to "local". # Redis Cache Args host (str, optional): The host address for the Redis cache. Required if type is "redis". port (int, optional): The port number for the Redis cache. Required if type is "redis". password (str, optional): The password for the Redis cache. Required if type is "redis". namespace (str, optional): The namespace for the Redis cache. Required if type is "redis". ttl (float, optional): The ttl for the Redis cache redis_flush_size (int, optional): The number of keys to flush at a time. Defaults to 1000. Only used if batch redis set caching is used. redis_startup_nodes (list, optional): The list of startup nodes for the Redis cache. Defaults to None. # Qdrant Cache Args qdrant_api_base (str, optional): The url for your qdrant cluster. Required if type is "qdrant-semantic". qdrant_api_key (str, optional): The api_key for the local or cloud qdrant cluster. qdrant_collection_name (str, optional): The name for your qdrant collection. Required if type is "qdrant-semantic". similarity_threshold (float, optional): The similarity threshold for semantic-caching, Required if type is "redis-semantic" or "qdrant-semantic". # Disk Cache Args disk_cache_dir (str, optional): The directory for the disk cache. Defaults to None. # S3 Cache Args s3_bucket_name (str, optional): The bucket name for the s3 cache. Defaults to None. s3_region_name (str, optional): The region name for the s3 cache. Defaults to None. s3_api_version (str, optional): The api version for the s3 cache. Defaults to None. s3_use_ssl (bool, optional): The use ssl for the s3 cache. Defaults to True. s3_verify (bool, optional): The verify for the s3 cache. Defaults to None. s3_endpoint_url (str, optional): The endpoint url for the s3 cache. Defaults to None. s3_aws_access_key_id (str, optional): The aws access key id for the s3 cache. Defaults to None. s3_aws_secret_access_key (str, optional): The aws secret access key for the s3 cache. Defaults to None. s3_aws_session_token (str, optional): The aws session token for the s3 cache. Defaults to None. s3_config (dict, optional): The config for the s3 cache. Defaults to None. # Common Cache Args supported_call_types (list, optional): List of call types to cache for. Defaults to cache == on for all call types. **kwargs: Additional keyword arguments for redis.Redis() cache Raises: ValueError: If an invalid cache type is provided. Returns: None. Cache is set as a litellm param )r=r>r?rQ startup_nodes)r=r>r?rD use_asyncembedding_model)rSrTcollection_namerDquantization_configrZ) rFrGrHrIrJrKrLrMrNrOrP)disk_cache_dircacheNr,)LiteLLMCacheTypeREDISrr^REDIS_SEMANTICrQDRANT_SEMANTICrLOCALrS3rDISKrrinput_callbackappendsuccess_callback_async_success_callbackrEr;r@rQrAr&r'r< isinstance)"selfr;r<r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPredis_semantic_cache_use_async$redis_semantic_cache_embedding_modelrQrRr]rSrTrUrV%qdrant_semantic_cache_embedding_modelkwargss" r#__init__zCache.__init__@sO@ #)) )$.%!!11 %  %DJ%44 4+!%98 D DJ%55 5, /- 6%9$> E DJ%++ +&DJ %(( ( ---%# /%9)A%9#  DJ%** *".ADJ '00 0  " " ) )' 2 '22 2  $ $ + +G 4 '99 9  + + 2 27 ;$8! " 0#;y';'; 99(.. .3H3T,DH II)// /yy,;;;".+DH >> %*TZZ*L#'>>DJJ +M %r-returnc *d}|jdi|}|tjd||S|j}t}|D]w}||vr2|j ||}||t |dt |z }9||vs>tjdusQ||W||}|t |dt |z }ytjd|tj|}|j|fi|}|jdd|i||S)a  Get the cache key for the given arguments. Args: **kwargs: kwargs to litellm.completion() or embedding() Returns: str: The cache key generated from the arguments, or None if no cache key could be generated. z Returning preset cache key: %sz: Tz Created cache key: %spreset_cache_keyr,) !_get_preset_cache_key_from_kwargsrr'_get_relevant_args_to_use_for_cache_keyr_get_param_valuestrr3enable_caching_on_provider_specific_optional_paramsr/_get_hashed_cache_key!_add_redis_namespace_to_cache_key_set_preset_cache_key_in_kwargs) rkro cache_keyrtcombined_kwargslitellm_param_kwargsparam param_valuehashed_cache_keys r# get_cache_keyzCache.get_cache_keysX B4AAKFK  '  !CEU V# #FFH1E'-1-B-B5&-Q *CJ >vg>r-rrc\|jdg}|r|D]}||vst|cSy)Ncaching_groups)rrx)rkrrrgroups r#rzCache._get_caching_group5s9*26F)K '%'u:%(r-c|jd}|jdi}|jdi}|jdxs3t|ddxs$|jdxs|jdS)zo Handles getting the value for the 'file' param from kwargs. Used for `transcription` requests rrr file_checksumnameN file_name)rgetattr)rkrorrrs r#rzCache._get_file_param_value?s{zz&!::j"-$4b9 LL ) /tVT* /||K( /!!+.  r-c :|rd|vr|djddSy)a  Get the preset cache key from kwargs["litellm_params"] We use _get_preset_cache_keys for two reasons 1. optional params like max_tokens, get transformed for bedrock -> max_new_tokens 2. avoid doing duplicate / repeated work rrtN)r)rkros r#ruz'Cache._get_preset_cache_key_from_kwargsMs, 6)./334FMMr-rtc $|rd|vr ||dd<yyy)z Set the calculated cache key in kwargs This is used to avoid doing duplicate / repeated work Placed in kwargs["litellm_params"] rrtNr,)rkrtros r#r|z%Cache._set_preset_cache_key_in_kwargs[s* 6)?O'();<* r-c|j}|j}|j}|j}|j }|j }|j ||||}|j|}|S)zP Gets the supported kwargs for each call type and combines them )-_get_litellm_supported_chat_completion_kwargs-_get_litellm_supported_text_completion_kwargs'_get_litellm_supported_embedding_kwargs+_get_litellm_supported_transcription_kwargs$_get_litellm_supported_rerank_kwargs%_get_kwargs_to_exclude_from_cache_keyunion difference)rkchat_completion_kwargstext_completion_kwargsembedding_kwargstranscription_kwargs rerank_kwargsexclude_kwargsr~s r#rvz-Cache._get_relevant_args_to_use_for_cache_keygs"&!S!S!U!%!S!S!UGGI#OOQAAC CCE066 "     *44^Dr-cttjjj tt jj}|S)zl Get the litellm supported chat completion kwargs This follows the OpenAI API Spec )setr __annotations__keysrr )rkall_chat_completion_kwargss r#rz3Cache._get_litellm_supported_chat_completion_kwargs{sJ &) . > > C C E& %3CCHHJK L #*)r-cttjjj tt jj}|S)zl Get the litellm supported text completion kwargs This follows the OpenAI API Spec )r&TextCompletionCreateParamsNonStreamingrrr#TextCompletionCreateParamsStreaming)rkall_text_completion_kwargss r#rz3Cache._get_litellm_supported_text_completion_kwargssJ &) 2 B B G G I& %7GGLLNO P #*)r-cPttjjS)z9 Get the litellm supported rerank kwargs )rr model_fieldsrrks r#rz*Cache._get_litellm_supported_rerank_kwargss=--22455r-cPttjjS)zf Get the litellm supported embedding kwargs This follows the OpenAI API Spec )rr rrrs r#rz-Cache._get_litellm_supported_embedding_kwargss (88==?@@r-cPttjjS)zj Get the litellm supported transcription kwargs This follows the OpenAI API Spec )rr rrrs r#rz1Cache._get_litellm_supported_transcription_kwargss ,<<AACDDr-ctdgS)z> Get the kwargs to exclude from the cache key r)rrs r#rz+Cache._get_kwargs_to_exclude_from_cache_keysJ<  r-r}ctj|j}|j}t j d||S)z Get the hashed cache key for the given cache key. Use hashlib to create a sha256 hash of the cache key Args: cache_key (str): The cache key to hash. Returns: str: The hashed cache key. zHashed cache key (SHA-256): %s)hashlibsha256encode hexdigestrr)r} hash_objecthash_hexs r#rzzCache._get_hashed_cache_keys?nnY%5%5%78 ((*=xHr-rc |jdijdxs |j}|r|d|}tjd||S)a If a redis namespace is provided, add it to the cache key Args: hash_hex (str): The hashed cache key. **kwargs: Additional keyword arguments. Returns: str: The final hashed cache key with the redis namespace. rredis_namespace:zFinal hashed key: %s)rr@rr)rkrror@s r#r{z'Cache._add_redis_namespace_to_cache_keysRJJz2.223DEW #AhZ0H3X>r-c#Kd}tdt||D])}ddd||||zdigitjd+yw)Nrchoicesdelta assistant)rolecontentg{Gz?)rangelentimesleep)rkr chunk_sizeis r#generate_streaming_contentz Cache.generate_streaming_contents` q#g, 3A$/'.q1z>'B"   JJt 4sAA cached_resultmax_agecB|wt|trgd|vrc|d}tj}||z }|||kDry|jd} t|tr |St j |} |S|S#t $rtj|}Y|SwxYw)zL Common get cache logic across sync + async implementations N timestampresponse) rjdictrrjsonloadsr!ast literal_eval)rkrrr current_time response_agecached_responses r#_get_cache_logiczCache._get_cache_logics  %=$/},%k2I99;L()3L"|g'=,// ;O Dot4# " '+jj''O # " D"%"2"2?"C" " DsA<"A<<BBc  |jd i|dury|jdg}d|vr|d}n|jd i|}|m|jdi}|jd|jdtd}|jj || }|j || Sy#t$r#td tjYywxYw) a& Retrieves the cached result for the given arguments. Args: *args: args to litellm.completion() or embedding() **kwargs: kwargs to litellm.completion() or embedding() Returns: The cached result if it exists, otherwise None. TNmessagesr}r^ s-max-ages-maxageinf)rrrAn exception occurred: r,) should_use_cacherrfloatr^ get_cacherr!r$ traceback format_exc)rkrorr}cache_control_argsrrs r#rzCache.get_caches $t$$.v.d:zz*b1Hf$";/ .D..88 $%+ZZ%<",00!3!7!7 E%L!Q!% 4 4Y 4 R ,,"/- %  3I4H4H4J3KL M sB5BB55)C! C!c K |jd i|dury|jdgd|vr|d}n|jd i|}|u|jdi}|jd|jdtd}|jj |fi|d{}|j || Sy7#t$r#td tjYywxYww) zd Async get cache implementation. Used for embedding calls in async wrapper TNrr}r^rrrrrr,) rrrrr^async_get_cacherr!r$rr)rkror}rrrs r#rzCache.async_get_cache's $t$$.v.d: JJz2 &f$";/ .D..88 $%+ZZ%<",00!3!7!7 E%L!Q'Adjj&@&@&Uf&U U ,,"/- % !V  3I4H4H4J3KL M sEC0CC0B C&B?'C=C0?C)C-*C0,C--C0c  d|vr|d}n|jdi|}|t|tr|j}|j|j|d<|j dd}t|t r#|jD]\}}|dk(s ||d<tj|d}|||fStd#t$r}|d}~wwxYw)zO Common implementation across sync + async add_cache functions r}NrAr^)rrzcache key is Noner,) rrjrmodel_dump_jsonrArritemsrr!) rkresultror} _cache_kwargskv cached_dataes r#_add_cache_logiczCache._add_cache_logicDs f$";/ .D..88 $fi0#335F88'$(HHF5M & 7D 9 mT2 - 3 3 51:,-F5M!6-1IIKVL  +v55 344 G s$BC"C< C CCCc  |jdi|dury|jdd|i|\}}}|jj||fi|y#t$r+}t j dt|Yd}~yd}~wwxYw)z Adds a result to the cache. Args: *args: args to litellm.completion() or embedding() **kwargs: kwargs to litellm.completion() or embedding() Returns: None TNr#LiteLLM Cache: Excepton add_cache: r,)rrr^ set_cacher!r exceptionrxrkrror}rrs r# add_cachezCache.add_cachebs U$t$$.v.d:-BT-B-B..!'. *I{F !DJJ K B6 B U  $ $'J3q6(%S T T UsA6A B!A==BcK |jdi|dury|jdk(r(|j|j|fi|d{y|jdd|i|\}}}|j j ||fi|d{y7D7#t$r+}tjdt|Yd}~yd}~wwxYww)z3 Async implementation of add_cache TNredisrrr,) rr;rQbatch_cache_writerr^async_set_cacher!rrrxrs r#async_add_cachezCache.async_add_cachews U$t$$.v.d:yyG#(=(=(I,d,,V>v>>>1F1F1F2!2%+2. ;1djj00KR6RRR ? S U  $ $'J3q6(%S T T UshC BC 1B B BC 9B B BC BB C!C=C CC c K |jd i|dury|j|j|d<g}t|dD]\\}}|jd ii|d|i}||d<|j|}|j d d|i|\}} }|j || f^|jjd d|i|d{y7#t$r+} tjdt| Yd} ~ yd} ~ wwxYww) z Async implementation of add_cache for Embedding calls Does a bulk write, to prevent using too many clients TNrAinputr}r cache_listrr,) rrA enumeraterdatarrgr^async_set_cache_pipeliner!rrrx) rkrroridxrrtembedding_responser}rrs r#async_add_cache_pipelinezCache.async_add_cache_pipelines*  U$t$$.v.d:xx# $u J#F7O4Q#54#5#5#O8N68N7A8N#O &6{#%+[[%5"1F1F1F2-22. ;!!9k":;56$**55VVvV V V U  $ $'J3q6(%S T T UsKDC DB+C C C D C D!C;6D;DDc |jtjk(ry|jdd}t j d|||r%t |tr|jddduryy)z Returns true if we should use the cache for LLM API calls If cache is default_on then this is True If cache is default_off then this is only true when user has opted in to use cache Tr^Nz(should_use_cache: kwargs: %s; _cache: %sz use-cacheF)r<r&r'rrrrjr)rkro_caches r#rzCache.should_use_cachesd 99 ,, ,GT*GQWX j.zz+u-5r-cK|jdd|i|\}}}|jj||fi|d{y7w)Nrr,)rr^r)rkrror}rs r#rzCache.batch_cache_writesI)>)>)>)Wf)WPV)W& ;*djj**9kLVLLLs:AAAc^Kt|jd}|r|d{Sy7w)Npingrr^)rk cache_pings r#rz Cache.pings,TZZ0 #% %&s #-+-c`Kt|jd}|r||d{Sy7w)Ndelete_cache_keysr)rkrcache_delete_cache_keyss r#rzCache.delete_cache_keyss1")$**6I"J "066 67s $.,.cKt|jdr#|jjd{yy7w)N disconnect)hasattrr^rrs r#rzCache.disconnects2 4::| ,**'') ) ) - )s 4?=?cX|jr|jtjk(ryy)z Internal method to check if the cache type supports async get/set operations Only S3 Cache Does NOT support async operations FT)r;r_rdrs r#_supports_asynczCache._supports_asyncs# 99&6&9&99r-)2r)r*r+r_rcr&r'rrxrrCachingSupportedCallTypesboolr rintrprrrwrrrrur|rrvrrrrrr staticmethodrzr{rrrrrrrr rrrrrrr,r-r#r/r/?s ,<+A+A  """&#'#150404 K )-(,(,%)04)-.226.2#'!%',-E*..2)-(,0448.F_f2'(f2  f2 sm f2smf23-f2C=f2e_f2 (f2'uof2'uof2't,E'FGf26! 7f28! 9f2:! ;f2<TN=f2>E$),-?f2@"#Af2B'smCf2D#+3-Ef2F'smGf2HC=If2J#Kf2P#3-Qf2R&d^Sf2V"#Wf2X! Yf2Z!) [f2\%-SM]f2P, , \    # ?T?c?"+3C= #  D  S   Xc]  P PRV PS( *s3x * *s3x *6c#h6 ASAESXE!s3x! $#C" %}%%%N@:<U*U&"UH$M  *  r-r/r0r;r=r>r?rEc tddtjvrtjjddtjvrtjjddtj vrtj jdtj td|||||d|t_tdtj tdttj y)a Enable cache with the specified configuration. Args: type (Optional[Literal["local", "redis", "s3", "disk"]]): The type of cache to enable. Defaults to "local". host (Optional[str]): The host address of the cache server. Defaults to None. port (Optional[str]): The port number of the cache server. Defaults to None. password (Optional[str]): The password for the cache server. Defaults to None. supported_call_types (Optional[List[Literal["completion", "acompletion", "embedding", "aembedding"]]]): The supported call types for the cache. Defaults to ["completion", "acompletion", "embedding", "aembedding"]. **kwargs: Additional keyword arguments. Returns: None Raises: None zLiteLLM: Enabling Cacher^Nr;r=r>r?rEz&LiteLLM: Cache enabled, litellm.cache=LiteLLM Cache: r,) r$rrfrgrhrir^r/varsr;r=r>r?rEros r# enable_cacher$sJ+,g,,,%%g.g...  ''0g555''..w7}} !5     :7==/JKOD$7#89:r-c tdtd|||||d|t_tdtjtdt tjy)a Update the cache for LiteLLM. Args: type (Optional[Literal["local", "redis", "s3", "disk"]]): The type of cache. Defaults to "local". host (Optional[str]): The host of the cache. Defaults to None. port (Optional[str]): The port of the cache. Defaults to None. password (Optional[str]): The password for the cache. Defaults to None. supported_call_types (Optional[List[Literal["completion", "acompletion", "embedding", "aembedding"]]]): The supported call types for the cache. Defaults to ["completion", "acompletion", "embedding", "aembedding"]. **kwargs: Additional keyword arguments for the cache. Returns: None zLiteLLM: Updating Cacher z&LiteLLM: Cache Updated, litellm.cache=r!Nr,)r$r/rr^r"r#s r# update_cacher&sgF+,   1   GM:7==/JKOD$7#89:r-cpddlm}td|t5tj j dtjj dtjj dddddt_ tdtjy#1swY1xYw)aF Disable the cache used by LiteLLM. This function disables the cache used by the LiteLLM module. It removes the cache-related callbacks from the input_callback, success_callback, and _async_success_callback lists. It also sets the litellm.cache attribute to None. Parameters: None Returns: None rsuppresszLiteLLM: Disabling Cacher^Nz'LiteLLM: Cache disabled, litellm.cache=) contextlibr)r$ ValueErrorrrfremoverhrir^r(s r# disable_cacher-Js$,- * %%g.  ''0''..w7  GM;GMM?KL  s AB,,B5)rAs]  88TG+#.2!!!*6#4  T ^ ^ D(8'='=" G 7; # $7; 3-7; 3-7;sm 7; #4(A#BC 7;v(8'='=" G -; # $-; 3--; 3--;sm -; #4(A#BC -;`Mr-