g 4ddlmZddlmZmZmZmZmZmZm Z ddl Z ddl m Z ddl mZddlmZer ddlmZeZneZded ed efd Zd eeded eeeeeeffd Zdeededeededededed efdZdedeedeededed ef dZdededefdZdededefdZd eeed efdZ d e eeeeeeffdefdZ!y))Enum) TYPE_CHECKINGAnyDictListOptionalTupleUnionN)verbose_router_logger) CustomLogger)LiteLLMParamsTypedDict)Router model_group fallback_keyreturnctjD]R}t|tr |j}n|}|j |ds7|j |dd}||k(sRyy)z Handles wildcard routing scenario where fallbacks set like: [{"gpt-3.5-turbo": ["claude-3-haiku"]}] but model_group is like: "openai/gpt-3.5-turbo" Returns: - True if the stripped model group == fallback_key /TF)litellm provider_list isinstancervalue startswithreplace)rrprovider _providerstripped_model_groups a/var/www/openai/venv/lib/python3.12/site-packages/litellm/router_utils/fallback_event_handlers.py_check_stripped_model_grouprsi)) h % I I  ! !YKq/ 2#.#6#6)A#K #|3*  fallbackscd}d}d}t|D]\}}t|trt|j d|k(r||}nt |t|j dr |t|j d}t|j ddk(s|}t|t s|j|g}|||}||fS|||d}||fS)a) Returns: - fallback_model_group: List[str] of fallback model groups. example: ["gpt-4", "gpt-3.5-turbo"] - generic_fallback_idx: int of the index of the generic fallback in the fallbacks list. Checks: - exact match - stripped model group match - generic fallback Nr)rr*) enumeraterdictlistkeysrstrpop)r!rgeneric_fallback_idxstripped_model_fallbackfallback_model_groupidxitems rget_fallback_model_groupr/*s+/3704y) T dD !DIIK #{2'+K'8$,'d499;6G6J+/tDIIK/@/C*D'diik"1%,'*$ c "$-MM#$6#7 *# " .#:  !5 55" -#,-A#B3#G !5 55r argslitellm_routerr,original_model_grouporiginal_exception max_fallbacksfallback_depthcxK||k\r||}|D]} | |k(r |j||}tjd| t| tr| |d<n!t| t r|j | |jdij d|jddi|dz|d<||d <|j|i|d{} tjd t||| d{| cS|767 #t$r$} | }t||| d{7Yd} ~ $d} ~ wwxYww) a Loops through all the fallback model groups and calls kwargs["original_function"] with the arguments and keyword arguments provided. If the call is successful, it logs the success and returns the response. If the call fails, it logs the failure and continues to the next fallback model group. If all fallback model groups fail, it raises the most recent exception. Args: litellm_router: The litellm router instance. *args: Positional arguments. fallback_model_group: List[str] of fallback model groups. example: ["gpt-4", "gpt-3.5-turbo"] original_model_group: The original model group. example: "gpt-3.5-turbo" original_exception: The original exception. **kwargs: Keyword arguments. Returns: The response from the successful fallback model group. Raises: The most recent exception if all fallback model groups fail. kwargseFalling back to model_group = modelmetadatarNr5r4Successful fallback b/w models.r2r8r3) log_retryr inforr(r%update setdefaultgetasync_function_with_fallbackslog_success_fallback_event Exceptionlog_failure_fallback_event) r1r,r2r3r4r5r0r8error_from_fallbacksmgresponser9s rrun_async_fallbackrLRsr@&  -" % %  #--V?Q-RF ! & &)Gt'L M"c""$wB% b!   j" - 4 4 7D 9: (6'9F# $&3F? #I^IIH " & &'H I,%9#5   O5#D %    #$ ,%9#5   s_D:B7D D*D :D;D D:D D  D7D2&D)'D2,D:2D77D:cP|}|D]}||k(r |j||}tjd|||d<|jdij d|i|j |i|}tjd|cS|#t $r } | }Yd} ~ d} ~ wwxYw)a Synchronous version of run_async_fallback. Loops through all the fallback model groups and calls kwargs["original_function"] with the arguments and keyword arguments provided. If the call is successful, returns the response. If the call fails, continues to the next fallback model group. If all fallback model groups fail, it raises the most recent exception. Args: litellm_router: The litellm router instance. *args: Positional arguments. fallback_model_group: List[str] of fallback model groups. example: ["gpt-4", "gpt-3.5-turbo"] original_model_group: The original model group. example: "gpt-3.5-turbo" original_exception: The original exception. **kwargs: Keyword arguments. Returns: The response from the successful fallback model group. Raises: The most recent exception if all fallback model groups fail. r7r:r;r<rr>N)r@r rArCrBfunction_with_fallbacksrG) r1r,r2r3r0r8rIrJrKr9s rrun_sync_fallbackrOs:." % %  %#--V?Q-RF ! & &)Gt'L M F7O   j" - 4 4# >~==tNvNH ! & &'H IO#  %#$  %sA;B B%B  B%r8cKddlm}tjD]}t |t s|tj vs& d}|tj vr ||dd}n,t |t r|}ntj|ds|tj|d|j|||d{y7#t$r+}tjdt|Yd}~d}~wwxYww) a Log a successful fallback event to all registered callbacks. This function iterates through all callbacks, initializing _known_custom_logger_compatible_callbacks if needed, and calls the log_success_fallback_event method on CustomLogger instances. Args: original_model_group (str): The original model group before fallback. kwargs (dict): kwargs for the request Note: Errors during logging are caught and reported but do not interrupt the process. r$_init_custom_logger_compatible_classNlogging_integration llm_routerinternal_usage_cache( logger not found / initialized properlyz: logger not found / initialized properly, callback is Noner?z%Error in log_success_fallback_event: ) *litellm.litellm_core_utils.litellm_loggingrRr callbacksrr )_known_custom_logger_compatible_callbacksr exceptionrFrGerrorr(r2r8r3rR _callback_callback_custom_loggerr9s rrFrFs && i . JJ J BF' Q QQ.R,5#'-1/+  <8.7+)33$+%MN*2)33$+%_`-HH)=!'9I5'4  %++;CF8D ZC?CC?C C<!C72C?7C<<C?cKddlm}tjD]}t |t s|tj vs& d}|tj vr ||dd}n,t |t r|}ntj|ds|tj|d|j|||d{y7#t$r+}tjdt|Yd}~d}~wwxYww)a Log a failed fallback event to all registered callbacks. This function iterates through all callbacks, initializing _known_custom_logger_compatible_callbacks if needed, and calls the log_failure_fallback_event method on CustomLogger instances. Args: original_model_group (str): The original model group before fallback. kwargs (dict): kwargs for the request Note: Errors during logging are caught and reported but do not interrupt the process. rrQNrSrWr?z%Error in log_failure_fallback_event: ) rXrRrrYrr rZr r[rHrGr\r(r]s rrHrHs && i . JJ J BF' Q QQ.R,5#'-1/+  <8.7+)33$+%MN*2)33$+%MN-HH)=!'9I5'4  %++;CF8D r`c|t|trt|dk(rytd|Drytd|Dr:tj j D]}||dj vsyy)ar Checks if the fallbacks list is a list of strings or a list of dictionaries. If - List[str]: e.g. ["claude-3-haiku", "openai/o-1"] - List[Dict[, Any]]: e.g. [{"model": "claude-3-haiku", "messages": [{"role": "user", "content": "Hey, how's it going?"}]}] If [{"gpt-3.5-turbo": ["claude-3-haiku"]}] then standard format. rFc3<K|]}t|tywN)rr(.0r.s r z6_check_non_standard_fallback_format..Ks 7YT:dC YTc3<K|]}t|tywrd)rr%res rrgz6_check_non_standard_fallback_format..Ms : Zd # rh)rr&lenallr __annotations__r')r!keys r#_check_non_standard_fallback_formatrn?sx 9d ;s9~QR?R 7Y 77 : : :)99>>@Cil''))A r cyrd)r!rs r run_non_standard_fallback_formatrqUs r )"enumrtypingrrrrrr r rlitellm._loggingr "litellm.integrations.custom_loggerr litellm.types.routerr litellm.routerr_Router LitellmRouterr(boolrintr/rGrLrOr%rFrHrnrqrpr rr|sIII2;70MMS2%6Cy%6'*%6 8DI  -.%6PG :G!Gs)G G " G  GG GT.!. :.s). . " . .b66'+6AJ6r66'+6AJ6r8DI3F4, T#YT#s(^ 445 DG r