ge+ dZddlZddlmZmZmZmZmZddlZddl m Z ddl m Z ddl mZmZerddlmZdd lmZeZeZneZeZd Zd Zd ed eedeeefdedef dZd ed edeeefdedef dZ dd ededeeefd eedeedef dZd edeedeefdZ d edeedee!fdZ"d edeedeefdZ#d ed ededefdZ$d edefdZ%deeefdefdZ&y)a Router cooldown handlers - _set_cooldown_deployments: puts a deployment in the cooldown list - get_cooldown_deployments: returns the list of deployments in the cooldown list - async_get_cooldown_deployments: ASYNC: returns the list of deployments in the cooldown list N) TYPE_CHECKINGAnyListOptionalUnion)verbose_router_logger)router_cooldown_event_callback)*get_deployment_failures_for_current_minute+get_deployment_successes_for_current_minute)Span)Routerg?litellm_router_instance deploymentexception_statusoriginal_exceptionreturnc~|jry|y|j||t|sy||jvryy)a Helper that decides if cooldown logic should be run Returns False if cooldown logic should not be run Does not run cooldown logic when: - router.disable_cooldowns is True - deployment is None - _is_cooldown_required() returns False - deployment is in litellm_router_instance.provider_default_deployment_ids - exception_status is not one that should be immediately retried (e.g. 401) F)model_idr exception_strT)disable_cooldowns_is_cooldown_requiredstrprovider_default_deployment_ids)rrrrs [/var/www/openai/venv/lib/python3.12/site-packages/litellm/router_utils/cooldown_handlers.py_should_run_cooldown_logicr%sU"00 " 8 8),- 9  ,LLL ch|jt|durt||}t||}||z}d}|dkDr|||zz }t j d||||t |}|dk(ry|d k(ry|tkDrytjt | duryyt||| S) a_ Helper that decides if a deployment should be put in cooldown Returns True if the deployment should be put in cooldown Returns False if the deployment should not be put in cooldown Deployment is put in cooldown when: - v2 logic (Current): cooldown if: - got a 429 error from LLM API - if %fails/%(successes + fails) > ALLOWED_FAILURE_RATE_PER_MINUTE - got 401 Auth error, 404 NotFounder - checked by litellm._should_retry() - v1 logic (Legacy): if allowed fails or allowed fail policy set, coolsdown if num fails in this minute > allowed fails rF)r deployment_idgrzYpercent fails for deployment = %s, percent fails = %s, num successes = %s, num fails = %siTr ) status_code)rrr) allowed_fails_policy_is_allowed_fails_set_on_routerr r rdebugcast_exception_status_to_int!DEFAULT_FAILURE_THRESHOLD_PERCENTlitellm _should_retry-should_cooldown_based_on_allowed_fails_policy) rrrrnum_successes_this_minutenum_fails_this_minutetotal_requests_this_minute percent_failsexception_status_ints r_should_cooldown_deploymentr0Is 2 44< +$;    %P$;:% !!K$;:! &?AV%V" % )1),AAM ## g   % !   < >  ! !89IJ   <$;!1  rtime_to_cooldownc:t||||dus|yt|}tjd|d|jxsd}||}t ||||rB|j j||||tjt||||yy)aL Add a model to the list of models being cooled down for that minute, if it exceeds the allowed fails / minute or the exception is not one that should be immediately retried (e.g. 401) Returns: - True if the deployment should be put in cooldown - False if the deployment should not be put in cooldown FzAttempting to add z to cooldown listr )rrr cooldown_time)rr!rr3T) rr&rr%r3r0cooldown_cacheadd_deployment_to_cooldownasyncio create_taskr )rrrrr1r/r3s r_set_cooldown_deploymentsr8s& # #Z1ACU     78HI"4ZL@Q RS+99>QM#( "-=?Q ..II11' J   *(?(!1+    rparent_otel_spancDK|j}|jj||d{}g}|Ct|tr3t |dkDr%t|dt r|Dcgc]}|d }}tjd||S7ecc}ww)= Async implementation of '_get_cooldown_deployments'  model_idsr9Nrretrieve cooldown models: ) get_model_idsr4async_get_active_cooldowns isinstancelistlentuplerr%rr9r=cooldown_modelscached_value_deployment_idscvs r_async_get_cooldown_deploymentsrIs(557I%44OO-P  #%#  -  1 $ q)5 17F&Gr!u#&G"<_)r?r4r@rr%)rr9r=rFs r/_async_get_cooldown_deployments_with_debug_inforKsd(557I%44OO2BP   "<_RO#%#  -  1 $ q)5 17F&Gr!u#&G &&'Hs' A7c|j|xs |j}|jxst}|jj |xsd}|dz}||kDry|jj |||y)z Check if fails are within the allowed limit and update the number of fails. Returns: - True if fails exceed the allowed limit (should cooldown) - False if fails are within the allowed limit (should not cooldown) ) exception)keyrr T)rQvaluettlF)get_allowed_fails_from_policy allowed_failsr3DEFAULT_COOLDOWN_TIME_SECONDS failed_calls get_cache set_cache)rrrrUr3 current_fails updated_failss rr*r*s ==( >  1 # 0 0  --N1N,88BBzBRWVWM!A%M}$,,66-] 7  rcZ|jy|jtjk7ryy)z Check if Router.allowed_fails is set or is Non-default Value Returns: - True if Router.allowed_fails is set or is Non-default Value - False if Router.allowed_fails is None or is Default Value FT)rUr(r s rr$r$@s-,,4,,0E0EE rct|tr t|}|S|S#t$rt j d|dd}Y|SwxYw)Nz'Unable to cast exception status to int z. Defaulting to status=500.i)rArint Exceptionrr%)rs rr&r&Qsh"C( #"#34     # ! ' '9:J9KKfg  #    #s !$A A )NN)'__doc__r6typingrrrrrr(litellm._loggingr'litellm.router_utils.cooldown_callbacksr )router_callbacks.track_deployment_metricsr r opentelemetry.tracer _Spanlitellm.routerr_Router LitellmRouterr'rVrr^boolrr0floatr8rIrDrKrNr*r$r&rrrms<<2R 10M DM D"!"!*! !CHo! !  !HM*MMCHoM M  Mh!%(, 6*66CHo6 6 uo 6  6r'*'tn' #Y'8*tn %[$'*'>Ftn' #Y': *    F* " 5c? s r