g-dZddlZddlZddlmZmZmZddlmZmZm Z m Z ddl m Z ddl mZddlmZddlmZmZmZdd lmZdd lmZmZmZmZdd lmZmZd Zd Z d Z!d Z"GddeZ#y)z PagerDuty Alerting Integration Handles two types of alerts: - High LLM API Failure Rate. Configure X fails in Y seconds to trigger an alert. - High Number of Hanging LLM Requests. Configure X hangs in Y seconds to trigger an alert. N)datetime timedeltatimezone)ListLiteralOptionalUnion)verbose_logger) DualCache) SlackAlerting)AsyncHTTPHandlerget_async_httpx_clienthttpxSpecialProvider)UserAPIKeyAuth)AlertingConfigPagerDutyInternalEventPagerDutyPayloadPagerDutyRequestBody)StandardLoggingPayload&StandardLoggingPayloadErrorInformation<iXceZdZdZ ddeeeefffd ZdZ de de dede d d eee eeff d Zd eede fd ZdeedededefdZ ddeeded eefdZdedefdZxZS)PagerDutyAlertingz Tracks failed requests and hanging requests separately. If threshold is crossed for either type, triggers a PagerDuty alert. alerting_argsc ddlm}m}t|t j d}|s td||_|xsi}t|jdt|jdt|jdt|jdt |_g|_g|_|d ur"td |j$j&y) Nr)CommonProxyErrors premium_userPAGERDUTY_API_KEYzPAGERDUTY_API_KEY is not setfailure_threshold failure_threshold_window_secondshanging_threshold_seconds hanging_threshold_window_seconds)rr r!r"TzBPagerDutyAlerting is only available for LiteLLM Enterprise users. )litellm.proxy.proxy_serverrrsuper__init__osgetenv ValueErrorapi_keyrget#PAGERDUTY_DEFAULT_FAILURE_THRESHOLD2PAGERDUTY_DEFAULT_FAILURE_THRESHOLD_WINDOW_SECONDS+PAGERDUTY_DEFAULT_HANGING_THRESHOLD_SECONDS2PAGERDUTY_DEFAULT_HANGING_THRESHOLD_WINDOW_SECONDSr_failure_events_hanging_eventsnot_premium_uservalue)selfrkwargsrr_api_key __class__s ]/var/www/openai/venv/lib/python3.12/site-packages/litellm/integrations/pagerduty/pagerduty.pyr%zPagerDutyAlerting.__init__.s O 9901;< <$ %+ -;+//#%H.;->->2B.'4&7&7+-X'.;->->2B.. $>@=? t #TUfUwUwU}U}T~  $c0Ktjtj}|j d}|s t d|j dxsi}|j dxsi}|j jtd||j d|j d|j d|j d |j d |j d |j d |j d |j d|j d |jj dd} |jj dd} |j|j | | dd{y7w)z Record a failure event. Only send an alert to PagerDuty if the configured *failure* threshold is exceeded in the specified window. standard_logging_objectz9standard_logging_object is required for PagerDutyAlertingerror_informationmetadatafailed_response error_class error_code llm_provideruser_api_key_hashuser_api_key_aliasuser_api_key_org_iduser_api_key_team_iduser_api_key_user_iduser_api_key_team_aliasuser_api_key_end_user_id failure_event_type timestampr>r?error_llm_providerrArBrCrDrErFrGr rrzHigh LLM API Failure Rateeventswindow_seconds threshold alert_prefixN) rnowrutcr*r(r/appendrr!_send_alert_if_thresholds_crossed) r3r4 response_obj start_timeend_timerRstandard_logging_payload error_info_metarOrPs r7async_log_failure_eventz)PagerDutyAlerting.async_log_failure_eventWs{ ll8<<(EKZZ %F  (K  % ( ()< = C ),,Z8>B ## "#4&NN=9%>>,7#->>.#A"')),?"@#(99-A#B$)II.C$D%*YY/E%F%*YY/E%F(- 2K(L).3M)N  $++//0RTVW&&**+>B 44'')4 5   sF FFFuser_api_key_dictcachedata call_type) completiontext_completion embeddingsimage_generation moderationaudio_transcriptionpass_through_endpointrerankreturncKtjdtj|j ||yw)z Example of detecting hanging requests by waiting a given threshold. If the request didn't finish by then, we treat it as 'hanging'. z#Inside Proxy Logging Pre-call hook!) request_datar]N)r infoasyncio create_taskhanging_response_handler)r3r]r^r_r`s r7async_pre_call_hookz%PagerDutyAlerting.async_pre_call_hooksC( AB  ) )!5F *  s=?rkc2Ktjd|jjdtdt j |jjdtd{|j|d{ry|jjtdtjtjddd|j|j |j"|j$|j&|j(|j* |jjd t,}|jjd t}|j/|j||d d{y777 w) z Checks if request completed by the time 'hanging_threshold_seconds' elapses. If not, we classify it as a hanging request. z/Inside Hanging Response Handler!..sleeping for r!z secondsN)rkhanging_responseHangingRequestrHr"hanging_threshold_failsz#High Number of Hanging LLM RequestsrM)r debugrr*r-rmsleep_request_is_completedr0rTrrrRrrSr) key_aliasorg_idteam_iduser_id team_alias end_user_idr.rU)r3rkr]rOrPs r7roz*PagerDutyAlerting.hanging_response_handlers =d>P>P>T>TUps^?_>``h i mm    " "+-X    +++F F F  ## "#5",,x||4,+#3"3";";#4#>#>$5$<$<%6%>%>%6%>%>(9(D(D):)F)F  $++// . > ++// %'R 44'')> 5   K G> s7A.F0F1F F C>F F FFFrNrOrPrQc:Ktjtjt |z }|Dcgc](}|j dtj |kDs'|*}}|j|j|tjdt|d|d|t||k\rW|j|d}|dt|d |d } d |i} |j| | d {|jy y cc}w7w)z 1. Prune old events 2. If threshold is reached, build alert, send to PagerDuty 3. Clear those events )secondsrJzHave z events in the last z seconds. Threshold is ) max_errorsz: z in the last z seconds. recent_errors) alert_messagecustom_detailsN)rrRrrSrr*minclearextendr rulen_build_error_summariessend_alert_to_pagerduty) r3rNrOrPrQcutoffeprunederror_summariesrrs r7rUz3PagerDutyAlerting._send_alert_if_thresholds_crosseds!hll+i.OO#QVquu[(,,'G&'P!VQ   f CK= 4^4DD[\e[f g  v;) #"99&Q9OO.3v;-}^Drch|| d}g}|D]$}|jd|j|&|S)z Build short text summaries for the last `max_errors`. Example: "ValueError (code: 500, provider: openai)" NrJ)poprT)r3rNrrecent summariesfes r7rz(PagerDutyAlerting._build_error_summariessC % B FF;    R r8rrc jK tjd|ttj}t t |ddd||jd}|jd t|d d i d {S7#t$r"}tjd|Yd }~y d }~wwxYww)z Send [critical] Alert to PagerDuty https://developer.pagerduty.com/api-reference/YXBpOjI3NDgyNjU-pager-duty-v2-events-api zSending alert to PagerDuty: )r@criticalz LiteLLM AlertLiteLLM)summaryseveritysource componentrtrigger)payload routing_key event_actionz'https://events.pagerduty.com/v2/enqueuez Content-Typezapplication/json)urljsonheadersNz"Error sending alert to PagerDuty: ) r rurrLoggingCallbackrrr)postdict Exception exception)r3rr async_clientrrs r7rz)PagerDutyAlerting.send_alert_to_pagerdutys  O  #? !O P-C1AA.L-A()'*'#1 !LL& -G&**=']');<+   O  $ $'I!%M N N OsAB3A:B>B?BB3B B0B+&B3+B00B3)N)r)__name__ __module__ __qualname____doc__rr rrr%r\rr rrstrrprorrintrUrr __classcell__)r6s@r7rr(s FJ%%eND,@&AB%R/ b)      % 3,- .84 $TN4 ?M4 p%+,%% %  %PGH 12 @C  $ % O3OPTOr8r)$rrmr&rrrtypingrrrr litellm._loggingr litellm.cachingr 1litellm.integrations.SlackAlerting.slack_alertingr &litellm.llms.custom_httpx.http_handlerr rrlitellm.proxy._typesr$litellm.types.integrations.pagerdutyrrrrlitellm.types.utilsrrr+r,r-r.rr8r7rso 2211+%K 0  ')#572.0+582GO GOr8