g(" ddlmZmZmZddlmZmZddlZddlm Z ddl m Z ddl m Zgdddddddddddddddd fd ed ed eed eedeedeedeedeedeedeedeedeedeedeedeedeef dZdZdZy))FIRST_COMPLETEDThreadPoolExecutorwait)ListOptionalN) print_verbose)get_optional_params)handleriXdmodelmessages functions function_call temperaturetop_pnstream max_tokenspresence_penaltyfrequency_penalty logit_biasuserrequest_timeouttimeout max_workersc Tt}|}g}|}d}|jdddtjvr*|jddd}|jddd}|dk(rFt ||||||xsd|| | | | | ||}t j ||tj|}|Sd }t| 5}||d D]z}|D]s}|j}|jd ||d <i}d|vr|jd}|jtjfi||}|j|u| dddg}|D]"} |j|j$|S#1swY4xYw#t$r} |j| Yd} ~ Rd} ~ wwxYw)a  Batch litellm.completion function for a given model. Args: model (str): The model to use for generating completions. messages (List, optional): List of messages to use as input for generating completions. Defaults to []. functions (List, optional): List of functions to use as input for generating completions. Defaults to []. function_call (str, optional): The function call to use as input for generating completions. Defaults to "". temperature (float, optional): The temperature parameter for generating completions. Defaults to None. top_p (float, optional): The top-p parameter for generating completions. Defaults to None. n (int, optional): The number of completions to generate. Defaults to None. stream (bool, optional): Whether to stream completions or not. Defaults to None. stop (optional): The stop parameter for generating completions. Defaults to None. max_tokens (float, optional): The maximum number of tokens to generate. Defaults to None. presence_penalty (float, optional): The presence penalty for generating completions. Defaults to None. frequency_penalty (float, optional): The frequency penalty for generating completions. Defaults to None. logit_bias (dict, optional): The logit bias for generating completions. Defaults to {}. user (str, optional): The user string for generating completions. Defaults to "". deployment_id (optional): The deployment ID for generating completions. Defaults to None. request_timeout (int, optional): The request timeout for generating completions. Defaults to None. max_workers (int,optional): The maximum number of threads to use for parallel processing. Returns: list: A list of completion results. N/rvllmF)rrrrrrstoprrrrrr custom_llm_provider)r rcustom_prompt_dictoptional_paramsc3VKtdt||D] }||||zyw)z)Yield successive n-sized chunks from lst.rN)rangelen)lstris R/var/www/openai/venv/lib/python3.12/site-packages/litellm/batch_completion/main.pychunksz batch_completion..chunks_s-1c#h*!a!en$+s')rr rrkwargs)localssplitlitellm provider_listr vllm_handlerbatch_completionsr#rcopypopsubmit completionappendresult Exception)!r rrrrrrrr!rrrrr deployment_idrrrr-argsbatch_messages completionsr"r$resultsr+executor sub_batch message_listkwargs_modifiedoriginal_kwargsfutureexcs! r*batch_completionrG s` 8DNK E {{31!6!66#kk#q1!4 C#A&f$-'#?U!-/! 3 "00#&99+  L N; %  K 8H#NC8 $-L&*iikO#'' 62>OJ/&(O?2*9*=*=h*G,X__**.=APF &&v.%.99"!F $v}}/" N19 8* $s## $s%z)batch_completion_models..sv||AaD/A)keyN deployments model_listr-z waiting for next result ) return_whenz done list z3 got an exception, ignoring, removing from futuresz removing keyz new futuresz done looping through futures )r5rr'r6r0r7sorteditemsr9keysrrvaluesrr:)r<r-futuresr@r rErP nested_kwargs deploymentrOdone_r9 new_futuresvaluerIs @r*batch_completion_modelsr^s&& 76! 8 CK 8H!0&&")-"5:">D" "( %A" v==?.!==?*9 8 " 9t _ & ]+  =! <  8R0  C ,< =) !;;=C:-*0+ 3 ) 9J8-8/>x&&0*00 7+,*@Aw~~/_Ma D623"F!!'% )> ="#*DEg&7>R u9t )%!%S&g.&( *1--/JC% - SE.B C (38 C 0 +: #.% G9&=> !+>R sWA%H.H H)I8A;I8HI8 I8H A"I52I84I55I88Jcddl}d|vr|jdd|vr|d}|jdn tdg}|jj t |5}t |D][\}}|jtjg|d|i|}|j=|j|j] ddd|S#1swY|SxYw)a Send a request to multiple language models concurrently and return a list of responses from all models that respond. Args: *args: Variable-length positional arguments passed to the completion function. **kwargs: Additional keyword arguments: - models (str or list of str): The language models to send requests to. - Other keyword arguments to be passed to the completion function. Returns: list: A list of responses from the language models that responded. Note: This function utilizes a ThreadPoolExecutor to parallelize requests to multiple models. It sends requests concurrently and collects responses from all models that respond. rNr rIz'models' param not in kwargsr,) concurrent.futuresr5r:rWrr' enumerater6r0r7r9r8) r<r- concurrentrI responsesr@idxr rEs r*%batch_completion_models_all_responsesres$& 76! 8677I    . .3v; . G8#F+JC$X__W%7%7V$VeVvVF}}*  1, H  H s'AC0!CC&)r`rrrtypingrrr0litellm._loggingr litellm.utilsr llms.vllm.completionr r2strfloatintbooldictrGr^rerNr*rpsHHH!*-:  $#'#'!! $(,)-!%%) !$'q qq~ q C= q % q E?q}q TNq quoq qq 3-q"c]#q$c]%q&#'qlSl&rN