g~^ddlZddlZddlZddlZddlmZddlZddlZddlm Z m Z m Z m Z m Z mZmZddlmZmZddlmZej*eZdZddZddZd Zgd Zd gd d gd d gd ZGddZ ddZddZ y)N)Dict)AttributeProto GraphProto ModelProto NodeProto TensorProtohelper numpy_helper) infer_shapesinfer_shapes_path)versionc |Dcgc]8}tt|jdddjdd:c}Scc}w)z| Convert numpy float16 to python int. :param np_list: numpy float16 list :return int_list: python int list HN)intbinviewzfill)np_list_s U/var/www/openai/venv/lib/python3.12/site-packages/onnxruntime/transformers/float16.py_npfloat16_to_intrsC=D DGqCAFF3K $**2. 2G DD Ds=Ac Rd}|tj|dkDjddkDr|tj|dkDj}|tj|dkDj }||k\rt j d|d|||krt j d|d||tj|dkjddkDr|tj|dkj}|tj|dkj }|| krt j d|d| || k\rt j d|d| tj|d||||}tj|| |d| |}tj|||td||}tj|td|| | |}tj|S)a? Convert float32 numpy array to float16 without changing sign or finiteness. Positive values less than min_positive_val are mapped to min_positive_val. Positive finite values greater than max_finite_val are mapped to max_finite_val. Similar for negative values. NaN, 0, inf, and -inf are unchanged. c:tj||k||kSN)np logical_and)abcs rbetweenz&convert_np_to_float16..between0s~~a!eQU++rzthe float32 number z will be truncated to infz-inf) rwhereshapemaxminloggerdebugfloatfloat16)np_arraymin_positive_valmax_finite_valr" positive_max positive_min negative_max negative_mins rconvert_np_to_float16r4(s ,A&'--a014A 67;;= A 67;;= > ) LL.|n@PRZ[Hxx!1 18Q?BRART\]Hxx%,GYabHxxf x.IN?\deH ::h r#ct|tstdt||jtj k(rtj |_|jrSttj|j||}t|}||jddg|jdd|jrCtj|jd}t|||}|j|_ |S)aConvert tensor float to float16. Args: tensor (TensorProto): the tensor to convert. min_positive_val (float, optional): minimal positive value. Defaults to 1e-7. max_finite_val (float, optional): maximal finite value. Defaults to 1e4. Raises: ValueError: input type is not TensorProto. Returns: TensorProto: the converted tensor. 3Expected input type is an ONNX TensorProto but got Nfloat32dtype) isinstancer ValueErrortype data_typeFLOATFLOAT16 float_datar4rarrayr int32_dataraw_data frombuffertobytes)tensorr.r/ float16_dataint_list float32_list float16_lists rconvert_tensor_float_to_float16rKJs fk *NtTZ|n]^^ ;,,,&..   0&:K:K1LN^`noL(6H#+F  a #%F  a ??== JL0?OQ_`L*224FO Mr#ctj|j}tj|j |j |Sr)r to_arrayr&r make_tensor_value_infonamer=)rFr&s rmake_value_info_from_tensorrPos7  ! !& ) / /E  ( (f6F6F NNr#)ArrayFeatureExtractor BinarizerCastMapCategoryMapperDictVectorizerFeatureVectorizerImputer LabelEncoderLinearClassifierLinearRegressor Normalizer OneHotEncoderRandomUniformLike SVMClassifier SVMRegressorScalerTreeEnsembleClassifierTreeEnsembleRegressorZipMapNonMaxSuppressionTopKRoiAlignRangeCumSumMinMaxUpsampler)Resize GroupNorm SkipGroupNormc(eZdZdZdefdZdefdZy)InitializerTrackerz'Class for keeping track of initializer. initializerc.||_g|_g|_yr)rr fp32_nodes fp16_nodes)selfrrs r__init__zInitializerTracker.__init__s&r#nodecv|r|jj|y|jj|yr)rtappendru)rvrxis_node_blockeds radd_nodezInitializerTracker.add_nodes)  OO " "4 ( OO " "4 (r#N)__name__ __module__ __qualname____doc__rrwrr|r#rrqrqs1K )Y)r#rqc " |dk\sJd|ttjtjjksJd|in|} t |t r|} tjtjtjdk\rn|sltjtjj| 5} | j } t#| | tj$| }d}dddntj$| }t |t&st)dt+|d}|sAtjtjtjd k\r t,} |t.}|g}t1|}t1|}t2j5d |d |d |d |d|d|d|g}g}g}g}|||}|j7|i}t1}t1}|j8j:Dcgc]@}|j*j<j>t@jBk(s5|j B}}|j8jDDcgc]@}|j*j<j>t@jBk(s5|j B}}t |tFr)|Dcgc] }||vs| }}|Dcgc] }||vs| }}n|sg}g}tI|j8j:D]<\}}|j |vsdt |z}|||j <|jK|j dt |z}|j8jLjK}|jO|||_t@jP|j*j<_tSjTd|j g|gt@jP|g}|j8jVjY||j7||jK|?tI|j8jDD].\}}|j |vsdt |z}|||j <|jK|j dt |z}|j8jLjK}|jO|||_t@jP|j*j<_tSjTd|g|j gd|g}|j8jVjY||j7||jK|1i}|rg} |D]}!t |!t&r| j7|!j8t |!tZr>|!j\D]H}|j^t@jBk(s!|j |vsJta|||j <J|!jVD]}|j |vrtcte|j:D]3}|j:||vs||j:||j:|<5tcte|jDD]3}|jD||vs||jD||jD|<5|jf|vxs|j |v}"tI|j:D]e\}}||vs |"xsB|thjk|jfgvxr|| jk|jfgv}#||jm||#g|"r|j7|k|jfdk(rU|jnD]F}$|$j dk(s|$jpt@jBk(s1t@jP|$_8n|jfdvrd}%|jnD]G}$|$j dk(sd}%|$jpt@jBk(s3t@jP|$_8I|jfdvr@|%s>|jnjYtSjrdt@jPg|jfthvs|jf| vr$|jnD]}$| j7|$|j7|t |!ttr| j7|!jv|!jxD]}| j7||!jzjOt}|!jz|||!j~D]}t}|||}t |!tZs+tj|!j:|!jD|!jLD];}|j*j<j>t@jBk(rH|j |vr:t@jP|j*j<_|j7||j*jds|j*jj>j<j>t@jBk(s|j |vst@jP|j*jj>j<_|j7|>| }|r|jD]}&|s |&jst}|&j\|||&_.|j7t|&j\|&js_|rbt2jd|&j|D]\}'tI|'j:D]@\}}|th|'jfvs|| jk|'jfgvr;|D]}(||(j k(s|j8jLjK}|jO|(|'j dzt |z}||_t@jB|j*j<_|'j d zt |z}tSjTd|g|gd|g}|j8jVjY|||'j:|<@C_| rt@jnt@jB})|D]Y}'tcte|'j:D] }|'j:|}|D]}(||(j k(s|j8jLjK}|jO|(|'j dzt |z}||_|)|j*j<_|'j d zt |z}tSjTd|g|g|)|g}|j8jVjY|||'j:|<  tcte|'jDD] }|'jD|}*|D]}(|*|(j k(s|j8jLjK}|jO|(|'j d!zt |z}||_|)|j*j<_|'j d"zt |z}tSjTd|g|*gd#|g}|j8jVjY|||'jD|<  \|S#1swYxYw#wxYwcc}wcc}wcc}wcc}w)$aConvert tensor float type in the input ONNX model to tensor float16. Args: model (ModelProto or str): The ONNX model or path of the model to convert. min_positive_val (float, optional): minimal positive value. Defaults to 5.96e-08. max_finite_val (float, optional): maximal finite value of float16. Defaults to 65504. keep_io_types (Union[bool, List[str]], optional): It could be boolean or a list of float32 input/output names. If True, model inputs/outputs should be left as float32. Defaults to False. disable_shape_infer (bool, optional): Skips running onnx shape/type inference. Useful if shape inference has been done. Defaults to False. op_block_list (List[str], optional): List of op types to leave as float32. Defaults to None, which will use `float16.DEFAULT_OP_BLOCK_LIST`. node_block_list (List[str], optional): List of node names to leave as float32. Defaults to None. force_fp16_initializers(bool): force converting all float initializers to float16. Default to false, which will convert only the one needed to avoid precision loss. force_fp16_inputs(Dict[str, List[int]]): Force the conversion of the inputs of some operators to float16, even if this script's preference it to keep them in float32. Raises: ValueError: input type is not ModelProto. Returns: ModelProto: converted model. "\o>zginvalid min_positive_val. smallest positive float16 value: subnormal 5.96e-08, and normalized 6.104e-05z4invalid max_finite_val. largest float16 value: 65504Nz1.8.0)dirTz$Expected an ONNX ModelProto but got z1.2.0z"fp16 parameters: min_positive_val=z max_finite_val=z keep_io_types=z disable_shape_infer=z op_block_list=z node_block_list=z force_fp16_initializers=graph_input_cast_graph_input_castCast)torOgraph_output_cast_graph_output_castrlr)EyeLike Multinomial RandomNormalRandomNormalLike RandomUniformr] SequenceEmpty BernoulliFr9)rrr sequence_typezXinitializer is used by both fp32 and fp16 nodes. Consider add these nodes to block list: _input_cast_ _input_cast _output_cast_ _output_cast )Jr+rfinfor,r'r:strr parseonnx __version__tempfileNamedTemporaryFileospathdirnamerOr loadrr;r<r DEFAULT_OP_BLOCK_LISTsetr)r*rzgraphinput tensor_type elem_typerr>outputlist enumerateadd value_infoCopyFromr?r make_noderxextendrrrr=rqrangelenop_typeALWAYS_FLOAT_INPUTSgetr| attributeimake_attributerggraphstrKtensors itertoolschainHasFieldrvaluesrurPrtinfoBFLOAT16)+modelr.r/ keep_io_typesdisable_shape_infer op_block_listnode_block_listforce_fp16_initializersforce_fp16_inputs#use_bfloat16_as_blocked_nodes_dtypeforce_fp16_inputs_dict model_pathtmpfileshape_infer_model_pathfunc_infer_shapequeuevalue_info_list node_listmixed_float_type_node_list name_mappinggraph_io_to_skipio_castsn fp32_inputs fp32_outputsr output_name node_namenew_value_infonew_node input_namefp32_initializers next_levelqr{use_fp32_weightattr has_dtypevaluerxr accuracy_typers+ rconvert_float_to_float16rs J H$qpq$ U288BJJ#7#;#;< <t>tt <#4#>N~N^^mn{m|}RSfRggvwDvEEVWfVgg@AX@Y Z EOI "$# ' LLLuuH#(;;#4#4j#4a8J8J8T8TXcXiXi8i166#4Kj$)KK$6$6l$6q!&&:L:L:V:VZeZkZk:kAFF$6Ll-&"-D+Qm1Cq+ D#/F 2 LL #-"%++,,-1 66\ !-A6J#-L   (+c!f4I"[[33779N  # #A &",N 8C8K8KN   + + 5((*xAT]^_H KK   # #H -  " "> 2 LL #."8:  A!Z(!!!''*!Z(A{{k&7&77 vv->>>>4Fq4I)!&&1' Avv) "3qww<0771:5)5aggaj)AAGGAJ1#3qxx=188A;,6*6qxx{*CAHHQK2'(ii=&@&]AFFoD]O)2177); :%)::.=/ !%8%<% > J J T / 6 6q 9Jk@E H#))+ "e&6&6 ?@Q@QSces tE   " "#>u?P?P#Q R(? notoooAB ,+&tzz2MAz+DLL99QBXB\B\]a]i]ikmBn=n- 0%*[[%;%;%?%?%AN"++J7"&))n"r@rrArCrD RuntimeErrorr4amaxabsr7)rFr.r/ float32_datarGs rfloat_to_float16_max_diffrs fk *NtTZ|n]^^ ;,,,>??L xx 1 12  }}V__IF 677(7GXL 77266,L)AAB CCr#)r@) rrFFNNFNF)!rloggingrrtypingrnumpyrrrrrrrr r onnx.shape_inferencer r packagingr getLoggerr}r)rr4rKrPrrrqrrrr#rrs  eee@   8 $E D"JO B#$Aq6QPQFS ) )"!(-yx Dr#