g`tddlZddlZddlZddlZddlmZddlmZmZm Z m Z ddl Z ddl m Z ddlmZmZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lm Z ddl!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.m/Z/ddl0Z0ejbe2Z3ideddfdeddfdeddfdeddfdeddfdeddfde ddfd e ddfd!eddfd"e"ddfd#e$ddfd$eddfd%e(ddfd&e&dd'fd(e*ddfd)e,ddfd*eddfZ4dd+dd,gd+d+d-d.f dd/d0e e e5efd1e6d2e e5d3e e7d4ee5d5e6d6e6d7e5d8e7d9e e5d:e5fd;Z8 dJde7d?e7d@e e d:ef dAZ9 dKdd/dBe e5efd=e5d>e7d?e7d@e e d3e e7d1e6dCe6d5e6d9e e5d:efdDZ:d2e5d:ee5e7ffdEZ;dFZe2dIk(re>yy)LN)Path)DictListOptionalUnion) FusionOptions) ModelProto load_model) OnnxModel) BartOnnxModel) BertOnnxModel)BertOnnxModelKeras)BertOnnxModelTF) ClipOnnxModel)ConformerOnnxModel) Gpt2OnnxModel) PhiOnnxModel) Sam2OnnxModel) T5OnnxModel) TnlrOnnxModel) UnetOnnxModel) VaeOnnxModel)extract_raw_data_from_modelhas_external_databartpytorchbertbert_tftf2onnx bert_keras keras2onnxclip conformergpt2gpt2_tfgpt_neoxphisam2swintnlrt5unetvaevitFci)provider onnx_modeluse_gpuoptimized_model_path opt_leveldisabled_optimizersverbosesave_as_external_dataexternal_data_filenameexternal_data_file_thresholdr3returnc  |dvsJddlm} || jdd}|J|rG| Ett j j gdrtjd|St|trtt|d n t|} | jr|stjd t j} |d k(r tj j"| _nD|d k(r tj j&| _ntj j(| _|Mt|tr$tt+|j-d }nd}dj/|||rdnd}|| _|r]t3|dk(r"t4j6j9|dz}| j;d|| j;dt||rt=dd| _i}|r||d<|sdg}n| D| dk(rdg}n)| dk(rdg}n | dk(rddg}n| dk(rdg}n | d k(rd!dg}ndg}|jAdnBg}| jBr#|jAd|jAdn|jAdt|tDrHtG|r tId"tK|\}}| jMtO|tO|t|tDr|jQn|}t jR|| fd#|i|t4j6jU|rt4j6jW|sJtjYd$||S)%a Use onnxruntime to optimize model. Args: onnx_model (str | ModelProto): the path of input onnx model or ModelProto. use_gpu (bool): whether the optimized model is targeted to run in GPU. optimized_model_path (str or None): the path of optimized model. opt_level (int): graph optimization level. disabled_optimizers (List[str]): a list of names of disabled optimizers save_as_external_data (bool): whether to save external data outside of ONNX model external_data_filename (str): name of external data file. If not provided, name is automatically created from ONNX model. external_data_file_threshold (int): threshold to decide whether to save tensor in ONNX model or in external data file provider (str or None): execution provider to use if use_gpu Returns: optimized_model_path (str): the path of optimized model )rr-r1r)versionNonnx_model_path)CUDAExecutionProviderROCMExecutionProviderMIGraphXExecutionProviderz3There is no gpu for onnxruntime to do optimization.Fload_external_dataabThis model uses float16 in the graph, use_gpu=False might cause extra Cast nodes. Most operators have no float16 implementation in CPU, so Cast nodes are added to compute them in float32. If the model is intended to use in GPU, please set use_gpu=True. Otherwise, consider exporting onnx in float32 and optional int8 quantization for better performance. rr-r2optimized_modelz{}_o{}_{}.onnxgpucpuz.dataz7session.optimized_model_external_initializers_file_namez?session.optimized_model_external_initializers_min_size_in_bytesz@Using onnxruntime to optimize model - Debug level Set to verboser8CPUExecutionProviderdmlDmlExecutionProviderrocmrBmigraphxrCcudarAtensorrtTensorrtExecutionProviderzModelProto has external data not loaded into memory, ORT cannot create session. Please load external data before calling this function. See https://onnx.ai/onnx/repo-docs/ExternalData.html for more information. providersz)Save optimized model by onnxruntime to %s)-torchr?popset onnxruntimeget_available_providers isdisjointloggererror isinstancestrr r use_float16warningSessionOptionsGraphOptimizationLevelORT_ENABLE_BASICgraph_optimization_levelORT_ENABLE_EXTENDEDORT_ENABLE_ALLr with_suffixformatoptimized_model_filepathlenospathbasenameadd_session_config_entryprintlog_severity_levelappendhipr r ValueErrorradd_external_initializerslistSerializeToStringInferenceSessionexistsisfiledebug)r4r5r6r7r8r9r:r;r<r3deprecated_kwargs torch_versionmodel sess_options path_prefixkwargsrQexternal_namesexternal_valuess W/var/www/openai/venv/lib/python3.12/site-packages/onnxruntime/transformers/optimizer.pyoptimize_by_onnxruntimerHsu<  "" ".&**+?K+K/66{IX_uejk,@L) % &! +%'WW%5%56J%Kg%U "-- EG]  -- MsSoOp  PQ*+ ' F(;$% +,   u /0I  01I  #46MNI  01I  #46MNI01I/0      8 9   4 5   4 5*j) Z (]  +Fj*Q'..tN/CT/EZ[4>j*3U--/[eJ  \YYYRXY 77>>. /BGGNNCW4XX X LL<>RS rz model_type num_heads hidden_sizeoptimization_optionsc|dvr"|dk(s|dk(rtjd||tvr$tjd|dt|St|\}}}|jr5||jk7r&tjd|d|jd| t |}||||}|j ||jd |j_dd l m } | |j_ |S) aeOptimize Model by graph fusion logic. Note that ONNXRuntime graph optimizations (like constant folding) will not be applied. So it is better to enable constant folding during exporting ONNX model, or run optimize_by_onnxruntime on the model first like optimize_model. For BERT model, num_heads and hidden_size are optional. For other model types, you need to specify these parameters. Args: model (ModelProto): model object model_type (str, optional): model type - like bert, bert_tf, bert_keras or gpt2. Defaults to 'bert'. num_heads (int, optional): number of attention heads. Defaults to 0. 0 allows detect the parameter from graph automatically. hidden_size (int, optional): hidden size. Defaults to 0. 0 allows detect the parameter from graph automatically. optimization_options (FusionOptions, optional): optimization options that turn on/off some fusions. Defaults to None. Returns: object of an optimizer class. )rr*r.r/r#r)rzFPlease specify parameters of num_heads and hidden_size for model_type Unsupported model type: z) for graph fusion, directly return model.z&Model producer not matched: Expected "z", Got "z0".Please specify correct --model_type parameter.zonnxruntime.transformers) __version__) rXr] MODEL_TYPESr producer_nameroptimizetopological_sortrzrUrproducer_version) rzrrrroptimizer_classproducer_ optimizeronnxruntime_versions roptimize_by_fusionrs 6HHi[\n`kop`p_`j_klm$1*=fgh%0%<"_h x5+>+>>4XJhuGZGZF[\= = #,Z8y+>I +,  $>IOO!>':IOO$ rinputonly_onnxruntimec  B||dvsJ|tvrHtjd|dt|trt t |St |St|\} } } || }dg} d}tj}dj||rdnd}tjj|j|}d }t|tr t |d n|}t|rd }~|d kDr| |rgngd z } t||| ||| ||}n|d k(rt||| |d | ||}|r|stjd| t |}nt|tr t |}n|}|r | |||}nt!|||||}|j#|S)a Optimize Model by OnnxRuntime and/or python fusion logic. ONNX Runtime has graph optimizations (https://onnxruntime.ai/docs/performance/model-optimizations/graph-optimizations.html). However, the coverage is limited. We also have graph fusions that implemented in Python to improve the coverage. They can combined: ONNX Runtime will run first when opt_level > 0, then graph fusions in Python will be applied. To use ONNX Runtime only and no Python fusion logic, use only_onnxruntime flag and a positive opt_level like optimize_model(input, opt_level=1, use_gpu=False, only_onnxruntime=True) When opt_level is None, we will choose default optimization level according to model type. When opt_level is 0 and only_onnxruntime is False, only python fusion logic is used and onnxruntime is disabled. When opt_level > 1, use_gpu shall set properly since the optimized graph might contain operators for GPU or CPU only. If your model is intended for GPU inference only (especially float16 or mixed precision model), it is recommended to set use_gpu to be True, otherwise the model is not optimized for GPU inference. For BERT model, num_heads and hidden_size are optional. For other model types, you need specify these parameters. Args: input (str | ModelProto): input model path or ModelProto. model_type (str, optional): model type - like bert, bert_tf, bert_keras or gpt2. Defaults to 'bert'. num_heads (int, optional): number of attention heads. Defaults to 0. 0 allows detect the parameter from graph automatically. hidden_size (int, optional): hidden size. Defaults to 0. 0 allows detect the parameter from graph automatically. optimization_options (FusionOptions, optional): optimization options that turn on/off some fusions. Defaults to None. opt_level (int, optional): onnxruntime graph optimization level (0, 1, 2 or 99) or None. Defaults to None. When the value is None, default value (1 for bert and gpt2, 0 for other model types) will be used. When the level > 0, onnxruntime will be used to optimize model first. use_gpu (bool, optional): use gpu or not for onnxruntime. Defaults to False. only_onnxruntime (bool, optional): only use onnxruntime to optimize model, and no python fusion. Defaults to False. provider (str, optional): execution provider to use if use_gpu. Defaults to None. Returns: object of an optimizer class. Nrrr-r1rz) for optimization, directly return model.ConstantSharingzmodel_o{}_{}.onnxrGrHFrDTr)MatMulScaleFusionMatMulAddFusionMatmulTransposeFusionGemmActivationFusionBiasSoftmaxFusion)r5r3r6r7r8r9r:zKPlease specify a positive value for opt_level when only_onnxruntime is True)rrXr]rZr[r r tempfileTemporaryDirectoryrerhrijoinnamerrrcleanup)rrrrrr7r5rr9r3rrdefault_opt_levelr8temp_model_pathtemp_diroptimized_model_namer6has_external_data_fileoriginal_modelrzrs roptimize_modelrsl   ] :: :$1*=fgh/9%/EyE*+[9UZK[[.9*.E+_a*% --O**,H.55i'W\]77<< 7KL#DNuVYDZZ%@`eN(!%1}  2 !5 3"8   a 2 !5 3"8  de"?+ E3 5!#E9kB &uj)[Rfg   rcTt|dd}t|}|jS)z Get counter of fused operators in optimized model. Args: optimized_model_path (str): the path of onnx model. Returns: A dictionary with operator type as key, and count as value NT)rerE)r r get_fused_operator_statistics)r6rzrs rget_fusion_statisticsrs, +DT REe$I  2 2 44rc tjd}|jddtd|jddtd|jd d tjd t t jd d jt jz|jdd tdd|jdd tdd|jdd dd|jd |jdd dd|jd tj||jdd dd|jd |jd d dd!|jd "|jd#d tdd$|jd%d dd&|jd '|jd(d tgd)dd*+|jd,d dd-|jd .|jd/d dd0|jd 1|jd2d dd3|jd 4|j}|S)5NztGraph optimization tool for ONNX Runtime.It transforms ONNX graph to use optimized operators for Transformer models.) descriptionz--inputTzinput onnx model path)requiredtypehelpz--outputzoptimized onnx model pathz --model_typeFrz!Model type selected in the list: z, )rrdefaultchoicesrz --num_headsrznumber of attention heads like 12 for bert-base and 16 for bert-large. Default is 0 to detect automatically for BERT.For other model type, this parameter need specify correctly.)rrrrz --hidden_sizezhidden size like 768 for bert-base and 1024 for bert-large. Default is 0 to detect automatically for BERT. For other model type, this parameter need specify correctly.z --input_int32 store_truezyUse int32 (instead of int64) inputs. It could avoid unnecessary data cast when EmbedLayerNormalization is fused for BERT.)ractionr) input_int32z --float16zConvert all weights and nodes in float32 to float16. It has potential loss in precision compared to mixed precision conversion.)float16z --verbosezshow debug information.r9z --use_gpuzZUse GPU for inference. Set this flag if your model is intended for GPU when opt_level > 1.)r5z --providerz$Execution provider to use if use_gpuz--only_onnxruntimez 1 is used, optimized model for GPU might not run in CPU. Level 2 and 99 are intended for --only_onnxruntime.)rrrrrz--use_external_data_formatz4use external data format to store large model (>2GB))use_external_data_formatz--disable_symbolic_shape_inferzdiable symbolic shape inference)disable_symbolic_shape_inferz--convert_to_packing_modezEconvert the model to packing mode. Only available for BERT like model)convert_to_packing_mode)argparseArgumentParser add_argumentr[lowerrrrkeysrint set_defaultsr add_arguments parse_args)parserargss r_parse_argumentsrs  $ $VF  DsAXY  TB]^  YY[%%'( 0499[=M=M=O3P P   G   G  _  E* U  &'  eLOhi &  i   &   3   K   /  >   $ C   7 ( .   U; # T   6    D Krcd|rtjddytjdy)NDEBUGz8[%(filename)s:%(lineno)s - %(funcName)20s()] %(message)s)levelfmtz%(funcName)20s: %(message)s)r) coloredlogsinstallrs r _setup_loggerr)s*J  =>rc bt}t|jtj d|t j j|jt j j|jk(rtjdtj|}t|j|j|j|j |j"||j$|j&|j( }|j*r|j-d|j.r|j1|j3|j5}d|jvr'|j7|rtj9dntj9d|j:rA|jdk(r|j;|j< ntjd |j?|j|j@y) Nz arguments:zYSpecified the same input and output path. Note that this may overwrite the original model)r7rr5r3rT) keep_io_typesrz#The model has been fully optimized.zThe model has been optimized.z+Packing mode only supports BERT like models)!rrr9rXrwrhrirealpathroutputr]rparserrrrr7r5r3rrconvert_float_to_float16rchange_graph_inputs_to_int32get_operator_statisticsris_fully_optimizedinforrsave_model_to_filer)rrrfused_op_counts rmainr3s  D$,, LL:dV$% ww #rww'7'7 'DDrs(..t4    ..1 .. I ||***> ..0%%'<<>N  Y%A%A.%Q 9: 34 ## ??f $  - -$2S2S.S T NNH I   d.K.KLr__main__)rrrN)rrrNNFFF)?rloggingrhrpathlibrtypingrrrrrfusion_optionsronnxr r r4r onnx_model_bartr onnx_model_bertr onnx_model_bert_kerasronnx_model_bert_tfronnx_model_clipronnx_model_conformerronnx_model_gpt2ronnx_model_phironnx_model_sam2r onnx_model_t5ronnx_model_tnlrronnx_model_unetronnx_model_vaer onnx_utilsrrrU getLogger__name__rXrr[boolrrrrrrrrrrrs( ..(' ))4.)3)')%))'E   8 $ ]Iq ) ]Iq )A.%|Q7   ]Iq )  $i3  ]Iq ) y!, 1- L)Q ' ]Iq ) ]Iq ) ]Iq ) ; 1 % ]Iq )  L)Q '!" M9a (# ,48*.!%'"'"$(,J #J sJ/0J J #3-J } J c J  J  J  J #&J smJ  J ^48 8 888 8 #=1 8  8z48#"J#J j !JJJ J #=1 J } JJJJsmJJZ 5 5S#X 5zz?-M` zFr