g \ddlmZddlZddlZddlZddlmZddlmZm Z ddl Z ddl m Z m Z mZmZddlmZddlmZdd lmZmZmZmZmZmZmZdd lmZmZmZdd l m!Z!Gd d Z"Gdde"Z#e jHdejJejLdddddddddf ddZ'Gdde"Z(ddZ)ejTdddejLejLddde jHdf ddZ+dddejLddddf ddZ, ddZ-y)) annotationsN)Path)AnyCallable)CalibrationDataReaderCalibrationMethod TensorsDatacreate_calibrator) ONNXQuantizer) QDQQuantizer)MODEL_SIZE_THRESHOLD QuantFormatQuantizationMode QuantTypeload_model_with_shape_infermodel_has_pre_process_metadata&save_and_reload_model_with_shape_infer)IntegerOpsRegistry QDQRegistryQLinearOpsRegistry)TensorQuantOverridesHelpercLeZdZejej ddddddfdZy) QuantConfigNFc |xsg}|xsg}|xsg}||_||_||_||_||_||_||_||_y)a This is the Base class for both Static and Dynamic Quantize Configuration Args: activation_type: quantization data type of activation. Please refer to https://onnxruntime.ai/docs/performance/quantization.html for more details on data type selection weight_type: quantization data type of weight. Please refer to https://onnxruntime.ai/docs/performance/quantization.html for more details on data type selection op_types_to_quantize: specify the types of operators to quantize, like ['Conv'] to quantize Conv only. It quantizes all supported operators by default. nodes_to_quantize: List of nodes names to quantize. When this list is not None only the nodes in this list are quantized. example: [ 'Conv__224', 'Conv__252' ] nodes_to_exclude: List of nodes names to exclude. The nodes in this list will be excluded from quantization when it is not None. per_channel: quantize weights per channel reduce_range: quantize weights with 7-bits. It may improve the accuracy for some models running on non-VNNI machine, especially for per-channel mode use_external_data_format: option used for large size (>2GB) model. Set to False by default. N)op_types_to_quantize per_channel reduce_range weight_typeactivation_typenodes_to_quantizenodes_to_excludeuse_external_data_format) selfr rrr!r"rrr#s V/var/www/openai/venv/lib/python3.12/site-packages/onnxruntime/quantization/quantize.py__init__zQuantConfig.__init__!sfR,1r-339r$8!&(&.!2 0(@%)__name__ __module__ __qualname__rQUInt8QInt8r&r'r%rr s,"((OO!!&3Ar'rc eZdZejej ejejdddddddf dfd Z xZ S)StaticQuantConfigNFc tt ||||||| | | ||_||_||_| xsi|_y)a# This is the derived class for static Quantize Configuration Args: calibration_data_reader: a calibration data reader. It enumerates calibration data and generates inputs for the original model. calibrate_method: Current calibration methods supported are MinMax, Entropy and Percentile. quant_format: QuantFormat{QOperator, QDQ}. QOperator format quantizes the model with quantized operators directly. QDQ format quantize the model by inserting QuantizeLinear/DeQuantizeLinear on the tensor. extra_options: key value pair dictionary for various options in different case. Current used: extra.Sigmoid.nnapi = True/False (Default is False) ActivationSymmetric = True/False: symmetrize calibration data for activations (default is False). WeightSymmetric = True/False: symmetrize calibration data for weights (default is True). EnableSubgraph = True/False : Default is False. If enabled, subgraph will be quantized. Dyanmic mode currently is supported. Will support more in future. ForceQuantizeNoInputCheck = True/False : By default, some latent operators like maxpool, transpose, do not quantize if their input is not quantized already. Setting to True to force such operator always quantize input and so generate quantized output. Also the True behavior could be disabled per node using the nodes_to_exclude. MatMulConstBOnly = True/False: Default is False for static mode. If enabled, only MatMul with const B will be quantized. AddQDQPairToWeight = True/False : Default is False which quantizes floating-point weight and feeds it to solely inserted DeQuantizeLinear node. If True, it remains floating-point weight and inserts both QuantizeLinear/DeQuantizeLinear nodes to weight. OpTypesToExcludeOutputQuantization = list of op type : Default is []. If any op type is specified, it won't quantize the output of ops with this specific op types. DedicatedQDQPair = True/False : Default is False. When inserting QDQ pair, multiple nodes can share a single QDQ pair as their inputs. If True, it will create identical and dedicated QDQ pair for each node. QDQOpTypePerChannelSupportToAxis = dictionary : Default is {}. Set channel axis for specific op type, for example: {'MatMul': 1}, and it's effective only when per channel quantization is supported and per_channel is True. If specific op type supports per channel quantization but not explicitly specified with channel axis, default channel axis will be used. CalibTensorRangeSymmetric = True/False : Default is False. If enabled, the final range of tensor during calibration will be explicitly set to symmetric to central point "0". CalibMovingAverage = True/False : Default is False. If enabled, the moving average of the minimum and maximum values will be computed when the calibration method selected is MinMax. CalibMovingAverageConstant = float : Default is 0.01. Constant smoothing factor to use when computing the moving average of the minimum and maximum values. Effective only when the calibration method selected is MinMax and when CalibMovingAverage is set to True. QuantizeBias = True/False : Default is True which quantizes floating-point biases and it solely inserts a DeQuantizeLinear node. If False, it remains floating-point bias and does not insert any quantization nodes associated with biases. This extra option is only effective when quant_format is QuantFormat.QDQ. SmoothQuant = True/False : Default is False. If enabled, SmoothQuant algorithm will be applied before quantization to do fake input channel quantization. SmoothQuantAlpha = float : Default is 0.5. It only works if SmoothQuant is True. It controls the difficulty of weight and activation quantization. A larger alpha value could be used on models with more significant activation outliers to migrate more quantization difficulty to weights. SmoothQuantFolding = True/False : Default is True. It only works if SmoothQuant is True. If enabled, inserted Mul ops during SmoothQuant will be folded into the previous op if the previous op is foldable. UseQDQContribOps = True/False : Default is False. If enabled, the inserted QuantizeLinear and DequantizeLinear ops will have the `com.microsoft` domain, which forces use of ONNX Runtime's QuantizeLinear and DequantizeLinear contrib op implementations. The contrib op implementations may support features not standardized into the ONNX specification (e.g., 16-bit quantization types). MinimumRealRange = float|None : Default is None. If set to a floating-point value, the calculation of the quantization parameters (i.e., scale and zero point) will enforce a minimum range between rmin and rmax. If (rmax-rmin) is less than the specified minimum range, rmax will be set to rmin + MinimumRealRange. This is necessary for EPs like QNN that require a minimum floating-point range when determining quantization parameters. TensorQuantOverrides = dictionary : Default is {}. Set tensor quantization overrides. The key is a tensor name and the value is a list of dictionaries. For per-tensor quantization, the list contains a single dictionary. For per-channel quantization, the list contains a dictionary for each channel in the tensor. Each dictionary contains optional overrides with the following keys and values. 'quant_type' = QuantType : The tensor's quantization data type. 'scale' = Float : The scale value to use. Must also specify `zero_point` if set. 'zero_point' = Int : The zero-point value to use. Must also specify `scale` is set. 'symmetric' = Bool : If the tensor should use symmetric quantization. Invalid if also set `scale` or `zero_point`. 'reduce_range' = Bool : If the quantization range should be reduced. Invalid if also set `scale` or `zero_point`. 'rmax' = Float : Override the maximum real tensor value in calibration data. Invalid if also set `scale` or `zero_point`. 'rmin' = Float : Override the minimum real tensor value in calibration data. Invalid if also set `scale` or `zero_point`. QDQKeepRemovableActivations = True/False: Default is False. If true, "removable" activations (e.g., Clip or Relu) will not be removed, and will be explicitly represented in the QDQ model. If false, these activations are automatically removed if activations are asymmetrically quantized. Keeping these activations is necessary if optimizations or EP transformations will later remove QuantizeLinear/DequantizeLinear operators from the model. QDQDisableWeightAdjustForInt32Bias = True/False: Default is False. If true, QDQ quantizer will not adjust the weight's scale when the bias has a scale (input_scale * weight_scale) that is too small. execution_provider : A enum indicates the Execution Provider such as: CPU, TRT, NNAPI, SNE, etc. Raises: ValueError: Raise ValueError if execution provider is unknown )r rrr!r"rrr#N)superr&calibration_data_readercalibrate_method quant_format extra_options)r$r2r3r4r rrr!r"rrr#r5 __class__s r%r&zStaticQuantConfig.__init__XsWp +#!5/-#%%=  (?$ 0(*0br')r2r) r(r)r*r MinMaxrQDQrr,r& __classcell__r6s@r%r/r/WsM+11 __!OO!!&E1!6E1E1r'r/Fc J tjtjh}tjtjh}hd}t |t jr|nt j|d}t}d}t| rtj| ni}|jjD]$}t jj!|s#d}&g}| !t | t"r|j%| |jj&D]Q}|j)|j*| !t-| s-| ||s7|j/|j0S| | ||d|j3d}|r>gd}|Dcic]\}}||vs ||j5|}}}|j7|t9d|j:D}|j<dkrB|j?| tA fd |jCD}| vs| vs|rd|d <|r|j7|tE||tFjH||t#|jK|||| |xs|jMtNk\| Scc}}w) a  Returns a configuration suitable that quantizes the entire model to integer precision. Params: model_input: Path to the input model file or ModelProto. calibration_data_reader: Calibration data reader. calibrate_methode: The calibration method. Defaults to MinMax. activation_type: The default activation quantization type. Defaults to QUInt8. weight_type: The default weight quantization type. Defaults to QInt8. activation_symmetric: True if activations should be quantized symmetrically (i.e, rmax == -rmin) by default. Defaults to false. For int8 and int16, this results in zero-point values of 0. For uint8 and uint16, the zero-point values are 127 and 32,767, respectively. weight_symmetric: True if weights should be quantized symmetrically (i.e., rmax == -rmin) by default. Defaults to None. If set to None, weight_symmetric is assumed true if a weight's quant type is a signed int. per_channel: Global option that determines if a fixed set of operator types should be quantized per-channel. Defaults to false. Alternatively, use the tensor-level `tensor_quant_overrides` to select individual operators and their quantization axes. reduce_range: quantize weights with 1 less bit of precision (e.g., 7 bits for QInt8). Defaults to false. May improve the accuracy for some models running on non-VNNI machine, especially for per-channel mode. keep_removable_activations: Defaults to false. If true, "removable" activations (e.g., Clip or Relu) will not be removed, and will be explicitly represented in the QDQ model. If false, these activations are automatically removed if activations are asymmetrically quantized. Keeping these activations is necessary if optimizations or EP transformations will later remove QuantizeLinear/DequantizeLinear operators from the model. min_real_range: Default is None. If set to a floating-point value, the calculation of the quantization parameters (i.e., scale and zero point) will enforce a minimum range between rmin and rmax. If (rmax - rmin) is less than the specified minimum range, rmax will be set to rmin + min_real_range. tensor_quant_overrides: tensor-level quantization overrides. Defaults to None. The key is a tensor name and the value is a list of dictionaries. For per-tensor quantization, the list contains a single dictionary. For per-channel quantization, the list contains either a dictionary for each channel in the tensor or a single dictionary that is assumed to apply to all channels. An 'axis' key must be present in the first dictionary for per-channel quantization. Each dictionary contains optional overrides with the following keys and values. 'quant_type' = QuantType : The tensor's quantization data type. 'axis' = Int : The per-channel axis. Must be present for per-channel weights. 'scale' = Float : The scale value to use. Must also specify `zero_point` if set. 'zero_point' = Int : The zero-point value to use. Must also specify `scale` is set. 'symmetric' = Bool : If the tensor should use symmetric quantization. Invalid if also set `scale` or `zero_point`. 'reduce_range' = Bool : If the quantization range should be reduced. Invalid if also set `scale` or `zero_point`. Only valid for initializers. 'rmax' = Float : Override the maximum real tensor value in calibration data. Invalid if also set `scale` or `zero_point`. 'rmin' = Float : Override the minimum real tensor value in calibration data. Invalid if also set `scale` or `zero_point`. 'convert' = Dict : A nested dictionary with the same keys for an activation tensor that should be converted to another quantization type. 'convert["recv_nodes"] = Set : Set of node names that consume the converted activation, other nodes get the original type. If not specified, assume all consumer nodes get the converted type. nodes_to_exclude: List of nodes names to exclude from quantization. Alternatively, can provide a function that accepts an onnx.ModelProto and onnx.NodeProto as arguments and returns true if the give onnx.NodeProto should be excluded from quantization. extra_options: Additional options specified as string key/value pairs. Refer to the documentation for `quantize_static` for valid keys and values. Returns: A StaticQuantConfig object >CastQuantizeLinearDequantizeLinearF)load_external_dataT)MinimumRealRangeQDQKeepRemovableActivationsActivationSymmetricWeightSymmetricForceQuantizeNoInputCheckTensorQuantOverrides)) symmetricCalibTensorRangeSymmetric)moving_averageCalibMovingAverage)averaging_constantCalibMovingAverageConstant)max_intermediate_outputsCalibMaxIntermediateOutputs) percentileCalibPercentilec3^K|]%}|jdk(s|jdk(s"|'yw)zai.onnxN)domain).0xs r% z!get_qdq_config..gs)_!3Aqxx2~U^I^a!3s#--c3&K|]}|v ywNr-)rSt opset21_typess r%rUz!get_qdq_config..js*jGi!1 +=GisUseQDQContribOps) r3r4r rrr"rrr#r5)(rQInt16QUInt16QInt4QUInt4 isinstanceonnx ModelProto load_modelsetrcopydeepcopygraph initializerexternal_data_helperuses_external_datalistextendnodeaddop_typecallableappendnameget_dictgetupdatenext opset_importversionunionanyget_quant_typesr/rr8 differenceByteSizer)! model_inputr2r3calibrate_argsr ractivation_symmetricweight_symmetricrrkeep_removable_activationsmin_real_rangetensor_quant_overridesr"r5 q16_typesq4_typesop_types_to_excludemodelop_typesmodel_has_external_dataoverrides_helperrhfinal_nodes_to_excludermfinal_extra_optionscalib_extra_options_keysrrkeycalib_extra_options onnx_opsetoverrides_have_opset21_typesrZs! @r%get_qdq_configrsZ!!9#4#45I!1!12HH k4?? 3  __[U C uH#11G ,-R {{..  $ $ 7 7 D&* #/ # 3CT(J%%&67    T\\"  'H5E,Ft,&--dii8 !+'A3+%) 0 9 9 ; $  >V =UktSY]aoYoC##D) )=U   ""#67_!3!3__JB!1 '**jGWGgGgGi*j'j$ m +{m/KOk6:  2 3""=1 ) __'!("5"56I"JK/!"9"eU^^=MQe=e)  % s  J"JcDeZdZejdddddddffd ZxZS)DynamicQuantConfigNFc Ht |||||||||xsi|_y)a This is a class for dynamic Quant Configuration Args: extra_options: key value pair dictionary for various options in different case. Current used: extra.Sigmoid.nnapi = True/False (Default is False) ActivationSymmetric = True/False: symmetrize calibration data for activations (default is False). WeightSymmetric = True/False: symmetrize calibration data for weights (default is True). EnableSubgraph = True/False : Default is False. If enabled, subgraph will be quantized. Dynamic mode currently is supported. Will support more in the future. ForceQuantizeNoInputCheck = True/False : By default, some latent operators like maxpool, transpose, do not quantize if their input is not quantized already. Setting to True to force such operator always quantize input and so generate quantized output. Also the True behavior could be disabled per node using the nodes_to_exclude. MatMulConstBOnly = True/False: Default is True for dynamic mode. If enabled, only MatMul with const B will be quantized. execution_provider : A enum indicates the Execution Provider such as: CPU, TRT, NNAPI, SNE, etc. Raises: ValueError: Raise ValueError if execution provider is unknown )rrrrr!r"r#N)r1r&r5) r$rrr!r"rrr#r5r6s r%r&zDynamicQuantConfig.__init__s<B !5#%#/-%=  +0br')r(r)r*rr,r&r9r:s@r%rrs+OO!!&*1*1r'rch|tjk(r|tjk(r td|tjk7r"|tjk(rtd|d|tjk(r"|tjk7rtd|dtj tj g}||vs||vr|tjk7r td|tjk(r>|tjk(r*|tjk7rtjdyyyy)NzrONNXRuntime quantization doesn't support data format:activation_type=QuantType.QInt8, weight_type=QuantType.QUInt8zFONNXRuntime quantization doesn't support data format: activation_type=z@ !=QuantType.QFLOAT8E4M3FN, weight_type=QuantType.QFLOAT8E4M3FN.zkONNXRuntime quantization doesn't support data format: activation_type=QuantType.QFLOAT8E4M3FN, weight_type=z!=QuantType.QFLOAT8E4M3FNz8Only QuantFormat.QDQ supports 16-bit quantization types.zvPlease use QuantFormat.QDQ for activation type QInt8 and weight type QInt8. Or it will lead to bad performance on x64.) rr,r+ ValueError QFLOAT8E4M3FNr\r]rr8loggingwarning)r4r rrs r%check_static_quant_argumentsrs0)//)kY=M=M.M L  )111kYE\E\6\TUdTefN O  )111kYE\E\6\ &-'@ B  !!9#4#45I9$ y(@lVaVeVeFeSTT)//)kY__.LQ]alapapQp 9 Rq.L)r'cx )|tjk(s|tjk(r| tjk7r t d| xsi} | xsg} | xsg} |xsg}t j }|rt|dk(rQttj}ttj}tt||z}t|tjr t!|nt#t%|}t'|}|st)j*dgd}|Dcic]\}}|| vs || j-|}}}| j-ddrZddl} |j1dddl)dd lm})fd }|j>j@Dcgc]}|jB}}|}||||}~|jE| j-d d| j-dd}tGjHd}t%|jBjKdjM}|jO|| jQ|jRj>j@Dcgc]}|jB|vs|jBc}t#t%|}tGjHd5} t|tjr1tUt%| dz }!tjV||!d|!}tYt%||t%| jKdjM| | |}"| j-dd}#|#rat}$|$|#zdk7rt d|$d|#dt[d|$|#D]+}%|%|#z}&j]|%|&|"j_-n|"j_|"ja}'t|'tbs$tedtg|'dtg|"d ~"dddti||||tjjlurto||||d||'| | || }(ntq|||||'| | || }(|(js|(jRju|| |st)j*d| j-ddrjwyycc}}w#t2$r)}t)j4|d t7d |d}~wwxYwcc}wcc}w#1swYxYw) a) Given an onnx model and calibration data reader, create a quantized onnx model and save it into a file It is recommended to use QuantFormat.QDQ format from 1.11 with activation_type = QuantType.QInt8 and weight_type = QuantType.QInt8. If model is targeted to GPU/TRT, symmetric activation and weight are required. If model is targeted to CPU, asymmetric activation and symmetric weight are recommended for balance of performance and accuracy. Args: model_input: file path of model or ModelProto to quantize model_output: file path of quantized model calibration_data_reader: a calibration data reader. It enumerates calibration data and generates inputs for the original model. quant_format: QuantFormat{QOperator, QDQ}. QOperator format quantizes the model with quantized operators directly. QDQ format quantize the model by inserting QuantizeLinear/DeQuantizeLinear on the tensor. activation_type: quantization data type of activation. Please refer to https://onnxruntime.ai/docs/performance/quantization.html for more details on data type selection calibrate_method: Current calibration methods supported are MinMax and Entropy. Please use CalibrationMethod.MinMax or CalibrationMethod.Entropy as options. op_types_to_quantize: specify the types of operators to quantize, like ['Conv'] to quantize Conv only. It quantizes all supported operators by default. per_channel: quantize weights per channel reduce_range: quantize weights with 7-bits. It may improve the accuracy for some models running on non-VNNI machine, especially for per-channel mode weight_type: quantization data type of weight. Please refer to https://onnxruntime.ai/docs/performance/quantization.html for more details on data type selection nodes_to_quantize: List of nodes names to quantize. When this list is not None only the nodes in this list are quantized. example: [ 'Conv__224', 'Conv__252' ] nodes_to_exclude: List of nodes names to exclude. The nodes in this list will be excluded from quantization when it is not None. use_external_data_format: option used for large size (>2GB) model. Set to False by default. extra_options: key value pair dictionary for various options in different case. Current used: extra.Sigmoid.nnapi = True/False (Default is False) ActivationSymmetric = True/False: symmetrize calibration data for activations (default is False). WeightSymmetric = True/False: symmetrize calibration data for weights (default is True). EnableSubgraph = True/False : Default is False. If enabled, subgraph will be quantized. Dyanmic mode currently is supported. Will support more in the future. ForceQuantizeNoInputCheck = True/False : By default, some latent operators like maxpool, transpose, do not quantize if their input is not quantized already. Setting to True to force such operator always quantize input and so generate quantized output. Also, the True behavior could be disabled per node using the nodes_to_exclude. MatMulConstBOnly = True/False: Default is False for static mode. If enabled, only MatMul with const B will be quantized. AddQDQPairToWeight = True/False : Default is False which quantizes floating-point weight and feeds it to solely inserted DeQuantizeLinear node. If True, it remains floating-point weight and inserts both QuantizeLinear/DeQuantizeLinear nodes to weight. OpTypesToExcludeOutputQuantization = list of op type : Default is []. If any op type is specified, it won't quantize the output of ops with this specific op types. DedicatedQDQPair = True/False : Default is False. When inserting QDQ pair, multiple nodes can share a single QDQ pair as their inputs. If True, it will create identical and dedicated QDQ pair for each node. QDQOpTypePerChannelSupportToAxis = dictionary : Default is {}. Set channel axis for specific op type, for example: {'MatMul': 1}, and it's effective only when per channel quantization is supported and per_channel is True. If specific op type supports per channel quantization but not explicitly specified with channel axis, default channel axis will be used. CalibTensorRangeSymmetric = True/False : Default is False. If enabled, the final range of tensor during calibration will be explicitly set to symmetric to central point "0". CalibStridedMinMax = Optional[int] : Default is None. If set to an integer, during calculation of the min-max, only stride amount of data will be used and then all results will be merged in the end. CalibMovingAverage = True/False : Default is False. If enabled, the moving average of the minimum and maximum values will be computed when the calibration method selected is MinMax. CalibMovingAverageConstant = float : Default is 0.01. Constant smoothing factor to use when computing the moving average of the minimum and maximum values. Effective only when the calibration method selected is MinMax and when CalibMovingAverage is set to True. CalibMaxIntermediateOutputs = Optional[int] : Default is None. If set to an integer, during calculation of the min-max range of the tensors it will load at max value number of outputs before computing and merging the range. This will produce the same result as all computing with None, but is more memory efficient. SmoothQuant = True/False : Default is False. If enabled, SmoothQuant algorithm will be applied before quantization to do fake input channel quantization. SmoothQuantAlpha = float : Default is 0.5. It only works if SmoothQuant is True. It controls the difficulty of weight and activation quantization. A larger alpha value could be used on models with more significant activation outliers to migrate more quantization difficulty to weights. SmoothQuantFolding = True/False : Default is True. It only works if SmoothQuant is True. If enabled, inserted Mul ops during SmoothQuant will be folded into the previous op if the previous op is foldable. UseQDQContribOps = True/False : Default is False. If enabled, the inserted QuantizeLinear and DequantizeLinear ops will have the `com.microsoft` domain, which forces use of ONNX Runtime's QuantizeLinear and DequantizeLinear contrib op implementations. The contrib op implementations may support features not standardized into the ONNX specification (e.g., 16-bit quantization types). MinimumRealRange = float|None : Default is None. If set to a floating-point value, the calculation of the quantization parameters (i.e., scale and zero point) will enforce a minimum range between rmin and rmax. If (rmax - rmin) is less than the specified minimum range, rmax will be set to rmin + MinimumRealRange. This is necessary for EPs like QNN that require a minimum floating-point range when determining quantization parameters. TensorQuantOverrides = dictionary : Default is {}. Set tensor quantization overrides. The key is a tensor name and the value is a list of dictionaries. For per-tensor quantization, the list contains a single dictionary. For per-channel quantization, the list contains a dictionary for each channel in the tensor. Each dictionary contains optional overrides with the following keys and values. 'quant_type' = QuantType : The tensor's quantization data type. 'scale' = Float : The scale value to use. Must also specify `zero_point` if set. 'zero_point' = Int : The zero-point value to use. Must also specify `scale` is set. 'symmetric' = Bool : If the tensor should use symmetric quantization. Invalid if also set `scale` or `zero_point`. 'reduce_range' = Bool : If the quantization range should be reduced. Invalid if also set `scale` or `zero_point`. 'rmax' = Float : Override the maximum real tensor value in calibration data. Invalid if also set `scale` or `zero_point`. 'rmin' = Float : Override the minimum real tensor value in calibration data. Invalid if also set `scale` or `zero_point`. QDQKeepRemovableActivations = True/False: Default is False. If true, "removable" activations (e.g., Clip or Relu) will not be removed, and will be explicitly represented in the QDQ model. If false, these activations are automatically removed if activations are asymmetrically quantized. Keeping these activations is necessary if optimizations or EP transformations will later remove QuantizeLinear/DequantizeLinear operators from the model. QDQDisableWeightAdjustForInt32Bias = True/False: Default is False. If true, QDQ quantizer will not adjust the weight's scale when the bias has a scale (input_scale * weight_scale) that is too small. zIOnly Distribution calibration method is supported for float quantization.rPlease consider to run pre-processing before quantization. Refer to example: https://github.com/microsoft/onnxruntime-inference-examples/blob/main/quantization/image_classification/cpu/ReadMe.md ))rGrF)rIrH)rKrJ)rMrL)rOrN SmoothQuantFNz/neural_compressor.adaptor.ox_utils.smooth_quant.zLneural-compressor is not correctly installed. Please check your environment.)ORTSmoothQuantc3LKj}|D]}|df ywrX)rf) data_readerdatar2res r%inc_dataloaderz'quantize_static..inc_dataloaders,'$--(?@K#Dj $s!$SmoothQuantAlphag?SmoothQuantFoldingTz ort.quant.)prefixz sq_model.onnxzmodel_input.onnx)save_as_external_datazaugmented_model.onnx)augmented_model_pathr3r#r5CalibStridedMinMaxzTotal data size (z#) is not divisible by stride size (z).) start_index end_indexzUnexpected type z" for tensors_range and calibrator=zPlease consider pre-processing before quantization. See https://github.com/microsoft/onnxruntime-inference-examples/blob/main/quantization/image_classification/cpu/ReadMe.md )$?@ k4?? 3 /{; (k): ; 9?M    9Q8P$TX\iTi]  t $$8P. v  # #$U V R ! ',kk&6&67&6aff&6 7#% K\ B  ]../A3GIZIZ[oquIvw--\B7<<(11/BKKM  ;1B1B1G1G d1GA166YcKc1G de+D,=>  $ $L 9] k4?? 3d=14FFGK OO&*  &K&   !%m!4!=!=>T!U!^!^!`-%=-  ""#7> !"9:O'1, #4_4EEhiohppr!sttq/6:!FN '11ey1Y''(?@;  # #$; <"//1 -5"4 #6"77YZ^_iZjYkklm  I :L! L{,,,!             !            OO&&|5MN   ./_ v MMQCq' "mntu u v8!e : 9sC4 S+S+0S1)T&T+T+ET01 T#:$TT#0T9c | xsi} |xsg}|xsg}|xsg}tj} |rt|dk(rtt j }t |tjr t|ntt|} t| } | stjdd| vrd| d<t| ||| d|t j"d|||| } | j%| j&j)||y)a Given an onnx model, create a quantized onnx model and save it into a file Args: model_input: file path of model or ModelProto to quantize model_output: file path of quantized model op_types_to_quantize: specify the types of operators to quantize, like ['Conv'] to quantize Conv only. It quantizes all supported operators by default. per_channel: quantize weights per channel reduce_range: quantize weights with 7-bits. It may improve the accuracy for some models running on non-VNNI machine, especially for per-channel mode weight_type: quantization data type of weight. Please refer to https://onnxruntime.ai/docs/performance/quantization.html for more details on data type selection nodes_to_quantize: List of nodes names to quantize. When this list is not None only the nodes in this list are quantized. example: [ 'Conv__224', 'Conv__252' ] nodes_to_exclude: List of nodes names to exclude. The nodes in this list will be excluded from quantization when it is not None. use_external_data_format: option used for large size (>2GB) model. Set to False by default. extra_options: key value pair dictionary for various options in different case. Current used: extra.Sigmoid.nnapi = True/False (Default is False) ActivationSymmetric = True/False: symmetrize calibration data for activations (default is False). WeightSymmetric = True/False: symmetrize calibration data for weights (default is True). EnableSubgraph = True/False : Default is False. If enabled, subgraph will be quantized. Dynamic mode currently is supported. Will support more in the future. ForceQuantizeNoInputCheck = True/False : By default, some latent operators like maxpool, transpose, do not quantize if their input is not quantized already. Setting to True to force such operator always quantize input and so generate quantized output. Also the True behavior could be disabled per node using the nodes_to_exclude. MatMulConstBOnly = True/False: Default is True for dynamic mode. If enabled, only MatMul with const B will be quantized. rrMatMulConstBOnlyTFN)r IntegerOpsrrkrrr`rarbrrrrrrr rr+rrr)r~rrrrrr!r"r#r5rrrrs r%quantize_dynamicrsl"'RM'-2)/R/52  & &D 3';#<#A#$6$;$;$=> k4?? 3 /{; (k): ; 9?M   .,0 ()     I OO&&|5MNr'c@t|trt|||j|j|j |j |j|j|j|j|j|j|j|jyt|trft!|||j|j|j|j|j|j|j|j yddlm}m}t||rht|t(j*r|nt)j,|}|||}|j/|j0j3|dyt5d) a+Quantize a model with QuantConfig. Args: model_input (str | Path | ModelProto): Path to the model or ModelProto to quantize. model_output (str | Path): Path to save the quantized model. quant_config (QuantConfig | WeightOnlyQuantConfig): Quantization Configuration. ) r3r4r rrr!r"rrr#r5)rrr!r"rrr#r5r)MatMul4BitsQuantizerWeightOnlyQuantConfig) algo_configTztInvalid quantization config type, it must be either StaticQuantConfig, DynamicQuantConfig, or WeightOnlyQuantConfig.N)r`r/rr2r3r4r rrr!r"rrr#r5rrmatmul_4bits_quantizerrrrarbloadprocessrrr)r~r quant_configrrrquants r%quantizerasZ, 12    0 0)::%22(88$00!-!B!B*<<)::$00%22%1%J%J&44 " L"4 5  $00!-!B!B*<<)::$00%22%1%J%J&44  X l$9 :#-k4??#KKQUQZQZ[fQgE(LIE MMO KK * *< >@ r')r~str | Path | onnx.ModelProtor2rrzdict[str, Any] | Nonerboolrz bool | Nonerrrrrrrz float | Nonerz&dict[str, list[dict[str, Any]]] | Noner"zDlist[str] | Callable[[onnx.ModelProto, onnx.NodeProto], bool] | Noner5z dict | Nonereturnr/)r4rr rrr)r~rr str | Pathr2r)r~rrr)r~rrrrr). __future__rrerrpathlibrtypingrrra calibraterr r r onnx_quantizerr qdq_quantizerr quant_utilsrrrrrrrregistryrrrrrrr/r7r+r,rrrr8rrrr-r'r%rs #  __)'JI>4A4AnF1 F1X'--,0$$!&$(',#'EI]a!%^-^2^* ^^"^^^!%^!^C^[^^ !^B+1+1\ DOO"&--m-mm3mf "aO-aOaOH8-888r'