g~ddlmZddlZddlZddlZddlZddlZddlZddl m Z ddl Z ddl mZmZmZmZddlmZddlmZmZddlmZddlmZdd lmZmZej<d ej> ej@e!Z"Gd d Z#Gdde#Z$Gdde#Z%Gdde#Z&Gdde#Z'Gdde#Z(dZ)GddZ*dXNN)rstrrrrtuple[str, ...] | Noner"tuple[tuple[str, int], ...] | None)__name__ __module__ __qualname__rr rrrs: 8<9= YY"Y5 Y 7 Yr rc>eZdZdejdf dfd ZxZS)RTNWeightOnlyQuantConfigNct|tjk(sJd|i}t| d||||_y)a This is a class for round-to-nearest (RTN) algorithm Weight Only Quant Configuration. RTN is the most straightforward way to quantize weight using scale maps. Args: ratios: percentile of clip. Defaults to {}. quant_format (QuantFormat{QOperator, QDQ}, optional): QOperator format quantizes the model with quantized operators directly. QDQ format quantize the model by inserting QuantizeLinear/DeQuantizeLinear on the tensor. Defaults to QuantFormat.QOperator. op_types_to_quantize (optional): set of operator types to quantize. z"RTN only supports QOperator formatNRTNrrr)r QOperatorsuperrratios)rr0rr __class__s rrz!RTNWeightOnlyQuantConfig.__init__;sM({444Z6ZZ4 >F %!5   r )rr#r%r&r'rr.r __classcell__r1s@rr*r*:s( **7; 5 r r*cLeZdZddddddejdf dfd ZxZS)GPTQWeightOnlyQuantConfigNg{Gz?FTc |tjk(sJdt | d||||_||_||_||_||_||_ y)a This is a class for GPTQ algorithm Weight Only Quant Configuration. GPTQ algorithm provides more accurate quantization but requires more computational resources. Args: calibration_data_reader: a calibration data reader. It enumerates calibration data and generates inputs for the original model. percdamp: percent of the average Hessian diagonal to use for dampening. block_size (int, optional): channel number in one block to execute a GPTQ quantization iteration. actorder (bool, optional): whether rearrange Hessian matrix considering the diag's value. mse (bool, optional): whether get scale and zero point with mse error. perchannel (bool, optional): whether quantize weight per-channel. quant_format (QuantFormat{QOperator, QDQ}, optional): QOperator format quantizes the model with quantized operators directly. QDQ format quantize the model by inserting QuantizeLinear/DeQuantizeLinear on the tensor. Defaults to QuantFormat.QOperator. op_types_to_quantize (optional): set of operator types to quantize. z#GPTQ only supports QOperator formatGPTQr-N) rr.r/rcalibration_data_readerpercdamp block_sizeactordermse perchannel) rr:r;r<r=r>r?rrr1s rrz"GPTQWeightOnlyQuantConfig.__init__\shF{444[6[[4 %!5  (?$  $  $r )r:zCalibrationDataReader | Nonerr#r2r4s@rr6r6[sAAE  **7;/%!=/%5/%/%r r6cHeZdZdddejddf dfd ZxZS)HQQWeightOnlyQuantConfigr7r Nc|tjk(sJdt| d|||||_||_||_y)a This is a class for HQQ algorithm Weight Only Quant Configuration. HQQ algorithm quant weight without needing calibrate data. Args: block_size (int, optional): channel number in one block to execute a HQQ quantization iteration. bits (int, optional): how many bits to represent weight. axis (int, optional): 0 or 1. which axis to quantize. https://arxiv.org/pdf/2309.15531.pdf quant_format (QuantFormat{QOperator, QDQ}, optional): QOperator format quantizes the model with quantized operators directly. QDQ format quantize the model by inserting QuantizeLinear/DeQuantizeLinear on the tensor. Defaults to QuantFormat.QOperator. op_types_to_quantize (optional): set of operator types to quantize. quant_axes (dict[str, int], optional): op:axis, which axis to quantize for an op. Default {MatMul: 0, Gather: 1} z"HQQ only supports QOperator formatHQQrrrrN)rr.r/rr<bitsaxis)rr<rFrGrrrr1s rrz!HQQWeightOnlyQuantConfig.__init__sT:{444Z6ZZ4 %!5!  %  r )rr#rr$r2r4s@rrArAs;   **7;9=' 5 '7''r rAcTeZdZdddejddf dfd ZxZS)DefaultWeightOnlyQuantConfigr7FNcdt|d|||||_||_d|_||_y)a This is a class for weight only affine quantization configuration. Args: block_size (int, optional): channel number in one block to execute an affine quantization iteration. is_symmetric (bool, optional): whether quantize weight symmetrically. accuracy_level (int, optional): Accuracy level of the 4-bit quantized MatMul computation. Refer to the MatMulNBits contrib op's 'accuracy_level' attribute for details. (https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#commicrosoftmatmulnbits) quant_format (QuantFormat{QOperator, QDQ}, optional): QOperator format quantizes the model with quantized operators directly. QDQ format quantize the model by inserting QuantizeLinear/DeQuantizeLinear on the tensor. Defaults to QuantFormat.QOperator. op_types_to_quantize (optional): set of operator types to quantize. quant_axes (dict[str, int], optional): op:axis, which axis to quantize for an op. Default {MatMul: 0, Gather: 1} DEFAULTrErBN)r/rr< is_symmetricrFaccuracy_level)rr<rLrMrrrr1s rrz%DefaultWeightOnlyQuantConfig.__init__sC< %!5!  %( ,r ) r<intrLboolrM int | Nonerr#rr$r2r4s@rrIrIsW"%) **7;9='-'-'-# '- 5 '-7'-'-r rIc2eZdZ dfd ZdZdZxZS)NVAWQWeightOnlyQuantConfigct ddl}ddlm}||_||_ ddlm}||_ dd lm}m } ||_| |_ t |-d tjdd |jj|jjj!rdnd} |j#|||ddd| ddddd } | |_||_y#t$rt dtddwxYw#t$rt dtddwxYw#t$rt d td dwxYw)a= Configuration for the nvidia_awq quantization method. Args: tokenizer_dir (str): pathof the tokenizer dir. dataset_name (str): Name of the dataset. cache_dir (str): Directory for caching. calibration_method (str): calib method for nvidia_awq. rN) DataLoaderzfError: The 'torch' library is required but not installed. Please install it using 'pip install torch'.z torch is not installed. Exiting.) load_datasetzlError: The 'datasets' library is required but not installed. Please install it using 'pip install datasets'.z#datasets is not installed. Exiting.) AutoConfig AutoTokenizerztError: The 'transformers' library is required but not installed. Please install it using 'pip install transformers'.z'transformers is not installed. Exiting. nvidia_awqrEcudacpu r iTFr7) dataset_name model_name cache_dir calib_size batch_sizer<deviceuse_fp16use_buffer_shareadd_past_kv_inputsmax_calib_rows_to_loadadd_position_ids)torchtorch.utils.datarT ImportErrorprintdatasetsrU transformersrVrWr/rrQDQrarY is_availableget_calib_inputsr:calibration_method) r tokenizer_dirr\r^rprgrTrUrVrWra calib_inputsr1s rrz#NVAWQWeightOnlyQuantConfig.__init__sm" L  3DJ(DO O - ,D  S >(DO!.D  "$!%  ""TZZ__-I-I-K6QVW,,%$"##&!-  (4$"4o L x @At K  L O ~ CD$ N  O S G GHd R  Ss!C C2D!C/2!D!D7c b|j} |} |} t|tr<| j||| j} | j||| j} | j | j d} |rJ| j jddz } | j| dk(d| j | d<|r|r | jn | j}| j\}}|j}|j|j|jz}}t!|j"D]q}| j%|||r|nd|||}| j%|||r|nd|||}| j'd|d|j d|d |j is| S) N)radtype) input_idsattention_maskr r position_idszpast_key_values.z.keyz.value)rg isinstancelisttensorint64 contiguouslongcumsum masked_fill_float16float32shapemax_position_embeddingsnum_key_value_heads hidden_sizenum_attention_headsrangenum_hidden_layerszerosupdate)rconfig input_ids_argattention_mask_argrdrarbrcrfrgrurvinputsrx torch_dtyper`sequence_lengthmax_sequence_length num_heads head_sizeipast_key past_values rmake_model_inputz+NVAWQWeightOnlyQuantConfig.make_model_input5s ! + mT * ]6 UI"\\*F> " +3%--K*3// 'J"("@"@ **""f&@&@@!I6334 ;;+;'!% '#[[+;'!% )  *1#T2H4G4G4I*1#V4j6K6K6M#50 r c 4|j} |j}|j}| j|d|d}|j|d|d}|j ddi|j |_|| ksJdd|vr&|ddd jt| }d }n d |vr |d d}d}ntd|d||d|}|j|ddd|}|j|}|d}|d}|j}|||d}|||d}t|jt|jk(sJt|t|k(sJ||z}g}t|D] \}}|j!|||dz k(s ng}t|D] \}}|j!|||dz k(s nt#d|dt|dt|dg}t|D]t} || }!|| }"|j%||!|"| ||| | }#|#j'D$%cic]$\}$}%|$|%j)j+&}#}$}%|j!|#vt#dt|d|Scc}%}$w)NT)use_auth_tokenr^trust_remote_code pad_tokenz[PAD]z8calib size should be no more than max_calib_rows_to_loadcnn cnn_dailymailz3.0.0train)namesplitarticlepilezmit-han-lab/pile-val-backup validation)rtextz dataset "z" not supportedpt)return_tensorspadding truncation max_lengthrurvF)r`shuffler z/ --Quantize-Script-- number_of_batched_samples=z, batch-input-ids-list-len=z, batched_attention_mask= z0 --Quantize-Script-- number of batched inputs = )rVrWrUfrom_pretrainedadd_special_tokens eos_tokenrselectr ValueErrorbatch_encode_plustorTlendataset enumerateappendrjritemsrZnumpy)&rr\r]r^r_r`r<rarbrcrdrerf auto_configauto_tokenizerrUr tokenizerdataset2column batch_encodedbatch_encoded_input_idsbatch_encoded_attention_mask data_loadercalib_dataloader_input_idscalib_dataloader_attention_masknumber_of_batched_samplesbatched_input_idsidxdatabatched_attention_maskbatched_inputs_listrrurvr input_name torch_tensors& rroz+NVAWQWeightOnlyQuantConfig.get_calib_inputsvsH oo ++(( ,, tyTX- #22 tyTX3  $$k7%;<'11 33o5oo3 L #O'QXXY^_uYvwHF | ##$AVHFyoFG GF#KZ0!33 T4DU_4 &((0 "/ "<'45E'F$oo %01HU_in%o"*5 (Z+ '-556#>]>e>e:ffff-.#6U2VVVV$.*$<!"#=>IC  $ $T *0145? "$"#BCIC " ) )$ /0145D >?X>YZ((+,=(>'??XY\]sYtXuuw y !01A)!,I3A6N**"   F^d]i]i]kl]kAY\j,"2"2"4":":"<<]kFl  & &v .2" A#FYBZA[[]^_"" ms;)J)r./cacheawq_lite)r%r&r'rrror3r4s@rrRrRs!% N5`?Ba#r rRcPt|tj||z z|k(SN)rNnpceil)val1val2s r is_divisiblers$ tbggdTk** +t 33r c^eZdZ ddZe d d dZedZ d dZd dZy) HQQWeightOnlyQuantizerc||_yrrrrs rrzHQQWeightOnlyQuantizer.__init__  r Ncddl|dddddn|}|d|d|d |d f\}}} } |jr jn j} |j | } |j | }|j | }|ffd } d }t | D]}j | |z|zj|d|d }||z |z }| | |z |}j|| |z |zz |d}|| z}tj| |z j}|r t|tj |d||kr|}n~ ~~~||fS)Nrgffffff?g$@g)\(?)lp_normbetakappaitersrrrrc |dk(rLj|jjjj |d|z z zSj|jjjj |d|z j j |dz|dz zz zS)Nr ?g:0yE>)signnn functionalreluabspow)xrprgs r shrink_opz:HQQWeightOnlyQuantizer.optimize_weights..shrink_opsAvzz!}uxx':':'?'? ! sUYz@Y'ZZZzz!}uxx':':'?'?IIaLC$J%))EIIaL4Ma-34z)*  c3d{r c |jd|jdk(r|j}|j}|dvrA|jdz|z}t|D]}|ddxxx||d|||zzzcccyt d)Nr)rBrzOnly 2,4,8 bits are supported.)rT element_sizerNotImplementedError) pack_tensorori_int_tensorrFcompress_ratiojs rpack_on_row_fast_248bitz.HQQWeightOnlyQuantizer.pack_on_row_fast_248bits   Q >#7#7#: :+--N%--K 9 (557!;tCN>*AB>!2C^2C#DQR#TT+&&FG Gr c<ddl}|j} | j} || ||zz |z} |dk(r+|jjj | d| fdd} n,|jjj | ddd| fdd} | j} |-|r+|dk(r| j d|gn| j |dg} |dur#| j| j}} d}n,| j|dd} | j|dd}d|zdz }d}||g}||| z z jd }|| z }|dk(jjdkDr|||dk(<||z jd }| |z}|r|j|}|r|j| |||| \}}|j| |z|zj|d|d}|j | j}d |z }|dk(r+|j | dd}|j | dd}n*|j d| d}|j d| d}~ ~ ~||j|j |j|j fS) Nrr constantrwFTrrg@)max)r{rrrrGr)rgrrrrpadreshapeminrrsumitemrrrNrrt)rr{rF channel_wise group_sizeoptimize round_zerorGrgweight ori_shapepad_lenr_min_maxmax_vmin_vrr min_max_axisrrs rquantize_internalz(HQQWeightOnlyQuantizer.quantize_internal(s LL  $* << J 19XX((,,Va\:qQFXX((,,VaAw5GUVWF   " :>!)V^^R$45&..ZdfhYiJjF 5 vzz|$DH::4:6q9D::4:6q9D4! %.$+&--#-6d{ A  " " $ ) ) +a /.3L* +\)00S09Euu} ;;t$D //vUQU_fmq/rKE4kk&5.4/066wqz71:Nkk% $$&e  19MM%(B/E<<a"-DMM"eBi0E<<E"I.D D$EHHV\\*DGGFLL,AAAr c~ |jdk(r tdddl}tj d|j d|j d}t||\}}|tj d|gStjj|}t|jd k7rtj d |gS|j|}|jjr|j}|j!|j"|j$j&|j$j( \} } } | j+} | j+} | j+} |j-| jd| jdd zf|j.| j0 } |j3| | |j$j&| j5j7} | j5j7}| j9d } |j9d }|j\}}|j$j(}|d z}||zdz |z}| j9|||} tjj;| j5j7}|j dz|_|j D].}|j |k(s|j j=|ntjj;| }|j dz|_|j>jA||g|j d|j |j g}tjj;|}|j dz|_|j>jA|g|jC|j i}|j\}}||d<||d<|j$j&|d<|j$j(|d<tjDjF d||jHdg|j r|j dznddd|}tj d|j d|gS) Target node: QOperator node: QDQ nodes: MatMul MatMulNBits DeQuantizeLinear -> MatMul Gather GatherBlockQuantized Gather, Gather, Gather (optional) -> DequantizeLinear If the node is target node with fp32 or fp16 const weight, quantize the weight to int4 and return the new nodes. If QOperator format, return the corresponding QOperator nodes. If QDQ format, return the corresdponging QDQ nodes. Gather (quantized data) + Gather (scales) + Gather (optional, zero points) -> DequantizeLinear is not supported yet because Gather does not support int4 data. rz/Gather quantization is not supported yet in HQQrNstart to quantize  ...r 2MatMul doesn't have const weight. Skip to quantizer)MatMul weight is not 2D. Skip to quantize)rFr)rtrarw_Q4_scales _zero_pointsKNrFr< com.microsoftroutputsrdomaincomplete quantization of  MatMulNBits)%op_typerrgloggerinforinputget_initializeronnx numpy_helperto_arrayrr from_numpyrYrnr rrrFr<r}ruint8rar rZrr from_arrayremove initializerextendrhelper make_nodeoutput)rnode graph_stackrginput_bb_pbbs_graphb_array b_array_torchquant_weight_torch scales_torchzero_points_torch packed_torchscales zero_pointsrowscolsr< blob_sizek_blocksb_quantr7 scales_tensor input_names zp_tensorkwargsmatmul_q4_nodes rquantizezHQQWeightOnlyQuantizer.quantizeis <<8 #%&WX X ( 489**Q-(+>h < KKL M6M##,,T2 w}}  " KKC D6M((1 :: " " $)..0M>B>T>T OO$++"2"2t{{?U?U?U? ;L*;0::<#..0 -88:{{  % %a (*<*B*B1*E*J K++%,,# $$\3Et{{GWGWX!!#))+'++-335 #!))"- "(( d[[++ !O :%)j8#++D(IF ##..|/?/?/A/G/G/IJyy5( ^^EzzW$%%e,$ ))44V< !YY2 ##Wm$<=zz!}gllM4F4FG %%00= ^3 ##YK09>>*]] ds s ))v#{{55|..  [[^$&*iiU"R"      / {$?@r )rrA)rNF)rz list[int]rGrNrz dict | None)rBT@TTr rErrFlist[GraphProto]returnzlist[NodeProto]) r%r&r'r staticmethodrr r r\r(r rrrsw( "&2 2  2  22h H Hfg?BB[ r rctt|dz ddD]/}||}|jD]}|j|k(s||fccS1y)Nr rwr!)rrr@r)r graph_pathgidgraphr{s rr8r8sRS_q("b13''F{{d"u}$(2 r ceZdZd dZd dZd dZeddZeddZeddZ e ddZ d dZ d d Z y )DefaultWeightOnlyQuantizerc||_yrrrs rrz#DefaultWeightOnlyQuantizer.__init__s  r c zt|jdk7r td|j\}}|jj}||zdz |z}|jj t jk(r|dz}||z}||z }|dkDrtj|d|fdfd}tj|||fd} tj||dzdzzd} tj||z|j} t| || | ||||jjntj||zdzdzd} tj||zdzdzd} tj||f|j} t| || | ||||jj| | | fS) z24b quantize fp32 weight to int4 using C++ kernels.rz9Current int4 block quantization only supports 2D tensors!r r)rrr r=rt)rrrrr<rrr.rrrrtr rLr ) r fp32weightrRrSr<rUrT padded_rowsrpacked zero_pointrPs rint4_block_quantz+DefaultWeightOnlyQuantizer.int4_block_quants z A %XY Y%% d[[++ :%)j8 ;; # #{'<'< <"aI"Z/K!D(G{VVJ!Wv0F S XXtXy9IF$8agNJXXthz7G7GHF ! FJ D$PTP[P[PhPh XXtd{Q14GDF4(?Q#61"rrB make_tensortobytesr?r@rArr<rMrCrD)rrErFqtyperGb_tensorb_graph b_ndarrayrmrPrQrVrWr7 output_nodesrXrYrZrRrSr[dq_input_namesdq_output_namesmatmul_input_namesmatmul_output_names dq_kwargsdq_node matmul_nodes rquantize_matmulz*DefaultWeightOnlyQuantizer.quantize_matmulsk %)KK$<$<   +BSBS**Q-+G[A'   KKL M6M%%..x8 y 1 $ KKC D6M&*&;&;I&F# ;; # #{'<'< <''2268==5;PQG --88QZAZ[Mkk--hmmh.Fy`f`n`n`prvwG --88Q]A]^M]]EzzW$ $$U+# ""G]#;< ;; # #{'<'< <::a=',, 8J8JKK;;++ --88hmmVdFde ""9>>2##**I;7F"JD$F3KF3KF6N#';;#9#9F< {{))5+/;;+E+E'(![[22"Q(*.))TYY&&   N    /85&llM,>,>?N&||i78O"&**Q-1C!D #';;q>"2 ;;++ KK33MM$55ufllKL_L_Lacg %%inn5##**I;7!"$++2H2HIIkk++"%'-1YYTYY)B   G++//)+15TYY- 0K   + 6 7r ctj|dd}tj|dd}tjtj|tj|kD||}|dz }tj|dk(d||z j j ddjtj}||fS)Nr TrGkeepdimsg ri) rrrwhererrclipastypeint8)rmax_valmin_valabs_maxrquantized_slices rquant_slice_symmetricz0DefaultWeightOnlyQuantizer.quant_slice_symmetricOs&&A5&&A5((266'?RVVG_DDFKKArRYYZ\ZbZbc ((5A:q$,2KLRRTYYZ[]_`gghjhphpqz11r c|jd}t|dzdk7rtj|d}|ddddz|ddddzdzz}|j dS) z2Pack int8 data to int4 and store in uint8 ndarray.rwrrNrr rBr=)rrrrr)r data_flatquant_data_int4s rpack_int8_to_int4z,DefaultWeightOnlyQuantizer.pack_int8_to_int4esoLL$ y>A  " )Q/I$SqS>C/Yqt!t_s5Jq4PQ%%g..r cd}|j|}d}t|jD]\}}||kr||z}||kDs||z}||zdz |z} t|j} | | |<|j|||f} t j || |f|j } |rt j |||fd} n4t j |||fd} t j || |fd}td||D]}t||z|}| dd||ddf}|rtj|\}}ntj|\}}}|| dd||ddf<||z}|| dd||dzddf<|r{dd||dzddf<tj| }d}|stj}| j| } || |fS)zXQuantize ndarray data to int4 using numpy, return (quantized data, scales, zero points).r rjrr=rN) rrrzrrrrtrrrgrrr)r quantize_axisr<rLmknrdimrU scales_shape data_reshaperPquant_data_int8zero_point_int8end_idxslicequantized_slice_int8 scale_slicezero_point_slice_int8r rzero_point_int4s rquantize_ndarrayz+DefaultWeightOnlyQuantizer.quantize_ndarrayos  JJ} %  +FAs= S]"S ,  NQ&:5DJJ' &. ]#||Q1I. 1h*$**=  hh1ay?O hh1ay@O hh8Q'7wGOq!Z(A!j.!,G AgIq1E4N4d4dej4k1$k/EEeLI$k3H0DOAqy!O ,ZA(3F1a1q5k1$ %5J1A; 12)$5FFW8JJ?[O -77r c|jjtjk(sJd|jjrt j nt j}|jd}t||\}}|tjd|gStjj|}t|j }|jj"j%dd} |jj&} | |kr| | k\sJd| dk\r | dz | zdk(sJd| |z|z} |j)|| | |jj\} } } |jD].}|j*|k(s|jj-|ntj.j1|j*d z||j | j3d }tjj5| |j*d z}|j*|jd|j*g}|j6j9||g|jjstj.j1|j*d z|| j | j3d }|j;|j*|j6j9|g tj.j=|d }|| | d}tj.j@ d||jBdg|j*r|j*d znddd|}|gS#t>$rd}YdwxYw)z,Quantize weight data of Gather node to int4.z0Gather only supports QOperator format currently.rz4Gather doesn't have const weight. Skip quantization.rr z&Invalid quantize axis for Gather node.z#Invalid block size for Gather node.r'Tr(r)rG) gather_axisrr<r,r-r.)GatherBlockQuantized)"rrrr.rLrrwrxr7r8r5r6r9r:r;rrrgetr<rrr?rBryrzr>r@rArget_node_attr_valuerrCrD)rrErFr{data_argdata_tensorprotodata_graphproto data_ndarray data_rankrr<quantized_datarPrQr7quantized_data_tensorprotoscales_tensorprotorXzp_tensorprotorrZgather_q4_nodes rquantize_gatherz*DefaultWeightOnlyQuantizer.quantize_gathers0{{'';+@+@@tBtt@$(KK$<$<   +BSBS::a=,;Hk,R)/  # KKN O6M((112BC  **+  ..228Q? [[++ y(]yj-HrJrrHRj1n %Ba%GoJooH&2i? .2.C.C -T[[5M5M/ + %**EzzX%%%,,U3+ &*[[%<%<  ! !E )5,2D2DnF\F\F^`d& ""..99&BRBWBWZcBcd166 1 GYG^G^_ ##**,FHZ+[\{{''![[44 %%6v||[M`M`MbdhN   ~22 3  ' ' . ./? @ ++99$GK '*$  .. " [[^$&*iiU"R"    % K s5 M,, M:9M:cftjd|jd|jdk(r|j ||}nH|jdk(r|j ||}n&tj d|jd|g}tjd|jd|S)r"r#r$rrzUnsupported operator z1 for weight only quantization. Skip quantization.r1)r5r6rr4rrerror)rrErFresultss rr\z#DefaultWeightOnlyQuantizer.quantizes  ( 489 <<8 #**4=G \\X %**4=G LL0>op qfG / {$?@r N)rrI)rkz npt.ArrayLiker`)tuple[np.ndarray, np.ndarray, np.ndarray]r^)r np.ndarrayr`ztuple[np.ndarray, np.ndarray])rrr`r)rrr`r) rrrrNr<rNrLrOr`z0tuple[np.ndarray, np.ndarray, np.ndarray | None]) r%r&r'rrorrarrrrrr\r(r rrgrgs ,DWr&&22//58585858 58 : 5858n> @r rgc eZdZ ddZddZy)NVAWQWeightOnlyQuantizerc||_yrrrs rrz!NVAWQWeightOnlyQuantizer.__init__rr c ddlm}tj d|j j}|||j j|}tj d|S#t$rtdtddwxYw) z Perform nvidia_awq quantization using ModelOpt's int4 quantize function. Args: model (ModelProto): The ONNX model to quantize. Returns: ModelProto: The quantized ONNX model. r)r\zlPlease ensure that the 'modelopt' package is installed. Please install it using pip install nvidia_modelopt.zXmodelopt is not installed. Please install it using pip install nvidia_modelopt. Exiting.Nz#Starting nvidia_awq quantization...)rpr:"Completed nvidia_awq quantization.) modelopt.onnx.quantization.int4r\rirjr5r6rr:rp)rmodel quantize_int4rrquantized_models r quantize_awqz%NVAWQWeightOnlyQuantizer.quantize_awq s  Q  9:{{:: ( #{{==$0   89+  ~ j   s A))!B N)rrR)rModelProto | strr`r)r%r&r'rrr(r rrrs* !r rc xeZdZdZdddddej dddf d dZd dZdZdZ d Z y) MatMul4BitsQuantizera4 Target node: QOperator node: QDQ nodes: MatMul MatMulNBits DeQuantizeLinear -> MatMul Gather GatherBlockQuantized Gather, Gather, Gather (optional) -> DequantizeLinear Perform 4b quantization of constant weights for target nodes. If algo_config.quant_format is QOperator: - nodes are replaced by the corresponding QOperator nodes. - quantized weights are stored in the contrib ops. If algo_config.quant_format is QDQ: - the quantized weight is stored in a standard onnx node. For MatMul, it is DequantizeLinear. For Gather, it is the three Gathers, one for quantized data, one for scales and one for optional zero points. - The nodes are replaced by the corresponding QDQ nodes. - currently Gather is not supported in QDQ because Gather does not support int4 yet. Note: - for quantized gather, the memory usage of "DequantizeLinear + Gather" is the same as the original Gather during runtime. Therefor it is not recommended. r7FNc j|g}t|trttj|n t||_t|tr|nd|_||_||_||_ t||_ |r t|nd|_ d|_ | t|||||| } | |_| j dk(rt#|j|_ y| j dk(rt%|j|_ y| j dk(rt'|j|_ yy)Nr<rLrMrrrrDrKrX)ryr"r r9loadr model_pathr<rLrMrnodes_to_excludenodes_to_includenode_quantizerrI algo_configrrrgr) rrr<rLrMrrrrrrs rrzMatMul4BitsQuantizer.__init__Bs  #! 4>uc4JYtyy/0PYZ_P` #-eS#9%t$(, #$4 59I$4 5t"  6%)-)%9% K'  E )"89I9I"JD   " "i /".inc_dataloaders5--(8(8(P(PQK#Dj $s8;NrMzstart to quantize model with z algorithm...r,r) rtn_quantizer0)r weight_configr9) gptq_quantizer; blocksizer=r>r?rw n_samples)rr dataloaderz$complete quantization of model with z algorithm.r()rMrrrr5r6.neural_compressor.adaptor.ox_utils.weight_onlyrr0rrrr;r<r=r>r?)rrrZweight_only_node_configrrrrs` rint4_quant_algoz$MatMul4BitsQuantizer.int4_quant_algos  !     *'+':':F# $"&"?"?"A$$..  3I;mLM   S#//66F8 %)-)Ddoo$**JZJZ5DJ & T!%!1!1!:!:F: "&"2"2"="=F; !%!1!1!:!:F:  ,,00F5M#'#3#3#>#>F< "$F; ')J&)-)Ddoo$**JZJZ5% DJ  :9+[QRr c||jjdvrD|jjg}|jjt j k(r|jjdd|jjt jk(sd|jjvr{|jj}|D]\}|jdvs|jdks"tjd|jj|jd^|j||jj!y|jjdk(rtj#d |j$j'|j(|jjn |j(|_tj#d t+|j|_|jj!y t-j.d ddl}tj:|j<tj:dk\sJd|j?y#t0$r)}t3j4|d t7d |d}~wwxYw)N)rDrKr-r r)Nzai.onnxr,zThe opset of the input model is under 21 and doesn't support int4 data type. Force to update it to opset 21, but the generated model may not be a valid model.rXz%Processing nvidia_awq quantization...rneural_compressor.zLneural-compressor is not correctly installed. Please check your environment.rz2.3.2zGRequire neural-compressor >= 2.3.2 to support weight only quantization!) rrrrerrr.set_opset_importrmr opset_importr0rr5warningrclean_initializersr6rrrr importlib import_module Exceptionloggingr RuntimeErrorrparse __version__r)rrFr opseters rprocesszMatMul4BitsQuantizer.processs'    % %); ;::++-.K,, 0E0EE ++OQ?,, ?8tO_O_OtOtCt#zz668 )E||'<<QSASp 33ELL"E *  " "; / JJ ) ) +    ' '< 7 KK? @,,99$(OO$;   DJ KK< ="4::.DJ JJ ) ) + ''(;< %==!2!>!>?7==D YX Y  "  1g&"b s#J J;$J66J;)rrr<rNrLrOrMrPrzlist[str] | Nonerr#rr$rzWeightOnlyQuantConfig | None)rFr_) r%r&r'__doc__rr.rrrrrr(r rrr.s,"%)-1 **7;9=48'M'M'M 'M # 'M+'M5'M7'M2'MR.` -S^/#r rc&|jdvS)N)true1)lower)rs rort_convert_str_to_boolr s ;;=M ))r cD|jd\}}|t|fS)N:)rrN)skeyrs rparse_key_value_pairr s!JC E ?r c tjd}|jddd|jddd|jd d d td |jddtgdd|jddtd|jdd dddt dd gd|jdd td|jddd d !|j d "|jd#d$td gd%&|jd'd$td d()|jd*d+td+d,gd-|jd.td$d/d0gd12|jd3td$d d45|jd6d7}|jd8td9d:;|jd|jd?td d@dAgdBC|jdDtdEdF;|jS)GNa Blockwise int4 quantization for MatMul 2D weight matrices. A weight matrix is partitioned into into blocks, where each block is a continguous subset inside each column. Each block is quantized into a set of 4b integers with a scaling factor and an optional offset. ) descriptionz --input_modelTzPath to the input model file)requiredhelpz--output_modelzPath to the output model filez --block_sizeFr[zBlock size for quantization)r#defaultrr$z--quant_methodr%)r%hqqrtngptqrXuWthe algorithm used to quantize weight, rtn and gptq leverage Intel® Neural Compressor)r%rchoicesr$z--bitsrBz#the target bits to represent weight)r%rr$z --symmetric?zWIndicate whether to quantize the model symmetrically, symmetric is not supported by hqq)r#r%constnargsrr)r$z--accuracy_levelzAccuracy level of the 4-bit quantized MatMul computation. Refer to the MatMulNBits contrib op's 'accuracy_level' attribute for details (https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#commicrosoftmatmulnbits).)r#rr$z-vz --verbose store_true)r#action)rz--nodes_to_exclude+zBSpecify the nodes to be excluded from quantization with node names)r,rr#r%r$z--nodes_to_includezKSpecify the specific nodes to be included from quantization with node names)r,rr#r$z--quant_formatr.rmzQuantFormat {QOperator, QDQ}QOperator format quantizes the model with quantized operators directly.QDQ format quantize the model by inserting DeQuantizeLinear before the MatMul.z--op_types_to_quantizerrzPop_types_to_quantize {MatMul, Gather}. Operators to quantize. Default is MatMul.)rr,r)r$z --quant_axeszKey-value pairs in op_type:axis_to_quantize separated by space.Specify the axis to quantize for an op. Default {MatMul:0, Gather:1}Example: --quant_axes MatMul:0 Gather:1)rr,r#r$rXz-Arguments specific to nvidia_awq quantizationz--calib_dataset_namerz/Name of the calibration dataset for nvidia_awq.)rr%r$z--tokenizer_dirzPath of the tokenizer dir.)rr#r$z--calibration_methodawqawq_clipz]^ Spq  ? g  !#    )    # K   4     r __main__rzfile z already existsr&zAsymmetric is not supportted by hqq, will force to symmetric=FalseF)r<rFrrr%rr')rr()r<rrXzFQOperator is not applicable to nvidia_awq. overriding the value to QDQr0rr1)r\rqr^rpz!Unsupported quantization method: )rrMrrrT)rcr_r`ztuple[TensorProto, GraphProto])W __future__rr2rr rosrr numpy.typingtypingnptr9 onnx.onnx_pbrrrr packagingronnxruntime.capi._pybind_stater r calibrater onnx_modelr quant_utilsrr basicConfigINFO getLoggerr%r5rr*r6rArIrRrrr8rgrrrr r7argsrsetLevelDEBUG input_modelinput_model_path output_modeloutput_model_pathrrtuplerpathexistsrr symmetric quant_methodr rrr<rF quant_configrMr.rmrpcalib_dataset_namerqr^rrrquantrsave_model_to_filer(r rrYs#   GG[,!8OW^WcWcd   8 $YY64B0% 50%f(4(V(-#8(-Vs#!6s#l4f f Rood ((XY#Y#x*  l^ z