g{ L d dl mZ d dlZd dlZd dlZd dlZd dlZd dlmZ d dl m
Z
d dl Z ddl m
Z
mZmZmZ G d d Z G d d
Z d dd Z d dd Z d dd
ZddZddZd d dZd dZd d dZd Zd Z e dk( r e yy)! ) annotationsN)deque)IntEnum )ModelProtoWithShapeInfoget_producer_consumer_mapsis_fixed_size_tensoroptimize_modelc " e Zd ZdZd Zd Zd Zy)_SupportedOpsCheckera;
Class to process the md file with list of supported ops and caveats for an execution provider.
e.g. /tools/ci_build/github/android/nnapi_supported_ops.md
/tools/ci_build/github/apple/coreml_supported_mlprogram_ops.md
/tools/ci_build/github/apple/coreml_supported_neuralnetwork_ops.md
c n || _ i | _ t | _ t | 5 }|D ]r }|j d s|j
j d }t | dk( sC|d }|d }|j dd }d|v sd|| j |< t d d d y # 1 sw Y y xY w)N| r z
:)
_filename_opsset _ops_seenopen
startswithstripsplitlenreplace)selffilenameflinepieces domain_opcaveats g/var/www/openai/venv/lib/python3.12/site-packages/onnxruntime/tools/mobile_helpers/usability_checker.py__init__z _SupportedOpsChecker.__init__ s !
(^q ??3' !ZZ\ / / 4F6{a'$*1I !'!' != )+39DIIi 0 ^^s B+-B+/ B+B++B4c |j r |j nd}|dz |j z }|| j v }|r| j j | |S )Nai.onnxr )domainop_typer r add)r noder( r" is_supporteds r$ is_op_supportedz$_SupportedOpsChecker.is_op_supported. sN $ SL4<</ DII- NN y ) c g }t | j D ]* }| j | }|s|j | d| , |S )Nr )sortedr r append)r caveatsopr# s r$ get_caveatsz _SupportedOpsChecker.get_caveats8 sH (BYYr]F"Qvh /0 )
r. N)__name__
__module__ __qualname____doc__r% r- r4 r. r$ r r s :( r. r c h e Zd Z G d de Z ddZd dZd Zd
dZy) PartitioningInfoc e Zd ZdZdZdZy)PartitioningInfo.TryWithEP)r )r r N)r5 r6 r7 NOMAYBEYESr9 r. r$ TryWithEPr= C s
r. rA c || _ || _ || _ || _ || _ || _ || _ || _ | | _ |
| _ d| _
d| _ y )Nr ) num_nodesnum_supported_nodesnum_partitionssupported_ops_checkersupported_groupsunsupported_opsnodes_unsupported_due_to_op&nodes_unsupported_due_to_dynamic_input!num_unsupported_nodes_due_to_rankops_with_unsupported_rank
num_subgraphsnum_nodes_in_subgraphs) r rC rD rE rF rG rH rI rJ rK rL s r$ r% zPartitioningInfo.__init__H sc ##6 ,%:" 0 . +F(6\31R.)B& &'#r. c | xj |j z
c_ | xj |j z
c_ | xj |j z
c_ | j j |j | j
j
|j
| xj |j z
c_ | xj |j z
c_ | xj |j z
c_ | j j
|j | xj dz
c_ | xj |j z
c_ y)z]
Merge the information from another PartitioningInfo instance into this one.
r N)
rC rD rE rG extendrH updaterI rJ rK rL rM rN )r others r$ mergezPartitioningInfo.mergec s
%//) E$=$== u 3 33 $$U%;%;< ##E$9$9: ((E,M,MM( 33u7c7cc3 ..%2Y2YY. &&--e.M.MN
a ##u6#r. c | j | j z dz }| j dk( rX|dkD rt j j
S |dkD rt j j S t j j S | j dk( r9|dkD rt j j S t j j S t j j S )Nd r K 2 r )rD rC rE r; rA r@ r? r> )r
pct_supporteds r$ suitabilityz PartitioningInfo.suitabilityu s 004>>ACG
! # r!'11555 #'11777'11444 ! # r!'11777'11444 )),,,r. c
|j | j d| j d| j d| d | j r|j ddj | j D cg c] }t
t | c} d | j D ]L }|j dd j |D cg c] }|j d
|j c} N |j d | j | j r6|j d d j t | j | j j }|rGd
}|j ddj |D cg c] }t j" |z |z c} | j$ r |j d| j$ | j& rT|j d| j& |j dd j t | j( | j* dkD r+|j | j, d| j* d | j | j z dz } | j dk( r|j | d y)| j dk( rU| dkD r|j | d| dd y)| d kD r|j | d | dd y)|j | d | d!d" y)| j d#k( r | dkD r|j | d$| dd% y)|j | d&| j d'| dd( y)c c}w c c}w c c}w )*z
Analyze the partitioning information and log the analysis
:param logger: Logger to use
:param ep_name: Execution provider name to use in the log messages
z partitions with a total of /z nodes can be handled by the z EP.z Partition sizes: [z, ]zNodes in group: ,r z"Unsupported nodes due to operator=z Unsupported ops: z z\ Caveats that have not been checked and may result in a node not actually being supported: z8Unsupported nodes due to input having a dynamic shape=%dz,Unsupported nodes due to rank of input data=z Ops with unsupported rank: r z nodes are in z; subgraphs. Check EP as to whether subgraphs are supported.rU z$ cannot run any nodes in this model.r rV zD should work well for this model as there is one partition covering z.1fz % of the nodes in the model.rW z, may work well for this model, however only zD% of nodes will use it. Performance testing is required to validate.z4 will probably not work will for this model as only z.2fz% of nodes will use it.r zG can be considered for this model as there are two partitions covering z<% of the nodes. Performance testing is required to validate.z1 is not recommended with this model as there are z partitions covering zj% of the nodes in the model. This will most likely result in worse performance than just using the CPU EP.N)inforE rD rC rG joinstrr debugr) namerI rH r0 rF r4 oslineseprJ rK rL rM rN )
r loggerep_name partitiongroupr+ r2 indentr# pct_nodes_using_eps
r$ print_analysisz PartitioningInfo.print_analysis s ""##?@X@X?YYZ[_[i[iZj k++2)4
9
KK&tyyW[WlWl1mWl)#c)n2EWl1m'n&oopq
.. /di9jdi\`T\\N!DII;:Wdi9j0k/l mn / 89Y9Y8Z[\ KK-chhvd>R>R7S.T-UV W,,88: F KKo77wOwVBJJ /& 8wOPQS
6 6 KKJ;;
1 1 KKFtGmGmFno p KK7HfHfAg8h7ij k !
KK../~d>P>P=Q RB B
" 5 5 F L ! # KK7)#GH I
A
%!B& i 23 77SU $b( i KL^_bKc dP P
i STfgjSk l, ,
A
%*._is_fixed_shape_value s3 J '
5(9: : L E-E$E r. r c rO xs } | r j j j j y y N)r1 copyclear)keep_partitionon_group_closed_fnsupported_groupsupported_group_borderrG s r$ close_groupz2_check_partitioning_for_graph..close_group* sP !3 3 Z7I/7ZN ''(<(<(>? ! ! # " ( ( * r. c 3 . K | ] } | y wrx r9 ).0irv s r$ z0_check_partitioning_for_graph..> s GujtefH]^_H`jts T tensor_typeF Transposer' r r ) initializerrc r r+ r r1 r popleftr- allinputtypeHasFieldr shapedimr) attributeintsr* r( remover; )&graphnode_to_producersnode_to_consumersrF rt require_fixed_input_sizesru max_rankr in_degreenodes_to_process nodes_to_process_with_next_groupr+ node_input_edge_countrD num_unsupported_nodes_due_to_op*num_unsupported_nodes_due_to_dynamic_inputrK rH rL r r- is_input_shape_supportedis_rank_supported
node_input
input_rankis_node_supportedconsumerconsumer_node_in_degreerC rE r_ rv rs r| r} r~ rG s& ` ` @@@@@@r$ _check_partitioning_for_graphr s %*$5$56$5qAFF$56L 0 I w',w$
@DHY@Y$5d$; <_` / $ A % # #D ) O U&'#12.()%eO # + > M ? /4w , '')/??E'@#@#uCGujnjtjtGuDu "jj
*
":z*?U?Z?Z?c?cdq?r!$Z
%;%@%@%L%L%R%R%V%V!WJ!H,,1) ) <<; &3t~~a/@/E/E+F+Q % + ^0H ^M^ -- 177=" ##dkkt{{y&QQRSWS_S_R`$ab/14/+:a?:$1Q61)-- QZ0[[\]a]i]i\j.kl 1 $
" "4 ( --&--d3 (( 1$ 7H*..x8 !8 $ $ -d 3*3H*=''1,'*a/$++H5&= (#
4 >N MEJJI)*N '2)! D Ks 7s L;c r;t | j dk( r#t | j dkD r t d ddt | \ d d fd
| ri }|S d }|S )
as
Estimate the partitions the graph will be split into for nodes that is_node_supported_fn returns true for.
The check on whether a node is supported is purely based on the operator type. Additional limitations
(e.g. NNAPI EP only supports 2D Conv) are not checked, so partitions may not be 100% accurate. The limitations
for operators in the partitions are printed so the user can manually check.
:param main_graph: Graph to process
:param supported_ops_checker: Checker with info on supported ops.
:param require_fixed_input_sizes: If True, require that the inputs to a potentially supported node are fixed size
tensors for it to be considered as supported. This requires
onnx.shape_inference.infer_shapes to have been run on the model to populate the
shape information.
If False, shapes are ignored during the check.
:param max_rank: Set if EP has a limitation on the rank of tensors it supports.
:return PartitioningInfo instance with details
r r zURun onnx.shape_inference.infer_shapes on the model to populate the shape information.c | j D ] }|||j < | j D ] }|||j < | j D ] }|||j < y rx )r rc outputru )r value_to_shapevs r$ _update_value_infoz.check_partitioning.._update_value_info sX A%&N166 " A%&N166 " !!A%&N166 " "r. Nc
||j } | | ni }|
t }t | ||
}|r|j | n|}t | }| j D ] }|j |j | j D ]< }|j D ]+ } | j d s| j }
|
||| }- > |S )Ng) ry r r rS r r* rc r+ r r r )r outer_scope_value_infort partitioning_inforu r_ !subgraph_outer_scope_initializersr r+ attrsubgraph _check_graphr r r r r rF s r$ r z(check_partitioning.._check_graph s " -/446J uj 1J # +'*u $, ! $ %
# #D ) !% -00H,I) ,,K - 1 1+2B2B C - JJD ==% #vvH(4 *.OQb)% ' ! r. )r onnx.GraphProtor dict[str, onnx.ValueInfoProto])NN)
r r r z%dict[str, onnx.ValueInfoProto] | Nonert zset[str] | Noner zPartitioningInfo | Nonereturnr; )r ru r+
ValueErrorr )
main_graphrF r r aggregated_partitioning_infor r r r s ``` @@@@r$ check_partitioningr s . !S)>)>%?1%DZ__I]`aIapqq' ,Fj+Q((
5959 1! 1! E1! #21! 3 1!
1! 1!f $0
B[B#f '' bf#f ''r. c L t | }t | j ||| }|S rx )r r r )modelsupported_ops_configr r
supported_opspartition_infos r$ _check_ep_partitioningr s+ ))=>M' ]D]_ghN r. c t j t j }|dz }|j r|}n |j
d }|dz dz dz dz dz }t
| || S )Nznnapi_supported_ops.md toolsci_buildgithubandroidpathlibPath__file__parentexistsparentsr )r r
script_dir local_config config_pathort_roots r$ check_nnapi_partitionsr ss h'..J 88L " %%a( (:5@9LOgg !%6O PPr. c t j t j }||z }|j r|}n |j
d }|dz dz dz dz |z }d}t
| ||| S )Nr r r r apple r )r r config_filenamer r r r r s r$ check_coreml_partitionsr sy h'..J /L " %%a( (:5@7J_\ H !%6OQY ZZr. c j d}d}g }| j D ]e }t | sS|j | |r:|j ddj t | j
|dz
}a|dz
}g g }| j D ]e }t | sS|j | |r:|j ddj t | j
|dz
}a|dz
}g | j sJt | j dk( s2t | t | j k( s|j d | j D ] }t | r|dz
}|dz
} |rA|j d| d| |r(|r|j d ||fS |j d
||fS ) a
Check the shapes of graph inputs, values and graph outputs to determine if they have static or dynamic sizes.
NNAPI does not support dynamically sized values. CoreML does, but it will most likely cost performance.
:param graph: Graph to check. If shape inferencing has been run the checks on values will be meaningful.
:param logger: Optional logger for diagnostic information.
:return: Tuple of List of inputs with dynamic shapes, Number of dynamic values found
r z"Input is not a fixed size tensor: r r z#Output is not a fixed size tensor: ziUnable to check shapes within model. ONNX shape inferencing should be run on the model prior to checking.z Num values with fixed shape=z . Num values with dynamic shape=zModel has dynamic inputs and outputs. Consider re-exporting model with fixed sizes if NNAPI or CoreML can be used with this model.ao Model has dynamically sized inputs but fixed sized outputs.
If the sizes become fixed early in the model (e.g. pre-processing of a dynamic input size
results in a fixed input size for the majority of the model) performance with NNAPI and CoreML,
if applicable, should not be significantly impacted.) r r r1 r_ r` ra r r ru r r+ warning) r rf num_fixed_valuesnum_dynamic_valuesdynamic_inputsr dynamic_outputsovis r$ check_shapesr s N
[[#A& ! !! $ @#a&,,.AY@Z [\ ! # ! O
\\#A& " "1 % A#((3q6<<>BZA[ \] ! # ! S_%9S =PTWX]XcXcTd=d
S
# ! ! # *+;*<<\]o\p q
F - -- O - --r. c
t | }|j t j \
}
fd} |dt }dd}dd} |d| } |d| } |t
j j k7 s:|t
j j k7 s | t
j j k7 r2j t j kD rj d |t
j j k7 xs< |t
j j k7 xs | t
j j k7 S ) Nc f j d| d} | | }j t j k r|j | |j }j d| d|j |t j j k7 rrj d j d d} | | }j t j k r#j d |j | |j }j d| d |j |t j j k7 rj d
|j |j kD r|}j d j d |S )
N Checking Tz Model should perform well with z as is: z--------zHChecking if model will perform better if the dynamic shapes are fixed...FzHPartition information if the model was updated to make the shapes fixed:z) if modified to have fixed input shapes: zPShapes can be altered using python -m onnxruntime.tools.make_dynamic_shape_fixedz================r^ ) r_ getEffectiveLevelloggingINFOrl rY rc r; rA r@ r> rr )
rg checker_funcr r rY partition_info_with_fixed_shapesfixed_shape_suitabilityr rf model_with_shape_infos
r$ check_epzchecker..check_epb s iy)* %)!%&;=VW # # % 5 ) )&' :$002 5gYh{GWGWFXYZ *4488 8^ KK
# KKb c(- %/;.check_nn_coreml s &u.GIpqqr. c t | |d S )Nz!coreml_supported_mlprogram_ops.mdr r s r$ check_mlprogram_coremlz'checker..check_mlprogram_coreml s &u.GIlmmr. zCoreML NeuralNetworkzCoreML MLProgramzJRe-run with log level of INFO for more details on the NNAPI/CoreML issues.)r onnx.ModelProto)
r r r r r r; rA r@ r r r r_ r> )
model_pathrf model_with_shape_info_wrapperr r nnapi_suitabilityr r coreml_nn_suitability coreml_mlprogram_suitabilityr r s ` @@r$ checkerr \ s' $;J$G! 9 O O)56K6Q6Q)R&N&&P !*@Arn %%;_ M#+,>@V#W - 7 7 ; ;; $4$>$>$B$B B '+;+E+E+I+I I
"
"
$w||
3 `a - 7 7 : :: I $4$>$>$A$A A I '+;+E+E+H+H Hr. c |s4t j d }|j t j |j d| d t j 5 }|s2t j | | j z }t | |d |} t | j d | }ddd |S # 1 sw Y S xY w)a
Analyze the provided model to determine if it's likely to work well with the NNAPI or CoreML Execution Providers
:param model_path: Model to analyze.
:param skip_optimize: Skip optimizing to BASIC level before checking. When exporting to ORT format we will do this
optimization..
:param logger: Logger for output
:return: True if either the NNAPI or CoreML Execution Providers may work well with this model.
usability_checkerr z for usability with ORT Mobile.T)use_external_initializers)strictN)
r getLoggersetLevelr r_ tempfileTemporaryDirectoryr r rc r
r resolve)r
skip_optimizerf tmptmp_pathtry_epss r$
analyze_modelr s ""#67 %
KK)J<'FGH $ $ &# ||C(:??:H :x4 P!J*,,D,96B
' N
' Ns
AB;;Cc , t j t j j t
d } | j
dddgdd | j
dd d
| j
d t j d
| j S )Nz3Analyze an ONNX model for usage with the ORT mobile) descriptionz --log_levelrb r_ z
Logging level)choicesdefaulthelpz--skip_optimize
store_truezDon't optimize the model to BASIC level prior to analyzing. Optimization will occur when exporting the model to ORT format, so in general should not be skipped unless you have a specific reason to do so.)actionr
r zProvide path to ONNX model)r r
)
argparseArgumentParserrd pathbasenamer add_argumentr r
parse_args)parsers r$ r r s
$
$
"0iF
/@&Wfg
L 7<<>Z[ r. c t } t j d }| j dk( r |j t j
n}| j dk( r |j t j nN| j dk( r |j t j n |j t j | j j }t || j | y )Nr rb r_ r )
r r r log_levelr DEBUGr WARNINGERRORr r r r )argsrf r s r$ run_analyze_modelr s r% sJ # x x- -`d d^ d d@d Ad 0 d
' d $
d /d dV [( [(/[( $[( [(
[(~ ru 2>[_kn Q [ F.RFR4 $:" z r.