Fusion Pattern Pass Mechanism【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/ge1. What Problems Do These Passes SolveWhen GE compiles a model, the input is a computation graph. Each node in the graph is an operator, and the edges between nodes represent which operator a Tensor flows from to which operator.The goal of Fusion Pattern Pass is straightforward:find a small structure in the graph that matches a rule, then replace that structure with another equivalent structure.For example, if there is aMatMul Addsegment in the graph:a ----\ MatMul ----\ b ----/ Add ---- out c ----------------/If the target hardware can perform the same computation in one go withGEMM, it can be replaced with:a ----\ b ----- GEMM ---- out c ----/Another example: if the graph hasAdd(x, 0), its output is equivalent tox, so it can be directly replaced withx. Such optimizations dont change the model semantics, but can reduce the number of operators, reduce intermediate Tensor reads/writes, or transform the graph into a form thats easier for the backend to execute efficiently.GE provides two common interfaces:InterfaceApplicable ScenariosIntuitive UnderstandingPatternFusionPassMatch a subgraph segment, then replace with another subgraph segmentFind a local graph of this shape, then replace it entirelyDecomposePassMatch a single operator, then replace with a subgraph composed of multiple operatorsSplit a complex operator into several basic operatorsPython and C implementations differ, but the underlying mechanism is the same: GE runs passes during the compilation phase to complete matching, filtering, replacement, and reconnection.Development Guides:Python Fusion Pass Development GuideC Fusion Pass Development Guide2. How a PatternFusionPass ExecutesA singlePatternFusionPassexecution can be broken into five steps:Define pattern - Match in target graph - Filter by conditions - Generate replacement - Replace and reconnect edges2.1 Define patternApatternis a very small template graph used to express what I want to find in the real graph.TakeAdd(x, 0)as an example, the pattern only needs to express:External input x ----\ Add ---- pattern output Constant input 0 ----/Here external input is not a fixed real node, but a placeholder. During matching, GE will map the Tensors connected to the outside of this structure in the real graph to this placeholder.2.2 Match in target graphGE will search the real graph for isomorphic structures using the pattern. It can be understood as:The types of regular operators in the pattern must match the operator types in the real graph.Data edges in the pattern must have corresponding connections in the real graph.The input and output boundaries of the pattern must be complete so the real graph remains connected after replacement.After successful matching, GE generates aMatchResult. It records the real nodes, real edges, and actively captured Tensors in this match.2.3 Filter with MeetRequirementsTopological matching only indicates the shape looks right, not necessarily can be replaced.For example,Add(x, Const)topology can match all input plus constant structures, but only when the constant value equals 0 can it be replaced withx. This judgment should be placed inMeetRequirements.MeetRequirementsreturns:true: This match satisfies the conditions and can proceed to replacement.false: Skip this match and continue searching for the next one.2.4 Generate replacement graph with ReplacementReplacementreturns another small graph used to replace the matched real subgraph.If the pattern isAdd(x, 0), the replacement can just return inputx:x ---- replacement outputIf the pattern isMatMul Add, the replacement can returnGEMM:a ----\ b ----- GEMM ---- replacement output b ----- GEMM ---- replacement output c ----/2.5 Replace and ReconnectGE will delete the matched old subgraph, insert replacement, then reconnect external consumers to replacement according to patterns declared input/output boundaries.The most important here is boundary. If boundaries are written incorrectly, even if pattern can match, it may not be able to safely replace.3. Pattern Boundary RulesBoundary rules are the most error-prone part when writing Pattern Pass. You can first remember one sentence:Pattern must completely explain where this subgraph receives external inputs, and which outputs still need to be passed to external use after replacement.3.1 Input BoundaryAny Tensor from outside the matching subgraph must be represented with input placeholder in pattern.For example, to matchAdd(x, 0),xcomes from outside, constant0is created inside pattern:Data/Input ----\ Add Const(0) ------/During matching,Data/Inputcan correspond to any upstream output in real graph.3.2 Output BoundaryAny Tensor that will still be used by outside of subgraph after replacement must be pattern output.For example, pattern is:X ---- A ---- outIf onlyAs output is used by outside in real graph, then just declareAas pattern output.IfXs output is also used by nodes outside pattern, must declareXs output too:X ---- A ---- out0 | out1Otherwise after replacement, external nodes still want to useXs output, but replacement doesnt provide output port for this Tensor, graph will disconnect. GE will try to reject such incomplete boundaries during matching phase.3.3 Self-contained ConstraintFor normal nodes inside pattern, if one of its outputs is not declared as pattern output, then all consumers of this output must also be inside pattern.Can check with following questions:Will this Tensor still be used by external nodes after replacement?If yes, has it been declared as pattern output?If no, are its consumers all inside pattern?3.4 Input Count Must Be ExactNormal operator nodes input count needs to match real graph. If an operator in real graph has 3 inputs, corresponding node in pattern must also have 3 inputs. Even if some inputs sources are not cared about, still need to use input placeholders to fill.3.5 Unsupported Pattern ContentsPattern matching focuses on data topology, not suitable for expressing all graph structures. During development should avoid using in pattern:ContentReasonControl edgesPattern matcher doesnt match by control dependenciesSubgraphsDoesnt support nested subgraph matchingNodes with dynamic input count or dynamic output countCannot determine fixed input/output boundaries during matching3.6 Multi-output PatternA pattern can have multiple outputs. Multi-output is not multiple patterns, but one match exposes multiple output Tensors.If just want to support multiple topologies, like simultaneously supportMatMul AddandBatchMatMulV2 Add, should define multiple patterns.4. Common Extension Points4.1 CaptureTensorSometimesMeetRequirementsorReplacementneeds to read the real node corresponding to some intermediate Tensor in pattern, for example readingMatMuls output description or attributes.Then can capture this Tensor when defining pattern. After successful matching, retrieve it fromMatchResultby capture order.Pythonpatternwriting will automatically capture visited external inputs andreturn-ed pattern outputs; if want to read intermediate Tensors not returned as outputs, still need to use explicit composition and callPattern.capture_tensor.Typical uses:Check dtype、shape、format inMeetRequirements.Read original node attributes inReplacement, write to new node.Print hit location, convenient for confirming match results.Reference samples:C capture tensor samplePython capture tensor sample4.2 PatternMatcherConfigBy default, pattern mainly matches topology and operator types. Some scenarios want matcher to be stricter, for example:Const values in pattern must match Const values in real graph.IR attributes and values declared in pattern must match real graph.Then can enablePatternMatcherConfig.Common switches:ConfigurationEffectEnableConstValueMatch/enable_const_value_matchMatch Const valuesEnableIrAttrMatch/enable_ir_attr_matchMatch IR attributes and valuesIf judgment logic is simple, strict and stable, can put into matcher configuration; if need tolerance, dtype normalization, multiple condition combinations, usually putting inMeetRequirementsis clearer.Reference samples:C PatternMatcherConfig samplePython PatternMatcherConfig sample5. How to Understand DecomposePassDecomposePassis a more special replacement: it doesnt need to first define a pattern graph, but directly declares I want to handle which operator types.For example, to splitConv2Dwithgroups 1into multiple normalConv2D:Grouped Conv2D | v Split(input) Split(filter) Conv2D * N ConcatExecution flow is:Find nodes by op type - MeetRequirements judge if this node needs split - Replacement generate replacement subgraphDecomposePasssuits:Target is single operator.Whether to replace mainly determined by this operators attributes.Replacement will expand this operator into multiple operators.Reference samples:C DecomposePass samplePython DecomposePass sample6. Pass Execution StagePass needs to specify execution stage when registering. Stage determines what graph state pass can see, also determines whether replacement needs to do shape derivation itself.Mechanism StagePython EnumC EnumUsage RecommendationBefore InferShapePassStage.BEFORE_INFER_SHAPECustomPassStage::kBeforeInferShapeMost commonly used. Replacement will enter unified shape derivation flow afterwardsAfter InferShapePassStage.AFTER_INFER_SHAPECustomPassStage::kAfterInferShapeReplacement needs to ensure output shape etc. info correct itselfAfter builtin fusionPassStage.AFTER_BUILTIN_FUSION_PASSCustomPassStage::kAfterBuiltinFusionPassUse when want to handle after GE builtin fusion completesAfter original graph optimizationPassStage.AFTER_ORIGIN_GRAPH_OPTIMIZECustomPassStage::kAfterOriginGraphOptimizeUse when want to append custom handling after original graph optimization endsInitial development suggests first choosing before InferShape stage. Only consider after InferShape stage when your judgment must depend on already derived shapes, or replacement itself will explicitly call shape derivation.7. Python and C RelationshipBoth Python and C passes will connect to GEs unified pass scheduling flow.Main differences are in development experience and delivery method:DimensionPythonCIntegration methodLoad.pyfiles or directories at runtime throughASCEND_GE_PY_PASS_PATHCompile into.sothen loaded by GEPattern writingRecommendpatternexpression writing, also compatible with explicit compositionUseEsGraphBuilderexplicit compositionReplacement writingCan return expression, e.g.return inputs[0]ReturnGraphUniqPtrSuitable scenariosRapid development, business-side on-demand integrationProductized delivery, reuse C code, need stronger compile-time controlIf just adding one rule and verifying effect, suggest first write in Python. After rule stabilizes, if have delivery form or performance requirements, then consider C implementation.8. Pre-development ChecklistAnswer these questions before writing pass:Is what to handle a subgraph segment or single operator?If subgraph segment, are all external inputs of pattern declared?Are all outputs still used by outside after replacement declared?Is topology matching sufficient, or need to check dtype、shape、attributes or Const values inMeetRequirements?Is replacement input order consistent with pattern boundaries?Is registration stage appropriate? If running after InferShape, has replacement handled shape derivation?Can useDUMP_GE_GRAPH1to compare graph before and after replacement, confirm rule actually took effect?【免费下载链接】geGEGraph Engine是面向昇腾的图编译器和执行器提供了计算图优化、多流并行、内存复用和模型下沉等技术手段加速模型执行效率减少模型内存占用。 GE 提供对 PyTorch、TensorFlow 前端的友好接入能力并同时支持 onnx、pb 等主流模型格式的解析与编译。项目地址: https://gitcode.com/cann/ge创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考