hexagon: add support for basic and extended Op profiling by max-krasnyansky · Pull Request #22269 · ggml-org/llama.cpp

max-krasnyansky · 2026-04-23T01:28:44Z

Overview

The op-batching rewrite as part of #21705 disabled our original sync-profiler support.
This PR adds fully async profiler properly integrated with op-batching. The overall op queing logic has been updated to allocate additional shared buffers that (if enabled) the NPU now populates with the profiling data for each op.

The overall "user-interface" is the same as before where GGML_HEXAGON_PROFILE=1 enables Hexagon profiling output into the main GGML log. We now support additional modes: 1-basic (usec and cycles), 2-extended (usec, cycles + PMU). See docs/backends/snapdragon/README.md for details.

Additional information

This PR also includes a new post-processing tool that generates nice reports from the logs.
Here is an example:

M=../gguf/LFM2.5-1.2B-Instruct-Q4_0.gguf D=HTP0 PROF=2 ./scripts/snapdragon/adb/run-completion.sh -f ../sample_prompt_1024.txt -st -n 64 2>&1 | tee hex-log

./scripts/snapdragon/ggml-hexagon-profile.py hex-log

# Profile Report (Top 100 Ops sorted by max-usec)

| Op             | Dims                                     | DTypes                       | Max usec | Avg usec | Max Cycles | Avg Cycles  | Count |
| -------------- | ---------------------------------------- | ---------------------------- | -------- | -------- | ---------- | ----------- | ----- |
| MUL_MAT        | 8192:2048 x 8192:255 -> 2048:255         | q4_0 x f32 -> f32            | 6153     | 6126.27  | 12998214   | 12940843.53 | 15    |
| MUL_MAT        | 2048:8192 x 2048:255 -> 8192:255         | q4_0 x f32 -> f32            | 5493     | 5450.40  | 11602839   | 11513942.20 | 30    |
| MUL_MAT        | 8192:2048 x 8192:256 -> 2048:256         | q4_0 x f32 -> f32            | 5461     | 5052.17  | 11536246   | 10672342.00 | 30    |

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Not for the core C/C++ code; Used Gemini to generate initial version of the post-proc tool

Add separate profile descriptors.

opsync is no longer needed since the profiler is fully async now. opmask name was confusing and opstage is more accurate.

Also move all the PMU setup stuff out of the hex-utils since it's not inteded for normal use.

On older devices getting PMU counters is expensive so it's now optional.

max-krasnyansky · 2026-04-23T20:10:41Z

@ggml-org/maintainers need the second approval here (any input is welcome too of course).

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

max-krasnyansky · 2026-04-23T21:13:42Z

@lhez @CISC sorry, need re-approvals after applying suggestions.
That new feature (ie making approvals stale) is a bit of a time waster, makes sense for general contribs but @ggerganov maybe it can be disabled if the PRs authors are maintainers?

) * hexagon: restore HTP_OPMASK_QUEUE * hexagon: honor OPMASK_SKIP_COMPUTE in hmx-matmul * hex-prof: restore op profiling * hex-prof: enable PMU * hexagon: simplify and improve op-queuing with full profiling support Add separate profile descriptors. * hexagon: remove opsync and rename opmask into opstage opsync is no longer needed since the profiler is fully async now. opmask name was confusing and opstage is more accurate. * hexagon: refactor opbatch queue handling * hexagon: add iface hooks for enabling profiler from the host Also move all the PMU setup stuff out of the hex-utils since it's not inteded for normal use. * hexagon: make profiler mode configurable On older devices getting PMU counters is expensive so it's now optional. * hexagon: add support for setting profiler pmu events from env * hexagon: simplify profiler output (no need to print buffs, etc) * hexagon: simplify pmu counter formating * hexagon: add a simple profile post-proc tool * hex-prof: add support for reading logs from stdin * hexagon: document GGML_HEXAGON_PROFILE * hex-prof: update default width for dims field * hex-prof: fix linter warnings and errors * Update ggml/src/ggml-hexagon/htp/htp-ops.h Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update scripts/snapdragon/ggml-hexagon-profile.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Trivikram Reddy <tamarnat@qti.qualcomm.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* hexagon: restore HTP_OPMASK_QUEUE * hexagon: honor OPMASK_SKIP_COMPUTE in hmx-matmul * hex-prof: restore op profiling * hex-prof: enable PMU * hexagon: simplify and improve op-queuing with full profiling support Add separate profile descriptors. * hexagon: remove opsync and rename opmask into opstage opsync is no longer needed since the profiler is fully async now. opmask name was confusing and opstage is more accurate. * hexagon: refactor opbatch queue handling * hexagon: add iface hooks for enabling profiler from the host Also move all the PMU setup stuff out of the hex-utils since it's not inteded for normal use. * hexagon: make profiler mode configurable On older devices getting PMU counters is expensive so it's now optional. * hexagon: add support for setting profiler pmu events from env * hexagon: simplify profiler output (no need to print buffs, etc) * hexagon: simplify pmu counter formating * hexagon: add a simple profile post-proc tool * hex-prof: add support for reading logs from stdin * hexagon: document GGML_HEXAGON_PROFILE * hex-prof: update default width for dims field * hex-prof: fix linter warnings and errors * Update ggml/src/ggml-hexagon/htp/htp-ops.h Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * Update scripts/snapdragon/ggml-hexagon-profile.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> --------- Co-authored-by: Trivikram Reddy <tamarnat@qti.qualcomm.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

max-krasnyansky and others added 16 commits April 22, 2026 14:23

hexagon: restore HTP_OPMASK_QUEUE

95cfa37

hexagon: honor OPMASK_SKIP_COMPUTE in hmx-matmul

3c2e5d7

hex-prof: restore op profiling

93bca53

hex-prof: enable PMU

b2bd6fc

hexagon: simplify and improve op-queuing with full profiling support

4bd50ac

Add separate profile descriptors.

hexagon: remove opsync and rename opmask into opstage

8b1bbba

opsync is no longer needed since the profiler is fully async now. opmask name was confusing and opstage is more accurate.

hexagon: refactor opbatch queue handling

3681c6a

hexagon: add iface hooks for enabling profiler from the host

c3d46c7

Also move all the PMU setup stuff out of the hex-utils since it's not inteded for normal use.

hexagon: make profiler mode configurable

fc9479d

On older devices getting PMU counters is expensive so it's now optional.

hexagon: add support for setting profiler pmu events from env

9b01190

hexagon: simplify profiler output (no need to print buffs, etc)

85396bf

hexagon: simplify pmu counter formating

64eab79

hexagon: add a simple profile post-proc tool

2ac34a0

hex-prof: add support for reading logs from stdin

7b04136

hexagon: document GGML_HEXAGON_PROFILE

e51b594

hex-prof: update default width for dims field

7590ccb

max-krasnyansky requested a review from a team as a code owner April 23, 2026 01:28

github-actions Bot added documentation Improvements or additions to documentation script Script related python python script changes ggml changes relating to the ggml tensor library for machine learning Hexagon labels Apr 23, 2026

hex-prof: fix linter warnings and errors

50c7b26

lhez approved these changes Apr 23, 2026

View reviewed changes

CISC approved these changes Apr 23, 2026

View reviewed changes

Comment thread ggml/src/ggml-hexagon/ggml-hexagon.cpp

Comment thread ggml/src/ggml-hexagon/htp/htp-ops.h Outdated

Comment thread scripts/snapdragon/ggml-hexagon-profile.py Outdated

max-krasnyansky and others added 2 commits April 23, 2026 14:09

Update ggml/src/ggml-hexagon/htp/htp-ops.h

6821aaf

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

Update scripts/snapdragon/ggml-hexagon-profile.py

2788e00

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

lhez approved these changes Apr 23, 2026

View reviewed changes

max-krasnyansky merged commit 5d2b52d into ggml-org:master Apr 23, 2026
5 of 50 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

hexagon: add support for basic and extended Op profiling #22269

hexagon: add support for basic and extended Op profiling #22269
max-krasnyansky merged 19 commits into
ggml-org:masterfrom
qualcomm:hexagon-profiler

max-krasnyansky commented Apr 23, 2026

Uh oh!

max-krasnyansky commented Apr 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

max-krasnyansky commented Apr 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

max-krasnyansky commented Apr 23, 2026

Overview

Additional information

Requirements

Uh oh!

max-krasnyansky commented Apr 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

max-krasnyansky commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

max-krasnyansky commented Apr 23, 2026 •

edited

Loading