model: GraniteSWAForCausalLM / GraniteMoeSWAForCausalLM

[gabe-l-hart]

Overview

This Pull Request mirrors huggingface/transformers#47179 to support forthcoming Granite models that will incorporate interleaved Sliding Window Attention with Attention Sinks.

Additional information

• Model weights are not available just yet, but they exist internally and will eventually appear here

• Some of the models in this family will have per-layer RoPE / NoPE. There is adjacent work to support this in the Granite Switch PR (Granite-Switch Architecture #25107). For the sake of keeping this encapsulated, I replicated the function signature for llama_hparams.has_rope. The logic behind the answer is different than the Granite Switch logic, so this will need to be rectified depending on merge order.

• This has been verified on internal checkpoints.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES

AI Disclosure

AI was used to create this PR. I manually reviewed all outputs and authored all commits myself. See stats below:

git-ai-stats

╔══════════════════════════════════════════════════════════╗
║           GIT AI USAGE ANALYSIS                          ║
╚══════════════════════════════════════════════════════════╝

📊 COMMITS BY AGENT

--- Aggregate ---
Commits                        |      Count
---------------------------------------------
IBM Bob                        |          5
IBM Bob, OpenCode + Qwen3.6-35b |          2
---------------------------------------------
TOTAL                          |          7

📊 COMMITS BY USAGE TYPE

--- Aggregate ---
Commits                        |      Count
---------------------------------------------
none                           |          0
draft                          |          0
full                           |          7
---------------------------------------------
TOTAL                          |          7

📈 LINES OF CODE BY AGENT

--- Aggregate ---
Agent                     |    Commits |  Additions |  Deletions
------------------------------------------------------------
IBM Bob                   |          5 |         85 |          8
IBM Bob, OpenCode + Qwen3.6-35b |          2 |        415 |          0
------------------------------------------------------------
TOTAL                     |          7 |        500 |          8

📈 LINES OF CODE BY USAGE TYPE

--- Aggregate ---
Usage Type           |    Commits |  Additions |  Deletions
-------------------------------------------------------
none                 |          0 |          0 |          0
draft                |          0 |          0 |          0
full                 |          7 |        500 |          8
-------------------------------------------------------
TOTAL                |          7 |        500 |          8

[pwilkin]

@gabe-l-hart and here I was about to ask you about https://huggingface.co/ibm-granite/granite-swash-2b/tree/main :) seems like at least one model from the new family made it through :)

[gabe-l-hart]

@pwilkin I didn't even catch that one, so it was even stealth within the house. I'll make sure everything is lined up for that model too (and get red tests green)

[gabe-l-hart]

The saga of Alex and Max is alive and well with granite-swash-2b:

model      : /Users/ghart/models/ibm-granite/granite-swash-2b/granite-swash-2B-BF16.gguf
ftype      : BF16
modalities : text

available commands:
  /exit or Ctrl+C     stop or exit
  /regen              regenerate the last response
  /clear              clear the chat history
  /read <file>        add a text file
  /glob <pattern>     add text files using globbing pattern



> Tell me a story about a developer and their dog
Once upon a time in a quaint little town, there lived a developer named Alex. Alex was an avid dog lover, and his dog, Max, was always by his side. Max was a golden retriever with a heart of gold and a spirit that could light up any room.

One sunny afternoon, while Alex was working on his latest project, he noticed that Max had been barking non-stop. He called his assistant, Sarah, over and asked if she had seen anything unusual. Sarah shrugged and said, "I don't know, maybe Max's hearing is a bit off today."
...

[gabe-l-hart]

We also have an MoE variant up now: https://huggingface.co/ibm-granite/granite-swash-3b-a600m. Needed a few changes, but this is also supported now.