fix(sftp): sshj/Cyberduck clients dropped on initial REALPATH/STAT via russh strict-parser SshEncoding::Length on incoming decode

[Yaminyam]

Summary

sshj/Java-based SFTP clients (Cyberduck, and likely others) cannot list any directory against bssh-server. Right after the SFTP handshake, on the client's initial SSH_FXP_REALPATH (type 16) + SSH_FXP_STAT (type 17) of the home directory, the server drops the connection with Session error error=SshEncoding: length invalid. The OpenSSH sftp CLI and FileZilla work against the exact same server, config, user, and directory. Reproduces with bssh-server v2.2.3.

Root Cause (corrects the original hypothesis)

The original report supposed the server mis-encodes the outgoing REALPATH/STAT response (SSH_FXP_NAME / SSH_FXP_ATTRS). A code audit shows that is not the cause:

• SshEncoding: length invalid is not produced anywhere in the SFTP crate. It is russh::Error::SshEncoding(ssh_encoding::Error::Length) (crates/bssh-russh/src/lib_inner.rs:232-233, rendered as "length invalid"), surfaced by bssh at src/server/mod.rs:369-374. It is raised only through the map_err!(...)? calls in russh's incoming packet parser (crates/bssh-russh/src/server/encrypted.rs).
• The SFTP NAME/ATTRS encoders (crates/bssh-russh-sftp/src/protocol/{name,attrs,file,file_attrs}.rs, ser.rs) write byte-correct u32 length prefixes, and the outgoing russh channel framing (crates/bssh-russh/src/session.rs:441-500) is correct. Crucially, the realpath/stat handler output (src/server/sftp.rs:983-1128) is client-independent: an outgoing-encode bug would break OpenSSH and FileZilla too, which contradicts the observation that they work.

So the error is an incoming russh decode: ssh_encoding::Error::Length fires when a length-prefixed field (String::decode / Bytes::decode / u32::decode) declares more bytes than remain in the packet reader, i.e. russh's (now strict) parser reads a field at an offset where sshj's byte layout does not match what russh expects. The most likely site is the CHANNEL_DATA handler's Bytes::decode at crates/bssh-russh/src/server/encrypted.rs:842, but any strict handler in that file qualifies (CHANNEL_REQUEST field reads :944-1183, or an sshj-initiated rekey KEXINIT hitting the name-list limits in crates/bssh-russh/src/helpers.rs:38-40,96-97).

Likely regression window: commit 3bcd56f ("update: sync russh/russh-sftp forks to upstream and unify ssh-key (#207)") introduced strict fixed-layout parsing plus ensure_end (crates/bssh-russh/src/parsing.rs:13-22) and the LimitedString/NameList limits, and touched encrypted.rs, helpers.rs, the whole SFTP crate, and src/server/sftp.rs. (Note: ensure_end returns TrailingData, not Length, so the specific length invalid points at a *::decode overrun, not trailing bytes.)

Proposed Solution

1. Identify the exact overrun site (runtime evidence, static analysis cannot see sshj's bytes). Temporarily log the message id byte and raw payload length just before the match msg in server_read_authenticated (crates/bssh-russh/src/server/encrypted.rs, near :803), or instrument each map_err!(…decode…)? site, then reproduce with Cyberduck/sshj to capture which decode returns Err(Length) and on which SSH message.
2. Fix the parser to accept the sshj-legal message. Depending on the finding: relax an over-strict fixed-layout assumption, correctly consume an optional/variable field sshj includes, or raise/adjust a limit if a rekey name-list is the trigger. Preserve the hardening intent of #207 for genuinely malformed packets while accepting well-formed sshj traffic.
3. Add an interop regression test exercising the REALPATH+STAT-then-OPENDIR sequence an sshj client sends, so this class of client stays covered.

Implementation Notes

• Error definition and surfacing: crates/bssh-russh/src/lib_inner.rs:232-233; src/server/mod.rs:369-374; bssh's SshHandler::Error = anyhow::Error (src/server/handler.rs:250-251) preserves the russh Display.
• Incoming parser (fault domain): crates/bssh-russh/src/server/encrypted.rs:797-1365; explicit Error::Length sites at crates/bssh-russh/src/helpers.rs:38-40,96-97; ensure_end at crates/bssh-russh/src/parsing.rs:13-22.
• Outgoing path (ruled out): crates/bssh-russh/src/session.rs:441-500; SFTP encoders under crates/bssh-russh-sftp/src/protocol/ and ser.rs.
• The SFTP request parser is tolerant (each SFTP packet is independently length-framed in crates/bssh-russh-sftp/src/utils.rs:12-25, and the SFTP stream is decoupled from SSH-packet decoding), so trailing bytes in an SFTP-level REALPATH request cannot cause this; the fault is at the SSH transport layer.
• Reproduction harness: a Java sshj snippet (or Cyberduck) doing connect → REALPATH(".") → STAT is enough to trigger it deterministically; pair it with the instrumentation in step 1.

Acceptance Criteria

• The exact map_err!(…decode…)? site and the offending SSH message from sshj are identified and documented.
• Cyberduck (and a raw sshj client) can connect, list, and transfer against bssh-server without the connection being dropped.
• OpenSSH sftp and FileZilla continue to work (no interop regression), and the #207 hardening remains effective against genuinely malformed packets.
• An automated interop/regression test covers the sshj REALPATH/STAT/OPENDIR sequence.

Related

• Same SFTP subsystem as fix(sftp): sftp.root chroot rejects all non-root path resolution (cd/get/open/stat) because resolve_chroot does not re-root absolute virtual paths #214 (chroot) and perf(sftp): bssh-server throughput ~50% of OpenSSH from 32 KiB channel fragmentation, serial server-side writes, and per-packet copies #187 (throughput). Likely regression from #207.

---

Original Suggestion

Title: SFTP: Cyberduck (sshj) connection dropped on initial REALPATH/STAT — server errors SshEncoding: length invalid

Summary

Cyberduck (and likely other sshj/Java-based SFTP clients) cannot list any directory against bssh-server: right after the SFTP handshake, on the client's initial REALPATH + STAT of the home directory, the server drops the connection with:

Session error  error=SshEncoding: length invalid

The OpenSSH sftp CLI and FileZilla work fine against the exact same server, config, user, and directory — so this is specific to the request/response pattern that sshj-based clients use.

Client-side symptom (Cyberduck)

디렉터리 sion의 목록을 얻어오는데 실패하였습니다. (127.0.0.1 – SFTP)
Broken transport; encountered EOF. The connection attempt was rejected.
The server may be down, or your network may not be properly configured.

Server-side log (bssh-server run -vv)

INFO  bssh::server::handler: Starting SFTP session user=sion peer=127.0.0.1:54766 home=/Users/sion
DEBUG bssh::server::sftp: Creating SFTP handler user=sion chroot=None cwd=/Users/sion
INFO  bssh::server::sftp: SFTP session initialized user=sion version=3
DEBUG bssh_russh_sftp::protocol: packet type 16      # SSH_FXP_REALPATH
DEBUG bssh_russh_sftp::protocol: packet type 17      # SSH_FXP_STAT
DEBUG russh::server: Connection closed with error
DEBUG bssh_russh_sftp::server: sftp stream ended
ERROR bssh::server: Session error error=SshEncoding: length invalid

The connection dies immediately after the REALPATH (16) / STAT (17) exchange — before any OPENDIR/READDIR. So the server appears to mis-encode the REALPATH/STAT response (an SSH_FXP_NAME / SSH_FXP_ATTRS packet) such that the outgoing SSH channel-data length is invalid, and russh's transport encoder aborts the connection.

Repro

1. bssh-server run -c config.yaml -D (no chroot: sftp.root unset; auth publickey or password; SFTP protocol v3 negotiated).
2. Connect with Cyberduck (SFTP, 127.0.0.1, the configured user) → fails as above.
3. Connect with OpenSSH sftp or FileZilla to the same server → works (list, get, put, rename, etc. all fine).

Impact

Many GUI/mobile SFTP clients are sshj/JSch-based (Cyberduck, and others). If they all hit this, bssh-server cannot be a drop-in SFTP endpoint for those clients.

Environment

• bssh-server 2.2.3 (bssh-server-macos-aarch64)
• SFTP protocol version 3
• Failing client: Cyberduck (sshj); working clients: OpenSSH sftp, FileZilla

[Yaminyam]

Root-cause investigation

This is very likely the same class of bug that historically bit OpenSSH sftp-server + Dropbear with sshj/JSch clients (PyCharm, Cyberduck). Two pieces of prior art:

1. OpenSSH sftp-server hard cap. It rejects any SFTP message whose declared length exceeds a build-time constant:

/* Maximum packet that we are willing to send/accept */
#define SFTP_MAX_MSG_LENGTH (256 * 1024)
...
if (msg_len > SFTP_MAX_MSG_LENGTH) {
    error("bad message from %s local user %s", client_addr, pw->pw_name);
    sftp_server_cleanup_exit(11);
}

When the SSH transport was throughput-tuned (bigger window/payload), sshj/JSch clients ended up putting a packet-header size larger than the server's max and the server killed the connection.

2. Dropbear rollback (backend.ai-agent 7f33f4f, "Rollback SFTP tput optimization due to PyCharm compat issue"): they had bumped DEFAULT_RECV_WINDOW=262144, RECV_MAX_PAYLOAD_LEN=327680, TRANS_MAX_PAYLOAD_LEN=327680 for throughput, which "PyCharm's SSHJ/JSch libraries and throughput-optimized Dropbear breaks with each other, while other SFTP clients work consistently fine". Fix was to revert those to defaults.

Where this maps in bssh

• crates/bssh-russh-sftp/src/server/mod.rs:43 — max_client_packet_len: 262144 (the analog of SFTP_MAX_MSG_LENGTH).
• src/server/sftp.rs:184 — MAX_READ_SIZE = 261120 (255 KiB); bssh is throughput-optimized (255 KiB reads, request pipelining) — same profile that broke sshj on Dropbear.

Failure sequence (server -vv), same channel id

Subsystem request subsystem=sftp channel=ChannelId(2)
Starting SFTP session user=sion home=/Users/sion
SFTP session initialized version=3
bssh_russh_sftp::protocol: packet type 16      # REALPATH
Received data from client channel=ChannelId(2) bytes=14
No shell_data_tx found for channel, dropping data channel=ChannelId(2)   # <-- see below
bssh_russh_sftp::protocol: packet type 17      # STAT
Received data from client channel=ChannelId(2) bytes=24
No shell_data_tx found for channel, dropping data channel=ChannelId(2)
russh::server: Connection closed with error
bssh_russh_sftp::server: sftp stream ended
ERROR bssh::server: Session error error=SshEncoding: length invalid

Two suspects

1. Channel-data routing / windowing. subsystem_request runs SFTP via russh_sftp::server::run(channel.into_stream(), …) (src/server/handler.rs:1354), but the overridden Handler::data() (src/server/handler.rs:1379) also fires for the same SFTP channel and, finding no shell_data_tx, logs "No shell_data_tx found for channel, dropping data". So the SFTP channel's bytes are handled by into_stream() and by data() (which drops them). Whether or not this double-path corrupts the stream, the interaction between the custom data() override and into_stream()'s window accounting is the most likely source of the transport-level SshEncoding: length invalid. sshj's tighter request cadence (REALPATH immediately followed by STAT) appears to trigger it where OpenSSH sftp/FileZilla do not.

2. Packet-size negotiation, per the Dropbear precedent — if bssh advertises a large SFTP limit / SSH window that leads sshj to frame a packet the transport later can't encode/decode.

Repro

• Server: bssh-server run (no chroot), SFTP v3.
• Cyberduck (sshj) → connection dropped on first listing, client shows Broken transport; encountered EOF.
• OpenSSH sftp / FileZilla against the same server → fine.

Happy to help bisect with an sshj-based minimal client if useful.

[Yaminyam]

Narrowing: not the SFTP request logic — sshj/transport-specific

Tried to reproduce with paramiko (Python) against the same server, including the exact REALPATH → STAT(home) cadence the Cyberduck/sshj session used:

realpath then stat(home): OK
realpath then stat('.'):   OK
stat first:                OK
lstat:                     OK
stat then listdir:         OK
normalize + listdir:       OK

All pass. So the SSH_FXP_REALPATH/SSH_FXP_STAT handling itself is fine — paramiko cannot reproduce it. The failure is specific to sshj/JSch and manifests as a russh transport-level error (SshEncoding: length invalid), not an SFTP-layer decode error. KEX (curve25519 / Ed25519) completes cleanly; the break is in the post-auth channel-data phase.

This points at an sshj-vs-russh interop issue at the SSH binary-packet layer (cipher/MAC/packet-length framing, or the custom Handler::data() override interacting with channel.into_stream() windowing), rather than anything in the SFTP request handlers.

Note: the SSH window/max-packet in the failing session are russh defaults (window=2097152, max_packet=32768), so this is not an over-tuned-window case like the Dropbear precedent — the throughput tuning in bssh is at the SFTP layer (255 KiB reads), not the SSH window.

A faithful minimal repro needs an actual sshj client (Java). Cyberduck is a reliable manual repro in the meantime.

[Yaminyam]

Root cause + fix

The SFTP subsystem never sent the server-side channel EOF/CLOSE when the session ended: it ran russh_sftp::server::run(), which spawned the request loop on a detached task and returned immediately, so the handler had no completion hook (unlike the shell/exec/SCP paths, which send exit_status + eof + close). OpenSSH sftp/FileZilla tolerate the missing server CLOSE; sshj/JSch strictly wait for the server's SSH_MSG_CHANNEL_CLOSE in the close handshake, so it hangs.

Reproduced (scriptable)

Minimal sshj 0.38 client — canonicalize(".") → stat(home) → ls(home) → close():

• before: SFTPClient.close() times out after ~30s
• after fix: session completes and closes in ~1s

(paramiko couldn't reproduce it — it doesn't strictly block on the channel-close handshake — which is why this only surfaced with sshj-based clients like Cyberduck.)

Fix

Proposed in Yaminyam#1 (stacked on #212):

• bssh-russh-sftp: run_with_config runs the loop inline instead of tokio::spawn-ing internally, so the caller can observe completion.
• handler: run the SFTP session on our own task and, when it ends, send handle.eof() + handle.close().

Regression-checked: OpenSSH sftp (ls/get/put), paramiko, and cargo test -p bssh-russh-sftp all fine.

The run()-should-be-awaitable part is worth mirroring upstream in russh-sftp (the fork's parent) — the fire-and-forget spawn leaves callers with no way to clean up the SSH channel after the SFTP session.

Cyberduck re-test against the patched binary pending to confirm it also clears the original Broken transport; encountered EOF symptom.

[Yaminyam]

Resolution: fixed by the russh 0.62.1 upgrade (#212) — it's a russh (SSH channel) issue, not an SFTP one

After isolating with a scriptable sshj repro across russh versions:

build	sshj (canonicalize→stat→ls→close)
russh 0.61.1 (vendored fork and vanilla)	❌ ~30s (SFTPClient.close timeout)
russh 0.62.1 (#212), no extra patch	✅ ~1s

The bug is in the SSH channel-close layer, not SFTP. The SFTP protocol itself worked fine on all builds (REALPATH/STAT/READDIR succeed, sshj gets the full listing). The hang is the SSH channel-close handshake: the client sends SSH_MSG_CHANNEL_CLOSE and waits for the server's reply, which russh 0.61.1 never sends (RFC 4254 violation). sshj/JSch strictly wait for it; OpenSSH/FileZilla don't.

Fixed upstream in russh by 6fc20b28 "Reply with CHANNEL_CLOSE in server handler per RFC 4254" (Eugeny/russh#675) (+ #674 "Forward ChannelMsg::Close to channel before dropping sender"), first in russh 0.62.1. Verified: clean #212 (russh 0.62.1, no other change) resolves it.

So this is resolved by merging #212 (the russh 0.61.1 → 0.62.1 upgrade). No SFTP-layer change is needed:

• The bssh-russh-sftp / handler change I first proposed (fix(sftp): send channel EOF+CLOSE when the SFTP session ends (fixes #215, sshj/Cyberduck hang) #216) is redundant — russh 0.62.1 already replies with CHANNEL_CLOSE.
• The russh-sftp JoinHandle proposal (feat(server): return JoinHandle from run/run_with_config AspectUnk/russh-sftp#93) was aimed at the wrong layer and isn't needed either.

Closing #216 and AspectUnk/russh-sftp#93 accordingly.

[Yaminyam]

Correction + actual root cause (compression, not channel-close) — fixed in #217

My earlier comment claiming this was resolved by the russh 0.62.1 upgrade (#212) was wrong. Retesting a real Cyberduck client against a clean 0.62.1 build reproduced the exact same failure. The 30s channel-close hang I fixed there was a separate sshj-0.38 issue; it never reproduced Cyberduck's symptom, so I mis-attributed the fix. Apologies for the noise (#216 and the russh-sftp PR remain correctly closed as redundant).

Real root cause: russh delayed-zlib (zlib@openssh.com) stream desync

Full russh=trace on the failing Cyberduck session showed the negotiated algorithms:

negotiated: cipher: chacha20-poly1305@openssh.com,
            client_compression: ZlibOpenSSH, server_compression: ZlibOpenSSH

The handshake and first SFTP requests (INIT / REALPATH / STAT) all succeed, then the connection dies with SshEncoding: length invalid on the next inbound packet — because compression activates after auth and russh's flate2 stream desyncs a few packets later, so the following packet's length prefix is decoded from corrupted plaintext.

It's compression, not Cyberduck-specific and not the SFTP/channel-close layer:

client	compression	result
OpenSSH sftp (default)	none	✅
FileZilla / paramiko	none	✅
OpenSSH sftp -C	zlib@openssh.com	❌ SshEncoding: length invalid
Cyberduck 9.5	zlib@openssh.com	❌ same

One-command repro against an unpatched server:

sftp -C -P 2223 user@host      # -> "SshEncoding: length invalid", closed by remote

Reproduces on both russh 0.61.1 and 0.62.1.

Fix (#217)

Server advertises only none compression (russh::Preferred { compression: [NONE], ..DEFAULT }), so clients fall back to the uncompressed transport — matching the Dropbear / OpenSSH sftp-server defaults used in the container SFTP path anyway. Verified: sftp -C now negotiates compression: none and a live Cyberduck 9.5 session connects and browses fine.

The underlying russh delayed-zlib desync should still be reported/fixed upstream; #217 is the safe server-side workaround.