[tracking] Client-side Graphics Pipeline (EGFX): wire the existing decode core into the connector and client stack

[glamberson]

Summary

IronRDP has a working client-side EGFX (MS-RDPEGFX Graphics Pipeline) decode core in ironrdp-egfx, but nothing above it was connected: the connector never advertised graphics-pipeline support, and neither ironrdp-client nor ironrdp-web opened the graphics channel or rendered what came back, so IronRDP clients couldn't use the Graphics Pipeline at all. I'm opening this to track wiring it end to end. The three foundational pieces are already filed as PRs (see the pipeline table): the client-side surface compositor (#1460), the session-side render that drains it (#1461), and native-client registration of the graphics channel (#1462). This issue tracks those, the remaining codec-coverage and negotiation-correctness work, and the coordination with the DVC and client refactors in flight. The sequencing below is a suggested cut, not a fixed plan.

Why this is now high priority

Recent GNOME Remote Desktop (reproduced on 49 and reported on 50) removed its legacy RDP graphics paths and now requires the Graphics Pipeline. A client that doesn't advertise EGFX is rejected during capability exchange:

[RDP] Client did not advertise support for the Graphics Pipeline, closing connection
[rdp_peer_handle_state_demand_active]: [CONNECTION_STATE_CAPABILITIES_EXCHANGE_DEMAND_ACTIVE] Capabilities() callback failed

So this isn't an optional codec feature anymore; it's the requirement for connecting to a large and growing class of servers (every modern GNOME desktop, and the direction of travel is EGFX-mandatory more broadly). Remmina hit the identical wall (Remmina#3302), and #1446 is the concrete report. Both ironrdp-web and the native ironrdp-client are affected.

A personal note

Since my products are server-focused, I haven't worried too much about the client side, but the need's becoming acute.

What already exists (the hard part)

ironrdp-egfx/src/client.rs (GraphicsPipelineClient) is a complete state machine: capability negotiation (parses V8 through V10.7), full surface lifecycle (Create/Delete/Map/Reset), the frame loop with FrameAcknowledge, and it processes every GfxPdu. It natively decodes AVC420 and Uncompressed, and codec dispatch for ClearCodec (#1175) and RemoteFX Progressive (#1443) is in review. The decode engine turns over; it's just not connected to anything.

What is missing

1. Connector: advertise the pipeline. The client core data never sets RNS_UD_CS_SUPPORT_DYNVC_GFX_PROTOCOL in earlyCapabilityFlags. Without it the server never opens the Microsoft::Windows::RDS::Graphics channel, so the decode core is never reached. This is feat(connector)!: expose SUPPORT_DYN_VC_GFX_PROTOCOL early-cap flag for EGFX clients #1237. (Related invariant: grd also rejects a client that advertises graphics but not 32-bit colour depth, so WANT_32_BPP_SESSION must be set at the same time.)
2. Client/web: open the channel. Neither ironrdp-client nor ironrdp-web registers GraphicsPipelineClient as a DVC. ironrdp-egfx is currently a dependency of ironrdp-server only.
3. Client/web: render the output. There's no path from the handler's BitmapUpdate callback into the client's framebuffer/canvas. ironrdp-web currently renders only legacy surface commands.
4. Codec coverage for real servers. grd sends non-AVC codecs, so making its output renderable needs ClearCodec (feat(egfx): add ClearCodec client-side decode dispatch #1175) and Progressive (feat(egfx): wire RemoteFX Progressive decode into WireToSurface2 dispatch #1443). The surface-composition PDUs (SolidFill, SurfaceToSurface) and the surface cache (SurfaceToCache/CacheToSurface/EvictCacheEntry) are now applied by the compositor in feat(egfx): composite client surface commands into pixel buffers #1460 rather than forwarded.
5. Advertise only what we can decode. The server picks the highest capability set the client advertises (MS-RDPEGFX 3.2.5.19). Advertising a codec the client can't decode (e.g. AVC444 without an AVC444 decoder) makes the server send it and produces a blank screen. The CapsAdvertise must match real decode capability. (This is the same negotiation-correctness area as fix(egfx): preserve AVC_DISABLED during negotiation #1455.)

What is already in the pipeline

PR/issue	Author	What it is	State
#1460	Lamco	client-side EGFX surface compositor (items 3, 4-composition/cache)	OPEN, filed 2026-07-20
#1461	Lamco	session-side render: drain compositor into DecodedImage (item 3)	OPEN, filed 2026-07-20, stacked on #1377+#1460
#1462	Lamco	native-client registration of the graphics DVC (item 2)	OPEN, filed 2026-07-20, stacked on #1377/#1460/#1461
#1237	GlassOnTin	connector SUPPORT_DYN_VC_GFX_PROTOCOL early-cap flag (item 1)	OPEN, rebased (per GlassOnTin 2026-07-20)
#1175	Lamco	ClearCodec client decode dispatch (item 4)	OPEN, mergeable
#1443	truebest	Progressive decode into WireToSurface2 dispatch (item 4)	OPEN, mergeable
#1455	digyear	preserve AVC_DISABLED during negotiation (item 5)	OPEN
#1197, #1057, #1099	Lamco	ProgressiveDecoder, egfx crate + server (merged)	done
#1158	Lamco	server-side multi-codec EGFX (context)	open

Suggested order

1. Connector flag first (feat(connector)!: expose SUPPORT_DYN_VC_GFX_PROTOCOL early-cap flag for EGFX clients #1237). Nothing can be tested end to end until the server opens the graphics channel. This is the gate (GlassOnTin's PR, rebased).
2. Rendering foundation (filed). The client-side GraphicsPipelineClient becomes a surface compositor (feat(egfx): composite client surface commands into pixel buffers #1460); the session drains it into the decoded image and emits graphics updates (feat(session): render the EGFX graphics pipeline output #1461); the native client registers the graphics DVC (feat(client): register the graphics pipeline dynamic virtual channel #1462). These are stacked on the DVC access-API refactor in flight (fix(dvc)!: replace DVC wrappers with typed accessors #1377: typed accessors), plus a by-type mutable accessor added in feat(session): render the EGFX graphics pipeline output #1461. Native client first; the browser rendering path follows the web-architecture decision (see coordination note below).
3. Codec coverage (feat(egfx): add ClearCodec client-side decode dispatch #1175 ClearCodec, feat(egfx): wire RemoteFX Progressive decode into WireToSurface2 dispatch #1443 Progressive) so a real grd session renders more than composition and uncompressed content, plus H.264 decoder wiring in the native client so AVC renders.
4. CapsAdvertise correctness so the advertised set matches the decoders present (coordinate with fix(egfx): preserve AVC_DISABLED during negotiation #1455).
5. Robustness: flow-control queueDepth, optional QoE frame ack, HiDPI scaled-output maps (grd 50 shipped HiDPI), surface-memory policy, and EGFX/DisplayControl coordination on resize (MS-RDPEDISP 1.3 restarts the pipeline via EGFX surface commands when EGFX is in use, not deactivate-reactivate).

Coordination with refactoring underway

This work touches the connector, the DVC layer, the session, and the client crates, all of which have active or recently landed refactors. Verified state (2026-07-20):

• DVC access API is mid-refactor (fix(dvc)!: replace DVC wrappers with typed accessors #1377, uchouT, refactor(dvc)!, open/mergeable). feat(dvc): expose dynamic channel accessors #1368 (merged 06-23) exposed dynamic-channel accessors; fix(dvc)!: replace DVC wrappers with typed accessors #1377 replaces the wrapper types with typed accessors and makes DynamicVirtualChannel private. Post-fix(dvc)!: replace DVC wrappers with typed accessors #1377 the access shape is ActiveStage::get_dvc::<T>() -> Option<DynamicChannelRef<'_, T>> (plus get_dvc_by_channel_id / ..._mut), the processor trait is DvcClientProcessor, and the old channel_processor_downcast_ref is gone. Registration is unchanged (DrdynvcClient::with_dynamic_channel(...)). ironrdp-egfx's GraphicsPipelineClient already implements DvcProcessor + DvcClientProcessor, and fix(dvc)!: replace DVC wrappers with typed accessors #1377 does not touch ironrdp-egfx, so no companion egfx change is required; the wiring's only fix(dvc)!: replace DVC wrappers with typed accessors #1377-sensitivity is that its access call uses get_dvc::<GraphicsPipelineClient>() instead of a downcast. The wiring PR should therefore be built on fix(dvc)!: replace DVC wrappers with typed accessors #1377 (or land after it); coordinate merge order with uchouT.
• Session no longer depends on ironrdp-connector (enforced). fix(session)!: reduce dependency on ironrdp-connector #1419 + fix(session)!: remove ironrdp-connector dependency #1435 (merged) removed that dependency and ci(xtask): add dependency-invariant guard for ironrdp-session #1438 added a CI guard for it. The EGFX wiring is connector-independent (it goes through DrdynvcClient + the handler), which keeps it on the right side of the guard; any session-side glue must not reach for connector types.
• Client config already refactored. refactor(client): drive RdpClient configuration from the PropertySet #1389 + refactor(client): drive viewer config through ConfigBuilder #1391 (merged 06-29/30) moved the native client to a PropertySet/ConfigBuilder shape; native wiring targets that. Add agentic RDP CLI and localhost CI workflow #1289 (agent CLI + ironrdp-client split, touches rdp.rs) is open but stale (review-required, untouched since 2026-05-28), so lower near-term collision risk; watch for revival. Switch ironrdp-client/ironrdp-web/ffi to the new CredSSP API; migrate ironrdp-web error mapping #1431 (an issue, not a PR) is the future client/web/ffi CredSSP migration on the same files.
• The web client architecture is itself an open question. The current ironrdp-web may be superseded by a WASM-first client (see ironrdp-web becomes unresponsive when remote framebuffer updated rapidly #790), so the browser rendering integration should land against whichever web architecture settles rather than the current ironrdp-web internals. The native client and the protocol/decode layers are architecture-neutral and can proceed regardless.
• feat(dvc): create channels with assigned id #1416 (DVC reservation API, uchouT) is server-side only; not on this surface.

Validation

grd requires this and reproduces it reliably. We have a grd 49 rig locally and strausmann has offered a grd 50 instance (#1446) to validate a wiring branch end to end.

---

[glamberson]

Progress update.

Validation against gnome-remote-desktop 49: with the foundation PRs (#1460, #1461, #1462) plus the connector flag (#1237), the client now completes the connection to grd and reaches an active session, past the capability-exchange rejection that #1446 reported. It is no longer rejected for not advertising the Graphics Pipeline.

One prerequisite surfaced during validation that was not in the original plan: the connector did not answer the connect-time Bandwidth Measure, and FreeRDP-based servers such as grd block waiting for it, stalling the connection before graphics ever start. Fixed in #1465. That is an independent connector bug rather than EGFX-specific, but it is on the path to a working grd session.

Remaining: getting rendered frames flowing (the graphics channel opening and surface commands applied). In progress; I will keep this updated.

[glamberson]

#1443 (truebest's Progressive WireToSurface2 dispatch) is the last-mile codec for the grd path here. Reviewed and validated against gnome-remote-desktop; grd sends the desktop as RemoteFX Progressive, and this decodes it cleanly. Endorsed on the PR, awaiting maintainer review + CI approval.

[glamberson]

Progress update.

#1175 merged (714dce46). With #1174 and #1197 already in, the codec side of the client pipeline is complete: ClearCodec and Progressive RFX both have decoders upstream, and ClearCodec is now wired into the EGFX client's WireToSurface1 dispatch.

Two corrections landed in the final review round from mamoreau-devolutions. Both are in code introduced by #1174 rather than in the dispatch, so they matter to anyone already building against the decoder:

• Glyph hits no longer require the destination rectangle to match the cached entry's dimensions. Per MS-RDPEGFX 4.1.1.5 a glyph is a dimensionless linear pixel stream, so any rectangle of the same area is valid; the check now compares pixel count.
• ResetGraphics no longer resets the ClearCodec decoder. Per 3.3.5.14 it only resizes the Graphics Output Buffer; CLEARCODEC_FLAG_CACHE_RESET (2.2.4.1) drives cache lifetime and resets the V-bar cursors only, leaving the glyph cache intact.

Still open on this line:

• feat(egfx): composite client surface commands into pixel buffers #1460 surface compositing, feat(session): render the EGFX graphics pipeline output #1461 session rendering, feat(client): register the graphics pipeline dynamic virtual channel #1462 native-client channel registration, the three foundation PRs from this issue's original plan.
• feat(connector)!: expose SUPPORT_DYN_VC_GFX_PROTOCOL early-cap flag for EGFX clients #1237 (GlassOnTin), the SUPPORT_DYN_VC_GFX_PROTOCOL early-capability flag.
• feat(egfx): wire RemoteFX Progressive decode into WireToSurface2 dispatch #1443 (truebest), Progressive WireToSurface2 dispatch, which is the last-mile codec for the grd path.
• fix(connector): answer connect-time Bandwidth Measure to unblock FreeRDP servers #1465, the connect-time Bandwidth Measure reply. Not EGFX-specific, but it sits on the path to a working GNOME Remote Desktop session and is currently under review.

Refs #1446.

[glamberson]

Progress update: the foundation PR is in.

#1460 merged (9cd36952). ironrdp-egfx now holds a client-side surface compositor: persistent RGBA8888 surfaces plus a bitmap cache, every surface command applied into them, surfaces mapped onto the graphics output, and the changed regions drained per EndFrame via GraphicsPipelineClient::drain_output(). That closes the gap this issue was opened for, where the client decoded WireToSurface1 bitmaps but forwarded SolidFill, SurfaceToSurface, the bitmap cache and MapSurfaceToOutput to no-op callbacks and so never held the pixel state they operate on.

A Copilot review round before it landed is worth recording here, because the findings apply to anyone building a compositor against this protocol rather than to our code specifically. Surfaces and cache slots are both keyed by u16, and rectCount and destPtsCount are u16 counts, so three separate paths let a small payload amplify into an unbounded allocation. All three are now closed: a 256 MiB cap across surfaces, cache and pending output; dirty regions recorded during a frame with pixels copied once at EndFrame rather than per command; and the cached tile no longer cloned on every CacheToSurface. ResetGraphics also discards deltas the consumer has not drained, since their regions were clipped against the previous output and would be painted into the new one.

Remaining in this line. #1461 (session-side render) and #1462 (client DVC registration) are both rebased onto current master with the duplicate compositor commit dropped now that #1460 has landed, so they carry only their own change and are ready to review. They remain stacked on uchouT's #1377, which is still open. Beyond those, #1237 (GlassOnTin, connector early-cap flag) and #1443 (truebest, Progressive) are unchanged.

[glamberson]

One gap this issue should have listed and did not: Codec1Type::Planar decode is unwired on the client.

ironrdp-graphics has had an RDP6 Planar decoder all along at rdp6/bitmap_stream/decoder.rs, and it is already used in anger by ironrdp-session/src/fast_path.rs for the 32bpp compressed RDP6_BITMAP_STREAM path. The EGFX client's WireToSurface1 dispatch handles Avc420, ClearCodec and Uncompressed, then drops Planar into the catch-all arm and forwards it to on_unhandled_pdu. That is the same decode-core-exists-but-is-not-wired shape this issue was opened for, and the same situation ClearCodec was in before #1175.

#1498 makes it visible rather than theoretical: an IronRDP server can now send Planar frames via send_planar_frame, and an IronRDP client will drop them.

The wiring looks small. ironrdp-egfx already depends on ironrdp-graphics and already imports ClearCodecDecoder from it, so the decoder is a use away, and decode_clearcodec is the pattern to mirror. The one new piece is pixel format: the RDP6 decoder emits RGB24 while BitmapUpdate carries RGBA, so it needs a 3-to-4 byte expansion rather than the channel swap ClearCodec uses. fast_path.rs notes the decoder writes top-down row-major, which the orientation handling should preserve.

I have not filed anything for this yet. Flagging it here so the tracking list reflects it.

[glamberson]

Filed as #1507.

Two things worth recording here beyond the link, since both were decisions rather than mechanics.

Alpha is deliberately synthesized, not taken from the stream. An RDP6_BITMAP_STREAM can carry an alpha plane, signalled by the NA bit in the FormatHeader ([MS-RDPEGDI] 2.2.2.5.1), and IronRDP's decoder discards it by emitting RGB24. That reads like a fidelity loss until you check how EGFX carries opacity, which is not through the Planar payload at all: it uses a separate RDPGFX_CODECID_ALPHA (0x000C) command holding an ALPHACODEC_BITMAP_STREAM ([MS-RDPEGFX] 2.2.4.3), whose stated purpose is to specify the opacity of each pixel. So a Planar command contributes color only and opaque alpha is correct. The reasoning is in a comment at the conversion site so it does not later get "fixed" into a bug.

destRect is exact for Planar. 2.2.2.1 gives the field both roles, target point and bitmap dimensions, and makes it a bounding rectangle only for AVC420, AVC444 and AVC444V2. Worth knowing for anyone wiring the remaining codecs, since the AVC paths genuinely do need the bounding-rectangle treatment and the others do not.

Also checked and worth noting for the tracking list: Planar is not capability-gated. Unlike the AVC codecs it has no negotiation flag, so a client cannot decline it and a server may send it unprompted. That is what made the gap reachable rather than theoretical.

Remaining in this line is unchanged: #1461 and #1462 ready and stacked on #1377, plus #1237 and #1443.

[glamberson]

#1507 merged (66c8a81b), so Codec1Type::Planar now decodes on the client rather than falling through to on_unhandled_pdu. The gap noted above is closed.