[nanvix-unstable] PEB scratch pointers use GVA instead of GPA when guest skips page-table generation

[ppenna]

Summary

When the nanvix-unstable feature is enabled, Hyperlight skips guest page-table generation and the guest runs with identity mapping (GVA == GPA). However, the PEB input_stack.ptr and output_stack.ptr fields are always written using scratch_base_gva() (derived from MAX_GVA), while the KVM memory slot for scratch is placed at scratch_base_gpa() (derived from MAX_GPA).

Before PR #1393, MAX_GPA == MAX_GVA == 0xFFFF_FFFF on i686 so this was not observable. After that fix lowered MAX_GPA to 0xFEDF_FFFF (to avoid the KVM APIC access page at 0xFEE00000), scratch_base_gva() and scratch_base_gpa() diverge and the guest reads PEB pointers that target unmapped physical addresses.

Affected code

src/hyperlight_host/src/mem/layout.rs — get_input_data_buffer_gva() and get_output_data_buffer_gva():

fn get_input_data_buffer_gva(&self) -> u64 {
    hyperlight_common::layout::scratch_base_gva(self.scratch_size)
}

pub(crate) fn get_output_data_buffer_gva(&self) -> u64 {
    hyperlight_common::layout::scratch_base_gva(self.scratch_size)
        + self.sandbox_memory_config.get_input_data_size() as u64
}

These are called from write_memory_layout() (lines ~765-782) to populate the PEB pointer fields that the guest reads at boot.

src/hyperlight_host/src/mem/mgr.rs — get_guest_memory_regions() (crashdump, i686-guest path) also uses scratch_base_gva() where it should use scratch_base_gpa() under identity mapping.

Suggested fix

Under #[cfg(feature = "nanvix-unstable")], the PEB pointer helpers should use scratch_base_gpa() instead of scratch_base_gva(), since no page tables translate GVA → GPA for the guest:

#[cfg(feature = "nanvix-unstable")]
fn get_input_data_buffer_gva(&self) -> u64 {
    // nanvix-unstable: guest runs with identity mapping (GVA == GPA),
    // no page tables translate scratch GVA → GPA.
    hyperlight_common::layout::scratch_base_gpa(self.scratch_size)
}

#[cfg(not(feature = "nanvix-unstable"))]
fn get_input_data_buffer_gva(&self) -> u64 {
    hyperlight_common::layout::scratch_base_gva(self.scratch_size)
}

And similarly for get_output_data_buffer_gva() and the crashdump path.

Alternatively, a more general approach would be to add a scratch_base_for_guest(size) helper in hyperlight_common::layout that dispatches to scratch_base_gpa() when nanvix-unstable is enabled and scratch_base_gva() otherwise, so all call sites are correct by construction.

Current workaround

Nanvix currently patches the PEB pointers at boot in the guest kernel (hyperlight_pre_kmain):

let gva_gpa_delta: u64 = (MAX_GVA - MAX_GPA) as u64;
if gva_gpa_delta != 0 {
    unsafe {
        (*peb_ptr).input_stack.ptr -= gva_gpa_delta;
        (*peb_ptr).output_stack.ptr -= gva_gpa_delta;
    }
}

This works but is fragile — any future change that adds more GVA-based pointers to the PEB would silently break the identity-mapped guest.

Context

• Hyperlight PR: don't let scratch region overlap with kvm apic page #1393 (APIC page overlap fix)
• Nanvix PR: [uservm] Model sparse physical memory layout for Hyperlight platform nanvix/nanvix#2082

[syntactically]

With #1385 I understand that Nanvix will use Hyperlight-coordinated page tables and virtual memory layout. If that happens, does it make sense to keep this as VA as it presently is on everything else, or to switch it to PA?

I can definitely see an argument that it's cleaner to fill this structure with PAs anyway, but then the hyperlight_guest code that reads it needs to find the appropriate VA for it.