fix: handle two invalid VirtualAlloc/Free edge cases correctly (#445)

Fixes #421 

Ensures sogen handles two edge cases correctly:

- VirtualAlloc with size 0 and invalid flags now returns
STATUS_INVALID_PARAMETER instead of terminating emulation with an
exception.
- VirtualFree with an invalid free_type now fails cleanly, as per
Windows behavior.

Using test code from #421  before change:

Executing syscall: NtAllocateVirtualMemory (0x18) at 0x180162bb2 via
0x103858074 (KernelBase.dll)
Syscall threw an exception: 18 (0x180162bb2) - Unsupported allocation
type!
Emulation terminated without status!

After:

Executing function: kernel32.dll - VirtualAlloc (0x103703c90) via
(0x14000119a) Valloc.exe
Executing syscall: NtAllocateVirtualMemory (0x18) at 0x180162bb2 via
0x103858074 (KernelBase.dll)
Executing function: kernel32.dll - GetLastError (0x1036e8640) via
(0x1400011a5) Valloc.exe

Terminating normally:

Emulation terminated with status: 0
This commit is contained in:
Maurice Heumann
2025-08-08 16:42:27 +02:00
committed by GitHub

View File

@@ -181,6 +181,12 @@ namespace syscalls
}
auto allocation_bytes = bytes_to_allocate.read();
if (allocation_bytes == 0)
{
return STATUS_INVALID_PARAMETER;
}
allocation_bytes = page_align_up(allocation_bytes);
bytes_to_allocate.write(allocation_bytes);
@@ -244,6 +250,11 @@ namespace syscalls
return STATUS_NOT_SUPPORTED;
}
if (free_type == 0)
{
return STATUS_INVALID_PARAMETER;
}
const auto allocation_base = base_address.read();
const auto allocation_size = bytes_to_allocate.read();