[oi-dev] Virtual address space and Firefox

[Alan Coopersmith]

On 3/12/21 11:58 PM, Carsten Grzemba via oi-dev wrote:
> It seems that this are raised by MOZ_ASSERTION because the JS-engine 
> (spidermonkey) expects that  addresses for JS Values are not in the upper memory 
> area:
> 
> ptr must be a valid user-mode pointer, with the top 16 bits clear

Yes - spidermonkey expects to be able to store its own data in the top bits of
the pointer, which has known issues on Solarish kernels:
https://bugzilla.mozilla.org/show_bug.cgi?id=577056

You can read more about what spidermonkey is doing at:
https://developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey/Internals#javascript_values
https://doar-e.github.io/blog/2018/11/19/introduction-to-spidermonkey-exploitation/#jsvalues-and-jsobjects

> In solaris-userland Oracle has placed a patch but only for Sparc: 
> sparc-47bit-VA-space.patch
> This use a mapfile with
> RESERVE_SEGMENT spidermonkey_reserve {
>        VADDR = 0x800000000000;
>        SIZE = 0xffff7fffffff0000;
> };
> Illumos ld don't know the option RESERVE_SEGMENT, but it is for Sparc how I 
> already stated.

This is the second version of the mapfile we've used - the first was:
CAPABILITY {
         SF += ADDR32;
};
but that limits a 64-bit program to a 32-bit address space, and Firefox really
needs more than 4gb of address space these days, so we had to find a better
solution.

Fortunately, we already had added RESERVE_SEGMENT to the mapfile syntax and
ELF program headers to block out a part of the address space, and ensure that
the system wouldn't map anything in that range, even if ASLR happened to
randomly pick an address there, so we used that instead:
https://blogs.oracle.com/solaris/virtual-address-reservation-in-solaris-113-v2
https://docs.oracle.com/cd/E37838_01/html/E36783/gjpky.html#OSLLGgjpte

> Do IllumOS use the whole 64bit address space and not only the expected 47bit? Or 
> I am completly wrong here in the maze of physical and virtual address spaces?

The address model is different than Linux - it's effectively just 47 bits, but
they're in different subsets of the 64-bit address space than Linux uses.

To quote from one of the engineers who worked on this:
"On x86-64 they rely on the fact that Mac OS X, Windows and Linux all put the
userspace/kernelspace boundary at the VA hole. In Solaris, instead, we still
dynamically restrict it to some arbitrary (depending on amount of memory and
other factors) boundary line above the VA hole."
Another engineer just pointed to:
https://en.wikipedia.org/wiki/X86-64#Virtual_address_space_details

This led to a workaround of putting this line in /etc/system:
set _userlimit=0x7fffc0000000
until the RESERVE_SEGMENT change was available.

But then we also changed the x86 kernel in Solaris 11.4 to make that the default
userlimit on x86 systems, making us match the other OS'es and no longer need to
do anything special to deal with this on x86 systems, only SPARC.

You can see this change in the address space layout diagrams for the AMD64 ABI
between the 11.3 & 11.4 docs:
https://docs.oracle.com/cd/E53394_01/html/E61689/fcowb.html#SSFDGfcpaf
https://docs.oracle.com/cd/E37838_01/html/E66175/fcowb.html#SSFDGfcpaf

	-alan-