[OpenIndiana-discuss] General ZFS questions (Michelle Knight)

[Calum Mackay]

On 13/01/11 14:03, Michelle Knight wrote:
> At the moment, I'm having more problems.
>
> In beginning the copy to the backup section, even with one device on the
> motherboard e-sata and the other on a USB port (a combination which worked
> under Open SOlaris 134) the OpenIndiana is freezing after copying about 200gig
> of data.

hi Michelle, I know you resolved your issue by moving the disks, but if 
you're still seeing it, you might try this, before the system hangs, 
e.g. after it's been copying for a while:

	echo "::memstat" | sudo mdb -k

If you see Kernel memory rising steadily, but the ZFS cache falling, 
over time, with the Kernel memory eventually gobbling up all of your 
memory (and then hanging), then a kernel memory leak would explain the hang.

If so, turn on kmem debugging in /etc/system:

	set kmem_flags = 0xf

and reboot.

Then try the test again, monitoring memstat; when it gets to around 90% 
of memory, force a crash dump:

	sudo reboot -d

[don't wait until it's hung; on some systems you may find it impossible 
to force a crash dump then]

[if you don't have savecore enabled with dumpadm, you will need to run 
savecore manually after reboot]

Uncompress the dump (assuming compression is on in dumpadm)

	sudo savecore -vf /var/crash/host/vmdump.n


Then run mdb's findleaks on the dump:

	echo "::findleaks -v" | mdb unix.n vmcore.n > findleaks.out

It may take quite a while.

Finally, assuming that findleaks finds lots of leaks, try a few 
representative lines and note the BUFCTL address, then use the bufctl 
dcmd to get the stack trace that allocated that memory.

e.g, from findleaks.out:

CACHE             LEAKED           BUFCTL CALLER
	...
ffffff02ce827688   40670 ffffff03065a48d8 bp_mapin_common+0x2cd
	...

then:

	mdb unix.n vmcore.n

	> ffffff03065a48d8::bufctl -v


Armed with the stack trace(s), we should be able to find the bug that's 
leaking memory...


cheers,
calum.

Calum Mackay
Oracle.