[OpenIndiana-discuss] Problem with high cpu load (oi_151a)

Thu Oct 20 19:20:32 UTC 2011

Ooops, something went wrong with my attachement. I'll try again...

Regards,
Gernot Wolf

Am 20.10.11 21:09, schrieb Gernot Wolf:
> You mean, besides being quite huge? I took a quick look at it, but other
> than getting a headache by doing that, my limited unix skills
> unfortunately fail me.
>
> I've zipped it an attached it to this mail, maybe someone can get
> anything out of it...
>
> Regards,
> Gernot
>
>
> Am 20.10.11 20:17, schrieb Michael Schuster:
>> Gernot,
>>
>> is there anything suspicious in /var/adm/messages?
>>
>> Michael
>>
>> On Thu, Oct 20, 2011 at 20:07, Michael Stapleton
>> <michael.stapleton at techsologic.com> wrote:
>>> That rules out userland.
>>>
>>> Sched tells me that it is not a user process. If kernel code is
>>> executing on a cpu, tools will report the sched process. The count was
>>> how many times the process was taken off the CPU while dtrace was
>>> running.
>>>
>>>
>>>
>>> Lets see what kernel code is running the most:
>>>
>>> #dtrace -n 'sched:::off-cpu { @[stack()]=count()}'
>>>
>>> #dtrace -n 'profile-1001 { @[stack()] = count() }'
>>>
>>>
>>>
>>> On Thu, 2011-10-20 at 19:52 +0200, Gernot Wolf wrote:
>>>
>>>> Yeah, I've been able to run this diagnostics on another OI box (at my
>>>> office, so much for OI not being used in production ;)), and noticed
>>>> that there were several values that were quite different. I just don't
>>>> have any idea on the meaning of this figures...
>>>>
>>>> Anyway, here are the results of the dtrace command (I executed the
>>>> command twice, hence two result sets):
>>>>
>>>> gernot at tintenfass:~# dtrace -n 'sched:::off-cpu { @[execname]=count()}'
>>>> dtrace: description 'sched:::off-cpu ' matched 3 probes
>>>> ^C
>>>>
>>>> ipmgmtd 1
>>>> gconfd-2 2
>>>> gnome-settings-d 2
>>>> idmapd 2
>>>> inetd 2
>>>> miniserv.pl 2
>>>> netcfgd 2
>>>> nscd 2
>>>> ospm-applet 2
>>>> ssh-agent 2
>>>> sshd 2
>>>> svc.startd 2
>>>> intrd 3
>>>> afpd 4
>>>> mdnsd 4
>>>> gnome-power-mana 5
>>>> clock-applet 7
>>>> sendmail 7
>>>> xscreensaver 7
>>>> fmd 9
>>>> fsflush 11
>>>> ntpd 11
>>>> updatemanagernot 13
>>>> isapython2.6 14
>>>> devfsadm 20
>>>> gnome-terminal 20
>>>> dtrace 23
>>>> mixer_applet2 25
>>>> smbd 39
>>>> nwam-manager 60
>>>> svc.configd 79
>>>> Xorg 100
>>>> sched 394078
>>>>
>>>> gernot at tintenfass:~# dtrace -n 'sched:::off-cpu { @[execname]=count()}'
>>>> dtrace: description 'sched:::off-cpu ' matched 3 probes
>>>> ^C
>>>>
>>>> automountd 1
>>>> ipmgmtd 1
>>>> idmapd 2
>>>> in.routed 2
>>>> init 2
>>>> miniserv.pl 2
>>>> netcfgd 2
>>>> ssh-agent 2
>>>> sshd 2
>>>> svc.startd 2
>>>> fmd 3
>>>> hald 3
>>>> inetd 3
>>>> intrd 3
>>>> hald-addon-acpi 4
>>>> nscd 4
>>>> gnome-power-mana 5
>>>> sendmail 5
>>>> mdnsd 6
>>>> devfsadm 8
>>>> xscreensaver 9
>>>> fsflush 10
>>>> ntpd 14
>>>> updatemanagernot 16
>>>> mixer_applet2 21
>>>> isapython2.6 22
>>>> dtrace 24
>>>> gnome-terminal 24
>>>> smbd 39
>>>> nwam-manager 58
>>>> zpool-rpool 65
>>>> svc.configd 79
>>>> Xorg 82
>>>> sched 369939
>>>>
>>>> So, quite obviously there is one executable standing out here, "sched",
>>>> now what's the meaning of this figures?
>>>>
>>>> Regards,
>>>> Gernot Wolf
>>>>
>>>>
>>>> Am 20.10.11 19:22, schrieb Michael Stapleton:
>>>>> Hi Gernot,
>>>>>
>>>>> You have a high context switch rate.
>>>>>
>>>>> try
>>>>> #dtrace -n 'sched:::off-cpu { @[execname]=count()}'
>>>>>
>>>>> For a few seconds to see if you can get the name of and executable.
>>>>>
>>>>> Mike
>>>>> On Thu, 2011-10-20 at 18:44 +0200, Gernot Wolf wrote:
>>>>>
>>>>>> Hello all,
>>>>>>
>>>>>> I have a machine here at my home running OpenIndiana oi_151a, which
>>>>>> serves as a NAS on my home network. The original install was
>>>>>> OpenSolaris
>>>>>> 2009.6 which was later upgraded to snv_134b, and recently to oi_151a.
>>>>>>
>>>>>> So far this OSOL (now OI) box has performed excellently, with one
>>>>>> major
>>>>>> exception: Sometimes, after a reboot, the cpu load was about 50-60%,
>>>>>> although the system was doing nothing. Until recently, another reboot
>>>>>> solved the issue.
>>>>>>
>>>>>> This does not work any longer. The system has always a cpu load of
>>>>>> 50-60% when idle (and higher of course when there is actually some
>>>>>> work
>>>>>> to do).
>>>>>>
>>>>>> I've already googled the symptoms. This didn't turn up very much
>>>>>> useful
>>>>>> info, and the few things I found didn't apply to my problem. Most
>>>>>> noticably was this problem which could be solved by disabling
>>>>>> cpupm in
>>>>>> /etc/power.conf, but trying that didn't show any effect on my system.
>>>>>>
>>>>>> So I'm finally out of my depth. I have to admit that my knowledge of
>>>>>> Unix is superficial at best, so I decided to try looking for help
>>>>>> here.
>>>>>>
>>>>>> I've run several diagnostic commands like top, powertop, lockstat
>>>>>> etc.
>>>>>> and attached the results to this email (I've zipped the results of
>>>>>> kstat
>>>>>> because they were>1MB).
>>>>>>
>>>>>> One important thing is that when I boot into the oi_151a live dvd
>>>>>> instead of booting into the installed system, I also get the high cpu
>>>>>> load. I mention this because I have installed several things on my OI
>>>>>> box like vsftpd, svn, netstat etc. I first thought that this problem
>>>>>> might be caused by some of this extra stuff, but getting the same
>>>>>> system
>>>>>> when booting the live dvd ruled that out (I think).
>>>>>>
>>>>>> The machine is a custom build medium tower:
>>>>>> S-775 Intel DG965WHMKR ATX mainbord
>>>>>> Intel Core 2 Duo E4300 CPU 1.8GHz
>>>>>> 1x IDE DVD recorder
>>>>>> 1x IDE HD 200GB (serves as system drive)
>>>>>> 6x SATA II 1.5TB HD (configured as zfs raidz2 array)
>>>>>>
>>>>>> I have to solve this problem. Although the system runs fine and
>>>>>> absolutely serves it's purpose, having the cpu at 50-60% load
>>>>>> constantly
>>>>>> is a waste of energy and surely a rather unhealthy stress on the
>>>>>> hardware.
>>>>>>
>>>>>> Anyone any ideas...?
>>>>>>
>>>>>> Regards,
>>>>>> Gernot Wolf
>>>>>> _______________________________________________
>>>>>> OpenIndiana-discuss mailing list
>>>>>> OpenIndiana-discuss at openindiana.org
>>>>>> http://openindiana.org/mailman/listinfo/openindiana-discuss
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> OpenIndiana-discuss mailing list
>>>>> OpenIndiana-discuss at openindiana.org
>>>>> http://openindiana.org/mailman/listinfo/openindiana-discuss
>>>>>
>>>>
>>>> _______________________________________________
>>>> OpenIndiana-discuss mailing list
>>>> OpenIndiana-discuss at openindiana.org
>>>> http://openindiana.org/mailman/listinfo/openindiana-discuss
>>>
>>>
>>> _______________________________________________
>>> OpenIndiana-discuss mailing list
>>> OpenIndiana-discuss at openindiana.org
>>> http://openindiana.org/mailman/listinfo/openindiana-discuss
>>>
>>
>>
>>
>
>
> _______________________________________________
> OpenIndiana-discuss mailing list
> OpenIndiana-discuss at openindiana.org
> http://openindiana.org/mailman/listinfo/openindiana-discuss