[OpenIndiana-discuss] Problem with high cpu load (oi_151a)

Thu Oct 20 20:29:05 UTC 2011

Another try to get /var/adm/messages out. This time I didn't zip it, 
just attached it as messages.txt.

Regards,
Gernot Wolf

Am 20.10.11 21:37, schrieb Gernot Wolf:
> Ok, for some reason this attachement refuses to go out :( Have to figure
> that out...
>
> Regards,
> Gernot Wolf
>
>
> Am 20.10.11 21:20, schrieb Gernot Wolf:
>> Ooops, something went wrong with my attachement. I'll try again...
>>
>> Regards,
>> Gernot Wolf
>>
>>
>> Am 20.10.11 21:09, schrieb Gernot Wolf:
>>> You mean, besides being quite huge? I took a quick look at it, but other
>>> than getting a headache by doing that, my limited unix skills
>>> unfortunately fail me.
>>>
>>> I've zipped it an attached it to this mail, maybe someone can get
>>> anything out of it...
>>>
>>> Regards,
>>> Gernot
>>>
>>>
>>> Am 20.10.11 20:17, schrieb Michael Schuster:
>>>> Gernot,
>>>>
>>>> is there anything suspicious in /var/adm/messages?
>>>>
>>>> Michael
>>>>
>>>> On Thu, Oct 20, 2011 at 20:07, Michael Stapleton
>>>> <michael.stapleton at techsologic.com> wrote:
>>>>> That rules out userland.
>>>>>
>>>>> Sched tells me that it is not a user process. If kernel code is
>>>>> executing on a cpu, tools will report the sched process. The count was
>>>>> how many times the process was taken off the CPU while dtrace was
>>>>> running.
>>>>>
>>>>>
>>>>>
>>>>> Lets see what kernel code is running the most:
>>>>>
>>>>> #dtrace -n 'sched:::off-cpu { @[stack()]=count()}'
>>>>>
>>>>> #dtrace -n 'profile-1001 { @[stack()] = count() }'
>>>>>
>>>>>
>>>>>
>>>>> On Thu, 2011-10-20 at 19:52 +0200, Gernot Wolf wrote:
>>>>>
>>>>>> Yeah, I've been able to run this diagnostics on another OI box (at my
>>>>>> office, so much for OI not being used in production ;)), and noticed
>>>>>> that there were several values that were quite different. I just
>>>>>> don't
>>>>>> have any idea on the meaning of this figures...
>>>>>>
>>>>>> Anyway, here are the results of the dtrace command (I executed the
>>>>>> command twice, hence two result sets):
>>>>>>
>>>>>> gernot at tintenfass:~# dtrace -n 'sched:::off-cpu {
>>>>>> @[execname]=count()}'
>>>>>> dtrace: description 'sched:::off-cpu ' matched 3 probes
>>>>>> ^C
>>>>>>
>>>>>> ipmgmtd 1
>>>>>> gconfd-2 2
>>>>>> gnome-settings-d 2
>>>>>> idmapd 2
>>>>>> inetd 2
>>>>>> miniserv.pl 2
>>>>>> netcfgd 2
>>>>>> nscd 2
>>>>>> ospm-applet 2
>>>>>> ssh-agent 2
>>>>>> sshd 2
>>>>>> svc.startd 2
>>>>>> intrd 3
>>>>>> afpd 4
>>>>>> mdnsd 4
>>>>>> gnome-power-mana 5
>>>>>> clock-applet 7
>>>>>> sendmail 7
>>>>>> xscreensaver 7
>>>>>> fmd 9
>>>>>> fsflush 11
>>>>>> ntpd 11
>>>>>> updatemanagernot 13
>>>>>> isapython2.6 14
>>>>>> devfsadm 20
>>>>>> gnome-terminal 20
>>>>>> dtrace 23
>>>>>> mixer_applet2 25
>>>>>> smbd 39
>>>>>> nwam-manager 60
>>>>>> svc.configd 79
>>>>>> Xorg 100
>>>>>> sched 394078
>>>>>>
>>>>>> gernot at tintenfass:~# dtrace -n 'sched:::off-cpu {
>>>>>> @[execname]=count()}'
>>>>>> dtrace: description 'sched:::off-cpu ' matched 3 probes
>>>>>> ^C
>>>>>>
>>>>>> automountd 1
>>>>>> ipmgmtd 1
>>>>>> idmapd 2
>>>>>> in.routed 2
>>>>>> init 2
>>>>>> miniserv.pl 2
>>>>>> netcfgd 2
>>>>>> ssh-agent 2
>>>>>> sshd 2
>>>>>> svc.startd 2
>>>>>> fmd 3
>>>>>> hald 3
>>>>>> inetd 3
>>>>>> intrd 3
>>>>>> hald-addon-acpi 4
>>>>>> nscd 4
>>>>>> gnome-power-mana 5
>>>>>> sendmail 5
>>>>>> mdnsd 6
>>>>>> devfsadm 8
>>>>>> xscreensaver 9
>>>>>> fsflush 10
>>>>>> ntpd 14
>>>>>> updatemanagernot 16
>>>>>> mixer_applet2 21
>>>>>> isapython2.6 22
>>>>>> dtrace 24
>>>>>> gnome-terminal 24
>>>>>> smbd 39
>>>>>> nwam-manager 58
>>>>>> zpool-rpool 65
>>>>>> svc.configd 79
>>>>>> Xorg 82
>>>>>> sched 369939
>>>>>>
>>>>>> So, quite obviously there is one executable standing out here,
>>>>>> "sched",
>>>>>> now what's the meaning of this figures?
>>>>>>
>>>>>> Regards,
>>>>>> Gernot Wolf
>>>>>>
>>>>>>
>>>>>> Am 20.10.11 19:22, schrieb Michael Stapleton:
>>>>>>> Hi Gernot,
>>>>>>>
>>>>>>> You have a high context switch rate.
>>>>>>>
>>>>>>> try
>>>>>>> #dtrace -n 'sched:::off-cpu { @[execname]=count()}'
>>>>>>>
>>>>>>> For a few seconds to see if you can get the name of and executable.
>>>>>>>
>>>>>>> Mike
>>>>>>> On Thu, 2011-10-20 at 18:44 +0200, Gernot Wolf wrote:
>>>>>>>
>>>>>>>> Hello all,
>>>>>>>>
>>>>>>>> I have a machine here at my home running OpenIndiana oi_151a, which
>>>>>>>> serves as a NAS on my home network. The original install was
>>>>>>>> OpenSolaris
>>>>>>>> 2009.6 which was later upgraded to snv_134b, and recently to
>>>>>>>> oi_151a.
>>>>>>>>
>>>>>>>> So far this OSOL (now OI) box has performed excellently, with one
>>>>>>>> major
>>>>>>>> exception: Sometimes, after a reboot, the cpu load was about
>>>>>>>> 50-60%,
>>>>>>>> although the system was doing nothing. Until recently, another
>>>>>>>> reboot
>>>>>>>> solved the issue.
>>>>>>>>
>>>>>>>> This does not work any longer. The system has always a cpu load of
>>>>>>>> 50-60% when idle (and higher of course when there is actually some
>>>>>>>> work
>>>>>>>> to do).
>>>>>>>>
>>>>>>>> I've already googled the symptoms. This didn't turn up very much
>>>>>>>> useful
>>>>>>>> info, and the few things I found didn't apply to my problem. Most
>>>>>>>> noticably was this problem which could be solved by disabling
>>>>>>>> cpupm in
>>>>>>>> /etc/power.conf, but trying that didn't show any effect on my
>>>>>>>> system.
>>>>>>>>
>>>>>>>> So I'm finally out of my depth. I have to admit that my
>>>>>>>> knowledge of
>>>>>>>> Unix is superficial at best, so I decided to try looking for help
>>>>>>>> here.
>>>>>>>>
>>>>>>>> I've run several diagnostic commands like top, powertop, lockstat
>>>>>>>> etc.
>>>>>>>> and attached the results to this email (I've zipped the results of
>>>>>>>> kstat
>>>>>>>> because they were>1MB).
>>>>>>>>
>>>>>>>> One important thing is that when I boot into the oi_151a live dvd
>>>>>>>> instead of booting into the installed system, I also get the high
>>>>>>>> cpu
>>>>>>>> load. I mention this because I have installed several things on
>>>>>>>> my OI
>>>>>>>> box like vsftpd, svn, netstat etc. I first thought that this
>>>>>>>> problem
>>>>>>>> might be caused by some of this extra stuff, but getting the same
>>>>>>>> system
>>>>>>>> when booting the live dvd ruled that out (I think).
>>>>>>>>
>>>>>>>> The machine is a custom build medium tower:
>>>>>>>> S-775 Intel DG965WHMKR ATX mainbord
>>>>>>>> Intel Core 2 Duo E4300 CPU 1.8GHz
>>>>>>>> 1x IDE DVD recorder
>>>>>>>> 1x IDE HD 200GB (serves as system drive)
>>>>>>>> 6x SATA II 1.5TB HD (configured as zfs raidz2 array)
>>>>>>>>
>>>>>>>> I have to solve this problem. Although the system runs fine and
>>>>>>>> absolutely serves it's purpose, having the cpu at 50-60% load
>>>>>>>> constantly
>>>>>>>> is a waste of energy and surely a rather unhealthy stress on the
>>>>>>>> hardware.
>>>>>>>>
>>>>>>>> Anyone any ideas...?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Gernot Wolf
>>>>>>>> _______________________________________________
>>>>>>>> OpenIndiana-discuss mailing list
>>>>>>>> OpenIndiana-discuss at openindiana.org
>>>>>>>> http://openindiana.org/mailman/listinfo/openindiana-discuss
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> OpenIndiana-discuss mailing list
>>>>>>> OpenIndiana-discuss at openindiana.org
>>>>>>> http://openindiana.org/mailman/listinfo/openindiana-discuss
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> OpenIndiana-discuss mailing list
>>>>>> OpenIndiana-discuss at openindiana.org
>>>>>> http://openindiana.org/mailman/listinfo/openindiana-discuss
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> OpenIndiana-discuss mailing list
>>>>> OpenIndiana-discuss at openindiana.org
>>>>> http://openindiana.org/mailman/listinfo/openindiana-discuss
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> OpenIndiana-discuss mailing list
>>> OpenIndiana-discuss at openindiana.org
>>> http://openindiana.org/mailman/listinfo/openindiana-discuss
>>
>>
>> _______________________________________________
>> OpenIndiana-discuss mailing list
>> OpenIndiana-discuss at openindiana.org
>> http://openindiana.org/mailman/listinfo/openindiana-discuss
>
> _______________________________________________
> OpenIndiana-discuss mailing list
> OpenIndiana-discuss at openindiana.org
> http://openindiana.org/mailman/listinfo/openindiana-discuss
>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: messages.txt
URL: <http://openindiana.org/pipermail/openindiana-discuss/attachments/20111020/fb14ef56/attachment-0001.txt>