[oi-dev] [developer] [Samba] gencache.tdb: device busy
Rouven WEILER
Rouven_Weiler at gmx.net
Sun Apr 21 10:10:30 UTC 2019
Dear all,
trying to make samba work and getting rid of my error message
[2019/04/01 09:51:41.112485, 0] ../../lib/tdb_wrap/tdb_wrap.c:64(tdb_wrap_log)
tdb(/var/samba/lock/gencache.tdb): tdb_open_ex: tdb_mutex_init failed for /var/samba/lock/gencache.tdb: Device busy
I followed the info of Youzhong. It definitely works. So thankx.
BUT: The OS (OpenIndiana; state of April,19th) is patched against the illumos issue 9959
So there might be some mechanism (maybe MUTEX_MAGIC) that avoids the initialization of the mutex before init.
I am really not deep in all that pthread stuff…
Is there the possiblity that there is a bug in the code base of issue 9959?
Best regards
Rouven
> Am 18.04.2019 um 19:20 schrieb Robert Mustacchi <rm at joyent.com>:
>
> On 4/18/19 10:12 , Youzhong Yang wrote:
>> Little bit confused. The page https://www.openindiana.org/download/ shows
>> latest ISO is dated 2018.10, but 9959 fix was submitted on 11/26/2018, I
>> guess Hipster doesn't have it.
>
> The ISO is just the install image. Generally, after using that to
> bootstrap, then folks will update to latest bits via IPS. As a result,
> the ISO isn't updated (and generally doesn't have to be) quite as often.
>
> Robert
>
>> On Thu, Apr 18, 2019 at 12:38 PM Rouven WEILER <Rouven_Weiler at gmx.net>
>> wrote:
>>
>>> On my system the issue is solved and the sources are patched accordingly.
>>> Thus, the issue seems to be somewhere else.
>>>
>>> Am 18.04.2019 um 18:27 schrieb Rouven WEILER <Rouven_Weiler at gmx.net>:
>>>
>>> concerning openindiana I am using the most recent one. I right now asked
>>> if the issue is solved in the latest version.
>>>
>>> another interesting thing is that older samba versions do not throw EBUSY,
>>> although on the same underlying system.
>>>
>>>
>>>
>>> Am 18.04.2019 um 18:07 schrieb Youzhong Yang <youzhong at gmail.com>:
>>>
>>> That is interesting. I am curious about how pthread_mutexattr_init ends up
>>> returning EBUSY. What verison of Openindiana are you running? is there a
>>> link pointing to the fix that it has for 9959?
>>>
>>> On Thu, Apr 18, 2019 at 11:53 AM Rouven WEILER <Rouven_Weiler at gmx.net>
>>> wrote:
>>>
>>>> I understand and see that it works, but as written below
>>>> pthread_mutexattr_init throws EBUSY, not pthread_mutex_init.
>>>>
>>>> Also Openindiana seems to include the patch for issue 9959 mentioned. I
>>>> will clarify that to make sure.
>>>>
>>>> I just want to make sure that the patch you suggest aims for the issue
>>>> and is not a workaround.
>>>>
>>>>
>>>> Am 18.04.2019 um 16:09 schrieb Youzhong Yang <youzhong at gmail.com>:
>>>>
>>>> I reproduced the issue on a smartos version which does not have the fix
>>>> for https://www.illumos.org/issues/9959.
>>>>
>>>> The pthread_mutex_init() at line 590 returns EBUSY:
>>>> ret = pthread_mutex_init(chain, &ma);
>>>>
>>>> https://git.samba.org/?p=samba.git;a=blob;f=lib/tdb/common/mutex.c;h=8a122d5d6a2ec321bce25cf06062acc808f1e126;hb=HEAD#l590
>>>>
>>>>
>>>> Then I applied the following patch to lib/tdb/common/mutex.c:
>>>>
>>>> # git grep -B 1 -n pthread_mutex_init lib/tdb/common/mutex.c
>>>> lib/tdb/common/mutex.c-590- bzero(chain,
>>>> sizeof(pthread_mutex_t));
>>>> lib/tdb/common/mutex.c:591: ret = pthread_mutex_init(chain,
>>>> &ma);
>>>> --
>>>> lib/tdb/common/mutex.c-599- bzero(&m->allrecord_mutex,
>>>> sizeof(pthread_mutex_t));
>>>> lib/tdb/common/mutex.c:600: ret =
>>>> pthread_mutex_init(&m->allrecord_mutex, &ma);
>>>> --
>>>> lib/tdb/common/mutex.c-693- bzero(&m, sizeof(pthread_mutex_t));
>>>> lib/tdb/common/mutex.c:694: ret = pthread_mutex_init(&m, &ma);
>>>> --
>>>> lib/tdb/common/mutex.c-891- bzero(ptr, sizeof(pthread_mutex_t));
>>>> lib/tdb/common/mutex.c:892: ret = pthread_mutex_init(ptr, &ma);
>>>>
>>>> It works like a charm. Fair enough?
>>>>
>>>> So again I recommend either of the following two:
>>>> - apply 9959 fix to your OS image, or use a newer version which has 9959
>>>> fix
>>>> - bzero pthread_mutex_t struct before calling pthread_mutex_init().
>>>>
>>>> In my environment I applied bzero patch to all pthread_mutex_init() calls
>>>> in the entire samba code base, you know, who knows what is going to happen.
>>>>
>>>>
>>>>
>>>> On Thu, Apr 18, 2019 at 4:29 AM Rouven WEILER <Rouven_Weiler at gmx.net>
>>>> wrote:
>>>>
>>>>> That sounds reasonable!
>>>>> Could you help me to dtrace that in a feasible way? Up to now I am not
>>>>> used to dtrace. Except knowing what is and that its powerful.
>>>>>
>>>>> Am 18.04.2019 um 09:29 schrieb Jason King <jason.brian.king at gmail.com>:
>>>>>
>>>>> It appears that the robust mutexes are mmapped from the .tdb file.
>>>>> EBUSY means that the mutex has already been intialized. It also appears
>>>>> that if the TDB_CLEAR_IF_FIRST flag is _not_ set when opening the tdb, it
>>>>> will try to initialize the mutexes. I’m wondering if the tdb file gets
>>>>> created, the memory image of the initialized mutexes is getting saved, then
>>>>> when it’s opened again, if that flag is not set, it tries to re-initialize
>>>>> the mutexes and fails (because they are already constructed) legitimately
>>>>> returning EBUSY. If the samba code is in fact trying to reinitialize an
>>>>> already initialized mutex (dtrace might be able to help determine this — at
>>>>> this point it’s merely a supposition), that would explain the error.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> From: Rouven WEILER <rouven_weiler at gmx.net> <rouven_weiler at gmx.net>
>>>>> Reply: illumos-developer <developer at lists.illumos.org>
>>>>> <developer at lists.illumos.org>
>>>>> Date: April 18, 2019 at 12:48:48 AM
>>>>> To: Youzhong Yang <youzhong at gmail.com> <youzhong at gmail.com>
>>>>> Cc: Robert Mustacchi <rm at joyent.com> <rm at joyent.com>, Toomas Soome via
>>>>> illumos-developer <developer at lists.illumos.org>
>>>>> <developer at lists.illumos.org>
>>>>> Subject: [developer] Aw: Re: [Samba] gencache.tdb: device busy
>>>>>
>>>>> You are totally right. That's the same I got from the man pages.
>>>>>
>>>>> The responsible code lines of samba also seems to bear no issue and also
>>>>> this issue arises on samba version 4.10.x not on earlier ones.
>>>>> In addition, there seems to be only a problem with
>>>>> illumos/openindiana...
>>>>> That's why I do not know how to get that fixed. Up to now I hadn't time
>>>>> to dtrace.
>>>>>
>>>>> Only thing I got so far: When the mutexattr is already set the return
>>>>> value will result on undefined behaviour.
>>>>> But why only after version 4.10.x on openindiana/illumos?
>>>>>
>>>>> Just for completion here the codelines of samba:
>>>>> https://git.samba.org/?p=samba.git;a=blob;f=lib/tdb/common/mutex.c;h=8a122d5d6a2ec321bce25cf06062acc808f1e126;hb=HEAD#l570
>>>>>
>>>>>
>>>>>
>>>>> Gesendet: Mittwoch, 17. April 2019 um 19:19 Uhr
>>>>> Von: "Youzhong Yang" <youzhong at gmail.com>
>>>>> An: "Rouven WEILER" <Rouven_Weiler at gmx.net>
>>>>> Cc: "Robert Mustacchi" <rm at joyent.com>, "Toomas Soome via
>>>>> illumos-developer" <developer at lists.illumos.org>
>>>>> Betreff: Re: [Samba] gencache.tdb: device busy
>>>>>
>>>>> Unless I am totally wrong, pthread_mutexattr_init() shouldn't return
>>>>> EBUSY:
>>>>>
>>>>> http://src.illumos.org/source/xref/illumos-gate/usr/src/lib/libc/port/threads/pthr_mutex.c#37
>>>>>
>>>>>
>>>>> On Tue, Apr 16, 2019 at 2:45 PM Rouven WEILER <Rouven_Weiler at gmx.net[
>>>>> mailto:Rouven_Weiler at gmx.net <Rouven_Weiler at gmx.net>]> wrote:Hi Robert
>>>>> & Youzhong,
>>>>>
>>>>> I got a response from the Samba-Devs.
>>>>> Would the issue mentioned by Youzhong also hot pthread_mutexattr_t?
>>>>>
>>>>> Concerning dtrace I hadn’t the time yet but I will check for that.
>>>>> Is there maybe a way to trace all forks of smbd? Usually the issue
>>>>> arises after the fork…
>>>>>
>>>>> FIND BELOW THE STATEMENT FROM samba.org[http://samba.org]:
>>>>>
>>>>> This doesn't look exactly the same. In your
>>>>> log below you say:
>>>>>
>>>>> ret = pthread_mutexattr_init(&ma); in line 570
>>>>> I got errorno 16 --> device busy.
>>>>>
>>>>> But the bug report is about needing the value
>>>>> passed to pthread_mutex_init() needing to be
>>>>> zero, not the value passed to pthread_mutexattr_init().
>>>>>
>>>>> Can you just try adding '= {0}' to all the
>>>>> places where we declare:
>>>>>
>>>>> pthread_mutexattr_t ma
>>>>>
>>>>> (i.e. make it say)
>>>>>
>>>>> pthread_mutexattr_t ma = {0};
>>>>>
>>>>> and see if it makes your problem go away ?
>>>>>
>>>>> END OF STATEMENT
>>>>>
>>>>>
>>>>>> Anfang der weitergeleiteten Nachricht:
>>>>>>
>>>>>> Von: Jeremy Allison via samba <samba at lists.samba.org[
>>>>> mailto:samba at lists.samba.org <samba at lists.samba.org>]>
>>>>>> Betreff: Aw: [Samba] gencache.tdb: device busy
>>>>>> Datum: 16. April 2019 um 02:07:05 MESZ
>>>>>> An: Rouven WEILER <Rouven_Weiler at gmx.net[mailto:Rouven_Weiler at gmx.net
>>>>> <Rouven_Weiler at gmx.net>]>
>>>>>> Kopie: samba at lists.samba.org[mailto:samba at lists.samba.org
>>>>> <samba at lists.samba.org>]
>>>>>> Antwort an: Jeremy Allison <jra at samba.org[mailto:jra at samba.org
>>>>> <jra at samba.org>]>
>>>>>>
>>>>>> On Fri, Apr 12, 2019 at 09:46:04AM +0200, Rouven WEILER via samba
>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hi Jeremy,
>>>>>>>
>>>>>>> I got some info on that topic from the illumos devs:
>>>>>>>
>>>>>>>
>>>>>>>> It's a sporadic issue, you're lucky enough to not encounter it on
>>>>> 4.9.5.
>>>>>>>>
>>>>>>>> I confirmed in 4.10.2, it happens:
>>>>>>>>
>>>>>>>> winbindd.log: tdb(/tmw-nas-3p/samba/var/lock/gencache.tdb):
>>>>> tdb_open_ex: tdb_mutex_init failed for
>>>>> /tmw-nas-3p/samba/var/lock/gencache.tdb: Device busy
>>>>>>>>
>>>>>>>> So either apply OS fix, or apply a software fix (zeroing
>>>>> pthread_mutex_t before pthread_mutex_init()).
>>>>>>>
>>>>>>> This all should be based on the following issue:
>>>>>>>
>>>>>>>
>>>>> https://www.illumos.org/issues/9959[https://www.illumos.org/issues/9959]
>>>>>
>>>>>>>
>>>>>>> As far as I understand the issue making an software fix in samba for
>>>>> the OS seems more reasonable since the problem can occur sporadic and thus
>>>>> again...
>>>>>>> Could you tell me how to help to fix that bug?
>>>>>>
>>>>>> This doesn't look exactly the same. In your
>>>>>> log below you say:
>>>>>>
>>>>>> ret = pthread_mutexattr_init(&ma); in line 570
>>>>>> I got errorno 16 --> device busy.
>>>>>>
>>>>>> But the bug report is about needing the value
>>>>>> passed to pthread_mutex_init() needing to be
>>>>>> zero, not the value passed to pthread_mutexattr_init().
>>>>>>
>>>>>> Can you just try adding '= {0}' to all the
>>>>>> places where we declare:
>>>>>>
>>>>>> pthread_mutexattr_t ma
>>>>>>
>>>>>> (i.e. make it say)
>>>>>>
>>>>>> pthread_mutexattr_t ma = {0};
>>>>>>
>>>>>> and see if it makes your problem go away ?
>>>>>>
>>>>>>
>>>>>>> Gesendet: Dienstag, 02. April 2019 um 18:59 Uhr
>>>>>>> Von: "Jeremy Allison" <jra at samba.org[mailto:jra at samba.org
>>>>> <jra at samba.org>]>
>>>>>>> An: "Rouven WEILER" <Rouven_Weiler at gmx.net[
>>>>> mailto:Rouven_Weiler at gmx.net <Rouven_Weiler at gmx.net>]>
>>>>>>> Cc: samba at lists.samba.org[mailto:samba at lists.samba.org
>>>>> <samba at lists.samba.org>]
>>>>>>> Betreff: Re: Re: [Samba] gencache.tdb: device busy
>>>>>>> On Tue, Apr 02, 2019 at 01:24:19PM +0200, Rouven WEILER wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> I added LOG statements to mutex.c printing the strerror from the
>>>>> return values.
>>>>>>>>
>>>>>>>> From
>>>>>>>>
>>>>>>>> ret = pthread_mutexattr_init(&ma); in line 570
>>>>>>>>
>>>>>>>> I got errorno 16 --> device busy.
>>>>>>>>
>>>>>>>> But due the manual of pthread_mutex_attr_init() there should only be
>>>>> ENOMEM and EINVAL...
>>>>>>>>
>>>>>>>> Doing gdb smbd -> run
>>>>>>>> Doe not reveal anything, but I never used gdb for debugging
>>>>> anything. I just gave it a shot...
>>>>>>>
>>>>>>> Sorry, looks a little like the libc/pthread_mutex support
>>>>>>> on Solaris is broken. You might want to raise this on a
>>>>>>> Solaris/Illumos forum.
>>>>>>>
>>>>>>> --
>>>>>>> To unsubscribe from this list go to the following URL and read the
>>>>>>> instructions:
>>>>> https://lists.samba.org/mailman/options/samba[https://lists.samba.org/mailman/options/samba]
>>>>>
>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list go to the following URL and read the
>>>>>> instructions:
>>>>> https://lists.samba.org/mailman/options/samba[https://lists.samba.org/mailman/options/samba]
>>>>>
>>>>>
>>>>> ------------------------------------------
>>>>> illumos: illumos-developer
>>>>> Permalink:
>>>>> https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M99a2afeb19714648c051ab5f
>>>>> Delivery options:
>>>>> https://illumos.topicbox.com/groups/developer/subscription
>>>>>
>>>>> *illumos <https://illumos.topicbox.com/latest>* / illumos-developer /
>>> see discussions <https://illumos.topicbox.com/groups/developer> +
>>> participants <https://illumos.topicbox.com/groups/developer/members> + delivery
>>> options <https://illumos.topicbox.com/groups/developer/subscription>
>>> Permalink
>>> <https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M2372380a78d4d5078f031f0c>
>>>
>>>
>
> ------------------------------------------
> illumos: illumos-developer
> Permalink: https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M1048b12f29cda459a4de0893
> Delivery options: https://illumos.topicbox.com/groups/developer/subscription
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openindiana.org/pipermail/oi-dev/attachments/20190421/c3de5376/attachment-0004.html>
More information about the oi-dev
mailing list