[oi-dev] [developer] [Samba] gencache.tdb: device busy

Rouven WEILER Rouven_Weiler at gmx.net
Sun Apr 21 12:51:15 UTC 2019 

Well, the only thing I got was that there is the return code EBUSY on all mutexes, when I run unpatched and the issue only exists on illumos not linux. Using debuggers etc i did not succeed due to my lacking knowledge.

> Am 21.04.2019 um 14:28 schrieb Richard Lowe <richlowe at richlowe.net>:
> 
> You could check whether the mutex already looks initialized to us using a debugger or logging statements added to the code.
> 
> If that's the case, we can think again about relaxing the check I added in 9959.
> 
> 
> 
>> On Sun, Apr 21, 2019, 08:20 Rouven WEILER <Rouven_Weiler at gmx.net> wrote:
>> Any chance to get to know what happens?
>> 
>>> Am 21.04.2019 um 14:12 schrieb Richard Lowe <richlowe at richlowe.net>:
>>> 
>>> It's possible there's a bug.  It's also possible you're getting really unlucky and the uninitialized mutex looks valid because the magic is valid.
>>> 
>>>> On Sun, Apr 21, 2019, 06:10 Rouven WEILER <Rouven_Weiler at gmx.net> wrote:
>>>> Dear all,
>>>> 
>>>> trying to make samba work and getting rid of my error message 
>>>> [2019/04/01 09:51:41.112485,  0] ../../lib/tdb_wrap/tdb_wrap.c:64(tdb_wrap_log)
>>>>  tdb(/var/samba/lock/gencache.tdb): tdb_open_ex: tdb_mutex_init failed for /var/samba/lock/gencache.tdb: Device busy
>>>> 
>>>> I followed the info of Youzhong. It definitely works. So thankx.
>>>> BUT: The OS (OpenIndiana; state of April,19th) is patched against the illumos issue 9959
>>>> 
>>>> So there might be some mechanism (maybe MUTEX_MAGIC) that avoids the initialization of the mutex before init.
>>>> I am really not deep in all that pthread stuff…
>>>> 
>>>> Is there the possiblity that there is a bug in the code base of issue 9959?
>>>> 
>>>> 
>>>> Best regards
>>>> 
>>>> Rouven
>>>> 
>>>> 
>>>>> Am 18.04.2019 um 19:20 schrieb Robert Mustacchi <rm at joyent.com>:
>>>>> 
>>>>> On 4/18/19 10:12 , Youzhong Yang wrote:
>>>>>> Little bit confused. The page  https://www.openindiana.org/download/ shows
>>>>>> latest ISO is dated 2018.10, but 9959 fix was submitted on 11/26/2018, I
>>>>>> guess Hipster doesn't have it.
>>>>> 
>>>>> The ISO is just the install image. Generally, after using that to
>>>>> bootstrap, then folks will update to latest bits via IPS. As a result,
>>>>> the ISO isn't updated (and generally doesn't have to be) quite as often.
>>>>> 
>>>>> Robert
>>>>> 
>>>>>> On Thu, Apr 18, 2019 at 12:38 PM Rouven WEILER <Rouven_Weiler at gmx.net>
>>>>>> wrote:
>>>>>> 
>>>>>>> On my system the issue is solved and the sources are patched accordingly.
>>>>>>> Thus, the issue seems to be somewhere else.
>>>>>>> 
>>>>>>> Am 18.04.2019 um 18:27 schrieb Rouven WEILER <Rouven_Weiler at gmx.net>:
>>>>>>> 
>>>>>>> concerning openindiana I am using the most recent one. I right now asked
>>>>>>> if the issue is solved in the latest version.
>>>>>>> 
>>>>>>> another interesting thing is that older samba versions do not throw EBUSY,
>>>>>>> although on the same underlying system.
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> Am 18.04.2019 um 18:07 schrieb Youzhong Yang <youzhong at gmail.com>:
>>>>>>> 
>>>>>>> That is interesting. I am curious about how pthread_mutexattr_init ends up
>>>>>>> returning EBUSY. What verison of Openindiana are you running? is there a
>>>>>>> link pointing to the fix that it has for 9959?
>>>>>>> 
>>>>>>> On Thu, Apr 18, 2019 at 11:53 AM Rouven WEILER <Rouven_Weiler at gmx.net>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> I understand and see that it works, but as written below
>>>>>>>> pthread_mutexattr_init throws EBUSY, not pthread_mutex_init.
>>>>>>>> 
>>>>>>>> Also Openindiana seems to include the patch for issue 9959 mentioned. I
>>>>>>>> will clarify that to make sure.
>>>>>>>> 
>>>>>>>> I just want to make sure that the patch you suggest aims for the issue
>>>>>>>> and is not a workaround.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Am 18.04.2019 um 16:09 schrieb Youzhong Yang <youzhong at gmail.com>:
>>>>>>>> 
>>>>>>>> I reproduced the issue on a smartos version which does not have the fix
>>>>>>>> for  https://www.illumos.org/issues/9959.
>>>>>>>> 
>>>>>>>> The pthread_mutex_init() at line 590 returns EBUSY:
>>>>>>>> ret = pthread_mutex_init(chain, &ma);
>>>>>>>> 
>>>>>>>> https://git.samba.org/?p=samba.git;a=blob;f=lib/tdb/common/mutex.c;h=8a122d5d6a2ec321bce25cf06062acc808f1e126;hb=HEAD#l590
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Then I applied the following patch to lib/tdb/common/mutex.c:
>>>>>>>> 
>>>>>>>> # git grep -B 1 -n pthread_mutex_init lib/tdb/common/mutex.c
>>>>>>>> lib/tdb/common/mutex.c-590-             bzero(chain,
>>>>>>>> sizeof(pthread_mutex_t));
>>>>>>>> lib/tdb/common/mutex.c:591:             ret = pthread_mutex_init(chain,
>>>>>>>> &ma);
>>>>>>>> --
>>>>>>>> lib/tdb/common/mutex.c-599-     bzero(&m->allrecord_mutex,
>>>>>>>> sizeof(pthread_mutex_t));
>>>>>>>> lib/tdb/common/mutex.c:600:     ret =
>>>>>>>> pthread_mutex_init(&m->allrecord_mutex, &ma);
>>>>>>>> --
>>>>>>>> lib/tdb/common/mutex.c-693-     bzero(&m, sizeof(pthread_mutex_t));
>>>>>>>> lib/tdb/common/mutex.c:694:     ret = pthread_mutex_init(&m, &ma);
>>>>>>>> --
>>>>>>>> lib/tdb/common/mutex.c-891-     bzero(ptr, sizeof(pthread_mutex_t));
>>>>>>>> lib/tdb/common/mutex.c:892:     ret = pthread_mutex_init(ptr, &ma);
>>>>>>>> 
>>>>>>>> It works like a charm. Fair enough?
>>>>>>>> 
>>>>>>>> So again I recommend either of the following two:
>>>>>>>> - apply 9959 fix to your OS image, or use a newer version which has 9959
>>>>>>>> fix
>>>>>>>> - bzero pthread_mutex_t struct before calling pthread_mutex_init().
>>>>>>>> 
>>>>>>>> In my environment I applied bzero patch to all pthread_mutex_init() calls
>>>>>>>> in the entire samba code base, you know, who knows what is going to happen.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Thu, Apr 18, 2019 at 4:29 AM Rouven WEILER <Rouven_Weiler at gmx.net>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> That sounds reasonable!
>>>>>>>>> Could you help me to dtrace that in a feasible way? Up to now I am not
>>>>>>>>> used to dtrace. Except knowing what is and that its powerful.
>>>>>>>>> 
>>>>>>>>> Am 18.04.2019 um 09:29 schrieb Jason King <jason.brian.king at gmail.com>:
>>>>>>>>> 
>>>>>>>>> It appears that the robust mutexes are mmapped from the .tdb file.
>>>>>>>>> EBUSY means that the mutex has already been intialized.  It also appears
>>>>>>>>> that if the TDB_CLEAR_IF_FIRST flag is _not_ set when opening the tdb, it
>>>>>>>>> will try to initialize the mutexes.  I’m wondering if the tdb file gets
>>>>>>>>> created, the memory image of the initialized mutexes is getting saved, then
>>>>>>>>> when it’s opened again, if that flag is not set, it tries to re-initialize
>>>>>>>>> the mutexes and fails (because they are already constructed) legitimately
>>>>>>>>> returning EBUSY.  If the samba code is in fact trying to reinitialize an
>>>>>>>>> already initialized mutex (dtrace might be able to help determine this — at
>>>>>>>>> this point it’s merely a supposition), that would explain the error.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> From: Rouven WEILER <rouven_weiler at gmx.net> <rouven_weiler at gmx.net>
>>>>>>>>> Reply: illumos-developer <developer at lists.illumos.org>
>>>>>>>>> <developer at lists.illumos.org>
>>>>>>>>> Date: April 18, 2019 at 12:48:48 AM
>>>>>>>>> To: Youzhong Yang <youzhong at gmail.com> <youzhong at gmail.com>
>>>>>>>>> Cc: Robert Mustacchi <rm at joyent.com> <rm at joyent.com>, Toomas Soome via
>>>>>>>>> illumos-developer <developer at lists.illumos.org>
>>>>>>>>> <developer at lists.illumos.org>
>>>>>>>>> Subject:  [developer] Aw: Re: [Samba] gencache.tdb: device busy
>>>>>>>>> 
>>>>>>>>> You are totally right. That's the same I got from the man pages.
>>>>>>>>> 
>>>>>>>>> The responsible code lines of samba also seems to bear no issue and also
>>>>>>>>> this issue arises on samba version 4.10.x not on earlier ones.
>>>>>>>>> In addition, there seems to be only a problem with
>>>>>>>>> illumos/openindiana...
>>>>>>>>> That's why I do not know how to get that fixed. Up to now I hadn't time
>>>>>>>>> to dtrace.
>>>>>>>>> 
>>>>>>>>> Only thing I got so far: When the mutexattr is already set the return
>>>>>>>>> value will result on undefined behaviour.
>>>>>>>>> But why only after version 4.10.x on openindiana/illumos?
>>>>>>>>> 
>>>>>>>>> Just for completion here the codelines of samba:
>>>>>>>>> https://git.samba.org/?p=samba.git;a=blob;f=lib/tdb/common/mutex.c;h=8a122d5d6a2ec321bce25cf06062acc808f1e126;hb=HEAD#l570
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Gesendet: Mittwoch, 17. April 2019 um 19:19 Uhr
>>>>>>>>> Von: "Youzhong Yang" <youzhong at gmail.com>
>>>>>>>>> An: "Rouven WEILER" <Rouven_Weiler at gmx.net>
>>>>>>>>> Cc: "Robert Mustacchi" <rm at joyent.com>, "Toomas Soome via
>>>>>>>>> illumos-developer" <developer at lists.illumos.org>
>>>>>>>>> Betreff: Re: [Samba] gencache.tdb: device busy
>>>>>>>>> 
>>>>>>>>> Unless I am totally wrong, pthread_mutexattr_init() shouldn't return
>>>>>>>>> EBUSY:
>>>>>>>>> 
>>>>>>>>> http://src.illumos.org/source/xref/illumos-gate/usr/src/lib/libc/port/threads/pthr_mutex.c#37
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On Tue, Apr 16, 2019 at 2:45 PM Rouven WEILER <Rouven_Weiler at gmx.net[
>>>>>>>>> mailto:Rouven_Weiler at gmx.net <Rouven_Weiler at gmx.net>]> wrote:Hi Robert
>>>>>>>>> & Youzhong,
>>>>>>>>> 
>>>>>>>>> I got a response from the Samba-Devs.
>>>>>>>>> Would the issue mentioned by Youzhong also hot pthread_mutexattr_t?
>>>>>>>>> 
>>>>>>>>> Concerning dtrace I hadn’t the time yet but I will check for that.
>>>>>>>>> Is there maybe a way to trace all forks of smbd? Usually the issue
>>>>>>>>> arises after the fork…
>>>>>>>>> 
>>>>>>>>> FIND BELOW THE STATEMENT FROM samba.org[http://samba.org]:
>>>>>>>>> 
>>>>>>>>> This doesn't look exactly the same. In your
>>>>>>>>> log below you say:
>>>>>>>>> 
>>>>>>>>> ret = pthread_mutexattr_init(&ma); in line 570
>>>>>>>>> I got errorno 16 --> device busy.
>>>>>>>>> 
>>>>>>>>> But the bug report is about needing the value
>>>>>>>>> passed to pthread_mutex_init() needing to be
>>>>>>>>> zero, not the value passed to pthread_mutexattr_init().
>>>>>>>>> 
>>>>>>>>> Can you just try adding '= {0}' to all the
>>>>>>>>> places where we declare:
>>>>>>>>> 
>>>>>>>>> pthread_mutexattr_t ma
>>>>>>>>> 
>>>>>>>>> (i.e. make it say)
>>>>>>>>> 
>>>>>>>>> pthread_mutexattr_t ma = {0};
>>>>>>>>> 
>>>>>>>>> and see if it makes your problem go away ?
>>>>>>>>> 
>>>>>>>>> END OF STATEMENT
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> Anfang der weitergeleiteten Nachricht:
>>>>>>>>>> 
>>>>>>>>>> Von: Jeremy Allison via samba <samba at lists.samba.org[
>>>>>>>>> mailto:samba at lists.samba.org <samba at lists.samba.org>]>
>>>>>>>>>> Betreff: Aw: [Samba] gencache.tdb: device busy
>>>>>>>>>> Datum: 16. April 2019 um 02:07:05 MESZ
>>>>>>>>>> An: Rouven WEILER <Rouven_Weiler at gmx.net[mailto:Rouven_Weiler at gmx.net
>>>>>>>>> <Rouven_Weiler at gmx.net>]>
>>>>>>>>>> Kopie: samba at lists.samba.org[mailto:samba at lists.samba.org
>>>>>>>>> <samba at lists.samba.org>]
>>>>>>>>>> Antwort an: Jeremy Allison <jra at samba.org[mailto:jra at samba.org
>>>>>>>>> <jra at samba.org>]>
>>>>>>>>>> 
>>>>>>>>>> On Fri, Apr 12, 2019 at 09:46:04AM +0200, Rouven WEILER via samba
>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Hi Jeremy,
>>>>>>>>>>> 
>>>>>>>>>>> I got some info on that topic from the illumos devs:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>>> It's a sporadic issue, you're lucky enough to not encounter it on
>>>>>>>>> 4.9.5.
>>>>>>>>>>>> 
>>>>>>>>>>>> I confirmed in 4.10.2, it happens:
>>>>>>>>>>>> 
>>>>>>>>>>>> winbindd.log:  tdb(/tmw-nas-3p/samba/var/lock/gencache.tdb):
>>>>>>>>> tdb_open_ex: tdb_mutex_init failed for
>>>>>>>>> /tmw-nas-3p/samba/var/lock/gencache.tdb: Device busy
>>>>>>>>>>>> 
>>>>>>>>>>>> So either apply OS fix, or apply a software fix (zeroing
>>>>>>>>> pthread_mutex_t before pthread_mutex_init()).
>>>>>>>>>>> 
>>>>>>>>>>> This all should be based on the following issue:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> https://www.illumos.org/issues/9959[https://www.illumos.org/issues/9959]
>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> As far as I understand the issue making an software fix in samba for
>>>>>>>>> the OS seems more reasonable since the problem can occur sporadic and thus
>>>>>>>>> again...
>>>>>>>>>>> Could you tell me how to help to fix that bug?
>>>>>>>>>> 
>>>>>>>>>> This doesn't look exactly the same. In your
>>>>>>>>>> log below you say:
>>>>>>>>>> 
>>>>>>>>>> ret = pthread_mutexattr_init(&ma); in line 570
>>>>>>>>>> I got errorno 16 --> device busy.
>>>>>>>>>> 
>>>>>>>>>> But the bug report is about needing the value
>>>>>>>>>> passed to pthread_mutex_init() needing to be
>>>>>>>>>> zero, not the value passed to pthread_mutexattr_init().
>>>>>>>>>> 
>>>>>>>>>> Can you just try adding '= {0}' to all the
>>>>>>>>>> places where we declare:
>>>>>>>>>> 
>>>>>>>>>> pthread_mutexattr_t ma
>>>>>>>>>> 
>>>>>>>>>> (i.e. make it say)
>>>>>>>>>> 
>>>>>>>>>> pthread_mutexattr_t ma = {0};
>>>>>>>>>> 
>>>>>>>>>> and see if it makes your problem go away ?
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>> Gesendet: Dienstag, 02. April 2019 um 18:59 Uhr
>>>>>>>>>>> Von: "Jeremy Allison" <jra at samba.org[mailto:jra at samba.org
>>>>>>>>> <jra at samba.org>]>
>>>>>>>>>>> An: "Rouven WEILER" <Rouven_Weiler at gmx.net[
>>>>>>>>> mailto:Rouven_Weiler at gmx.net <Rouven_Weiler at gmx.net>]>
>>>>>>>>>>> Cc: samba at lists.samba.org[mailto:samba at lists.samba.org
>>>>>>>>> <samba at lists.samba.org>]
>>>>>>>>>>> Betreff: Re: Re: [Samba] gencache.tdb: device busy
>>>>>>>>>>>> On Tue, Apr 02, 2019 at 01:24:19PM +0200, Rouven WEILER wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> I added LOG statements to mutex.c printing the strerror from the
>>>>>>>>> return values.
>>>>>>>>>>>> 
>>>>>>>>>>>> From
>>>>>>>>>>>> 
>>>>>>>>>>>> ret = pthread_mutexattr_init(&ma); in line 570
>>>>>>>>>>>> 
>>>>>>>>>>>> I got errorno 16 --> device busy.
>>>>>>>>>>>> 
>>>>>>>>>>>> But due the manual of pthread_mutex_attr_init() there should only be
>>>>>>>>> ENOMEM and EINVAL...
>>>>>>>>>>>> 
>>>>>>>>>>>> Doing gdb smbd -> run
>>>>>>>>>>>> Doe not reveal anything, but I never used gdb for debugging
>>>>>>>>> anything. I just gave it a shot...
>>>>>>>>>>> 
>>>>>>>>>>> Sorry, looks a little like the libc/pthread_mutex support
>>>>>>>>>>> on Solaris is broken. You might want to raise this on a
>>>>>>>>>>> Solaris/Illumos forum.
>>>>>>>>>>> 
>>>>>>>>>>> --
>>>>>>>>>>> To unsubscribe from this list go to the following URL and read the
>>>>>>>>>>> instructions:
>>>>>>>>> https://lists.samba.org/mailman/options/samba[https://lists.samba.org/mailman/options/samba]
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> --
>>>>>>>>>> To unsubscribe from this list go to the following URL and read the
>>>>>>>>>> instructions:
>>>>>>>>> https://lists.samba.org/mailman/options/samba[https://lists.samba.org/mailman/options/samba]
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> ------------------------------------------
>>>>>>>>> illumos: illumos-developer
>>>>>>>>> Permalink:
>>>>>>>>> https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M99a2afeb19714648c051ab5f
>>>>>>>>> Delivery options:
>>>>>>>>> https://illumos.topicbox.com/groups/developer/subscription
>>>>>>>>> 
>>>>>>>>> *illumos <https://illumos.topicbox.com/latest>* / illumos-developer /
>>>>>>> see discussions <https://illumos.topicbox.com/groups/developer> +
>>>>>>> participants <https://illumos.topicbox.com/groups/developer/members> + delivery
>>>>>>> options <https://illumos.topicbox.com/groups/developer/subscription>
>>>>>>> Permalink
>>>>>>> <https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M2372380a78d4d5078f031f0c>
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> ------------------------------------------
>>>>> illumos: illumos-developer
>>>>> Permalink: https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M1048b12f29cda459a4de0893
>>>>> Delivery options: https://illumos.topicbox.com/groups/developer/subscription
>>>> 
>>>> _______________________________________________
>>>> oi-dev mailing list
>>>> oi-dev at openindiana.org
>>>> https://openindiana.org/mailman/listinfo/oi-dev
>>> _______________________________________________
>>> oi-dev mailing list
>>> oi-dev at openindiana.org
>>> https://openindiana.org/mailman/listinfo/oi-dev
>> _______________________________________________
>> oi-dev mailing list
>> oi-dev at openindiana.org
>> https://openindiana.org/mailman/listinfo/oi-dev
> _______________________________________________
> oi-dev mailing list
> oi-dev at openindiana.org
> https://openindiana.org/mailman/listinfo/oi-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openindiana.org/pipermail/oi-dev/attachments/20190421/e809631b/attachment-0005.html>

More information about the oi-dev mailing list