<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Dear all,<div class=""><br class=""></div><div class="">trying to make samba work and getting rid of my error message </div><div class=""><pre style="white-space: pre-wrap; background-color: rgb(255, 255, 255);" class="">[2019/04/01 09:51:41.112485, 0] ../../lib/tdb_wrap/tdb_wrap.c:64(tdb_wrap_log)
tdb(/var/samba/lock/gencache.tdb): tdb_open_ex: tdb_mutex_init failed for /var/samba/lock/gencache.tdb: Device busy</pre></div><div class=""><br class=""></div><div class="">I followed the info of Youzhong. It definitely works. So thankx.</div><div class="">BUT: The OS (OpenIndiana; state of April,19th) is patched against the illumos issue 9959</div><div class=""><br class=""></div><div class="">So there might be some mechanism (maybe MUTEX_MAGIC) that avoids the initialization of the mutex before init.</div><div class="">I am really not deep in all that pthread stuff…</div><div class=""><br class=""></div><div class="">Is there the possiblity that there is a bug in the code base of issue 9959?</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Best regards</div><div class=""><br class=""></div><div class="">Rouven</div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">Am 18.04.2019 um 19:20 schrieb Robert Mustacchi <<a href="mailto:rm@joyent.com" class="">rm@joyent.com</a>>:</div><br class="Apple-interchange-newline"><div class=""><div class="">On 4/18/19 10:12 , Youzhong Yang wrote:<br class=""><blockquote type="cite" class="">Little bit confused. The page <a href="https://www.openindiana.org/download/" class="">https://www.openindiana.org/download/</a> shows<br class="">latest ISO is dated 2018.10, but 9959 fix was submitted on 11/26/2018, I<br class="">guess Hipster doesn't have it.<br class=""></blockquote><br class="">The ISO is just the install image. Generally, after using that to<br class="">bootstrap, then folks will update to latest bits via IPS. As a result,<br class="">the ISO isn't updated (and generally doesn't have to be) quite as often.<br class=""><br class="">Robert<br class=""><br class=""><blockquote type="cite" class="">On Thu, Apr 18, 2019 at 12:38 PM Rouven WEILER <<a href="mailto:Rouven_Weiler@gmx.net" class="">Rouven_Weiler@gmx.net</a>><br class="">wrote:<br class=""><br class=""><blockquote type="cite" class="">On my system the issue is solved and the sources are patched accordingly.<br class="">Thus, the issue seems to be somewhere else.<br class=""><br class="">Am 18.04.2019 um 18:27 schrieb Rouven WEILER <<a href="mailto:Rouven_Weiler@gmx.net" class="">Rouven_Weiler@gmx.net</a>>:<br class=""><br class="">concerning openindiana I am using the most recent one. I right now asked<br class="">if the issue is solved in the latest version.<br class=""><br class="">another interesting thing is that older samba versions do not throw EBUSY,<br class="">although on the same underlying system.<br class=""><br class=""><br class=""><br class="">Am 18.04.2019 um 18:07 schrieb Youzhong Yang <<a href="mailto:youzhong@gmail.com" class="">youzhong@gmail.com</a>>:<br class=""><br class="">That is interesting. I am curious about how pthread_mutexattr_init ends up<br class="">returning EBUSY. What verison of Openindiana are you running? is there a<br class="">link pointing to the fix that it has for 9959?<br class=""><br class="">On Thu, Apr 18, 2019 at 11:53 AM Rouven WEILER <<a href="mailto:Rouven_Weiler@gmx.net" class="">Rouven_Weiler@gmx.net</a>><br class="">wrote:<br class=""><br class=""><blockquote type="cite" class="">I understand and see that it works, but as written below<br class="">pthread_mutexattr_init throws EBUSY, not pthread_mutex_init.<br class=""><br class="">Also Openindiana seems to include the patch for issue 9959 mentioned. I<br class="">will clarify that to make sure.<br class=""><br class="">I just want to make sure that the patch you suggest aims for the issue<br class="">and is not a workaround.<br class=""><br class=""><br class="">Am 18.04.2019 um 16:09 schrieb Youzhong Yang <<a href="mailto:youzhong@gmail.com" class="">youzhong@gmail.com</a>>:<br class=""><br class="">I reproduced the issue on a smartos version which does not have the fix<br class="">for <a href="https://www.illumos.org/issues/9959" class="">https://www.illumos.org/issues/9959</a>.<br class=""><br class="">The pthread_mutex_init() at line 590 returns EBUSY:<br class="">ret = pthread_mutex_init(chain, &ma);<br class=""><br class=""><a href="https://git.samba.org/?p=samba.git;a=blob;f=lib/tdb/common/mutex.c;h=8a122d5d6a2ec321bce25cf06062acc808f1e126;hb=HEAD#l590" class="">https://git.samba.org/?p=samba.git;a=blob;f=lib/tdb/common/mutex.c;h=8a122d5d6a2ec321bce25cf06062acc808f1e126;hb=HEAD#l590</a><br class=""><br class=""><br class="">Then I applied the following patch to lib/tdb/common/mutex.c:<br class=""><br class=""># git grep -B 1 -n pthread_mutex_init lib/tdb/common/mutex.c<br class="">lib/tdb/common/mutex.c-590- bzero(chain,<br class="">sizeof(pthread_mutex_t));<br class="">lib/tdb/common/mutex.c:591: ret = pthread_mutex_init(chain,<br class="">&ma);<br class="">--<br class="">lib/tdb/common/mutex.c-599- bzero(&m->allrecord_mutex,<br class="">sizeof(pthread_mutex_t));<br class="">lib/tdb/common/mutex.c:600: ret =<br class="">pthread_mutex_init(&m->allrecord_mutex, &ma);<br class="">--<br class="">lib/tdb/common/mutex.c-693- bzero(&m, sizeof(pthread_mutex_t));<br class="">lib/tdb/common/mutex.c:694: ret = pthread_mutex_init(&m, &ma);<br class="">--<br class="">lib/tdb/common/mutex.c-891- bzero(ptr, sizeof(pthread_mutex_t));<br class="">lib/tdb/common/mutex.c:892: ret = pthread_mutex_init(ptr, &ma);<br class=""><br class="">It works like a charm. Fair enough?<br class=""><br class="">So again I recommend either of the following two:<br class="">- apply 9959 fix to your OS image, or use a newer version which has 9959<br class="">fix<br class="">- bzero pthread_mutex_t struct before calling pthread_mutex_init().<br class=""><br class="">In my environment I applied bzero patch to all pthread_mutex_init() calls<br class="">in the entire samba code base, you know, who knows what is going to happen.<br class=""><br class=""><br class=""><br class="">On Thu, Apr 18, 2019 at 4:29 AM Rouven WEILER <Rouven_Weiler@gmx.net><br class="">wrote:<br class=""><br class=""><blockquote type="cite" class="">That sounds reasonable!<br class="">Could you help me to dtrace that in a feasible way? Up to now I am not<br class="">used to dtrace. Except knowing what is and that its powerful.<br class=""><br class="">Am 18.04.2019 um 09:29 schrieb Jason King <jason.brian.king@gmail.com>:<br class=""><br class="">It appears that the robust mutexes are mmapped from the .tdb file.<br class="">EBUSY means that the mutex has already been intialized. It also appears<br class="">that if the TDB_CLEAR_IF_FIRST flag is _not_ set when opening the tdb, it<br class="">will try to initialize the mutexes. I’m wondering if the tdb file gets<br class="">created, the memory image of the initialized mutexes is getting saved, then<br class="">when it’s opened again, if that flag is not set, it tries to re-initialize<br class="">the mutexes and fails (because they are already constructed) legitimately<br class="">returning EBUSY. If the samba code is in fact trying to reinitialize an<br class="">already initialized mutex (dtrace might be able to help determine this — at<br class="">this point it’s merely a supposition), that would explain the error.<br class=""><br class=""><br class=""><br class=""><br class="">From: Rouven WEILER <rouven_weiler@gmx.net> <rouven_weiler@gmx.net><br class="">Reply: illumos-developer <developer@lists.illumos.org><br class=""><developer@lists.illumos.org><br class="">Date: April 18, 2019 at 12:48:48 AM<br class="">To: Youzhong Yang <youzhong@gmail.com> <youzhong@gmail.com><br class="">Cc: Robert Mustacchi <rm@joyent.com> <rm@joyent.com>, Toomas Soome via<br class="">illumos-developer <developer@lists.illumos.org><br class=""><developer@lists.illumos.org><br class="">Subject: [developer] Aw: Re: [Samba] gencache.tdb: device busy<br class=""><br class="">You are totally right. That's the same I got from the man pages.<br class=""><br class="">The responsible code lines of samba also seems to bear no issue and also<br class="">this issue arises on samba version 4.10.x not on earlier ones.<br class="">In addition, there seems to be only a problem with<br class="">illumos/openindiana...<br class="">That's why I do not know how to get that fixed. Up to now I hadn't time<br class="">to dtrace.<br class=""><br class="">Only thing I got so far: When the mutexattr is already set the return<br class="">value will result on undefined behaviour.<br class="">But why only after version 4.10.x on openindiana/illumos?<br class=""><br class="">Just for completion here the codelines of samba:<br class="">https://git.samba.org/?p=samba.git;a=blob;f=lib/tdb/common/mutex.c;h=8a122d5d6a2ec321bce25cf06062acc808f1e126;hb=HEAD#l570<br class=""><br class=""><br class=""><br class="">Gesendet: Mittwoch, 17. April 2019 um 19:19 Uhr<br class="">Von: "Youzhong Yang" <youzhong@gmail.com><br class="">An: "Rouven WEILER" <Rouven_Weiler@gmx.net><br class="">Cc: "Robert Mustacchi" <rm@joyent.com>, "Toomas Soome via<br class="">illumos-developer" <developer@lists.illumos.org><br class="">Betreff: Re: [Samba] gencache.tdb: device busy<br class=""><br class="">Unless I am totally wrong, pthread_mutexattr_init() shouldn't return<br class="">EBUSY:<br class=""><br class="">http://src.illumos.org/source/xref/illumos-gate/usr/src/lib/libc/port/threads/pthr_mutex.c#37<br class=""><br class=""><br class="">On Tue, Apr 16, 2019 at 2:45 PM Rouven WEILER <Rouven_Weiler@gmx.net[<br class="">mailto:Rouven_Weiler@gmx.net <Rouven_Weiler@gmx.net>]> wrote:Hi Robert<br class="">& Youzhong,<br class=""><br class="">I got a response from the Samba-Devs.<br class="">Would the issue mentioned by Youzhong also hot pthread_mutexattr_t?<br class=""><br class="">Concerning dtrace I hadn’t the time yet but I will check for that.<br class="">Is there maybe a way to trace all forks of smbd? Usually the issue<br class="">arises after the fork…<br class=""><br class="">FIND BELOW THE STATEMENT FROM samba.org[http://samba.org]:<br class=""><br class="">This doesn't look exactly the same. In your<br class="">log below you say:<br class=""><br class="">ret = pthread_mutexattr_init(&ma); in line 570<br class="">I got errorno 16 --> device busy.<br class=""><br class="">But the bug report is about needing the value<br class="">passed to pthread_mutex_init() needing to be<br class="">zero, not the value passed to pthread_mutexattr_init().<br class=""><br class="">Can you just try adding '= {0}' to all the<br class="">places where we declare:<br class=""><br class="">pthread_mutexattr_t ma<br class=""><br class="">(i.e. make it say)<br class=""><br class="">pthread_mutexattr_t ma = {0};<br class=""><br class="">and see if it makes your problem go away ?<br class=""><br class="">END OF STATEMENT<br class=""><br class=""><br class=""><blockquote type="cite" class="">Anfang der weitergeleiteten Nachricht:<br class=""><br class="">Von: Jeremy Allison via samba <samba@lists.samba.org[<br class=""></blockquote>mailto:samba@lists.samba.org <samba@lists.samba.org>]><br class=""><blockquote type="cite" class="">Betreff: Aw: [Samba] gencache.tdb: device busy<br class="">Datum: 16. April 2019 um 02:07:05 MESZ<br class="">An: Rouven WEILER <Rouven_Weiler@gmx.net[mailto:Rouven_Weiler@gmx.net<br class=""></blockquote><Rouven_Weiler@gmx.net>]><br class=""><blockquote type="cite" class="">Kopie: samba@lists.samba.org[mailto:samba@lists.samba.org<br class=""></blockquote><samba@lists.samba.org>]<br class=""><blockquote type="cite" class="">Antwort an: Jeremy Allison <jra@samba.org[mailto:jra@samba.org<br class=""></blockquote><jra@samba.org>]><br class=""><blockquote type="cite" class=""><br class="">On Fri, Apr 12, 2019 at 09:46:04AM +0200, Rouven WEILER via samba<br class=""></blockquote>wrote:<br class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><br class=""><br class="">Hi Jeremy,<br class=""><br class="">I got some info on that topic from the illumos devs:<br class=""><br class=""><br class=""><blockquote type="cite" class="">It's a sporadic issue, you're lucky enough to not encounter it on<br class=""></blockquote></blockquote></blockquote>4.9.5.<br class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><br class="">I confirmed in 4.10.2, it happens:<br class=""><br class="">winbindd.log: tdb(/tmw-nas-3p/samba/var/lock/gencache.tdb):<br class=""></blockquote></blockquote></blockquote>tdb_open_ex: tdb_mutex_init failed for<br class="">/tmw-nas-3p/samba/var/lock/gencache.tdb: Device busy<br class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><br class="">So either apply OS fix, or apply a software fix (zeroing<br class=""></blockquote></blockquote></blockquote>pthread_mutex_t before pthread_mutex_init()).<br class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><br class="">This all should be based on the following issue:<br class=""><br class=""><br class=""></blockquote></blockquote>https://www.illumos.org/issues/9959[https://www.illumos.org/issues/9959]<br class=""><br class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><br class="">As far as I understand the issue making an software fix in samba for<br class=""></blockquote></blockquote>the OS seems more reasonable since the problem can occur sporadic and thus<br class="">again...<br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Could you tell me how to help to fix that bug?<br class=""></blockquote><br class="">This doesn't look exactly the same. In your<br class="">log below you say:<br class=""><br class="">ret = pthread_mutexattr_init(&ma); in line 570<br class="">I got errorno 16 --> device busy.<br class=""><br class="">But the bug report is about needing the value<br class="">passed to pthread_mutex_init() needing to be<br class="">zero, not the value passed to pthread_mutexattr_init().<br class=""><br class="">Can you just try adding '= {0}' to all the<br class="">places where we declare:<br class=""><br class="">pthread_mutexattr_t ma<br class=""><br class="">(i.e. make it say)<br class=""><br class="">pthread_mutexattr_t ma = {0};<br class=""><br class="">and see if it makes your problem go away ?<br class=""><br class=""><br class=""><blockquote type="cite" class="">Gesendet: Dienstag, 02. April 2019 um 18:59 Uhr<br class="">Von: "Jeremy Allison" <jra@samba.org[mailto:jra@samba.org<br class=""></blockquote></blockquote><jra@samba.org>]><br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">An: "Rouven WEILER" <Rouven_Weiler@gmx.net[<br class=""></blockquote></blockquote>mailto:Rouven_Weiler@gmx.net <Rouven_Weiler@gmx.net>]><br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Cc: samba@lists.samba.org[mailto:samba@lists.samba.org<br class=""></blockquote></blockquote><samba@lists.samba.org>]<br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Betreff: Re: Re: [Samba] gencache.tdb: device busy<br class="">On Tue, Apr 02, 2019 at 01:24:19PM +0200, Rouven WEILER wrote:<br class=""><blockquote type="cite" class=""><br class=""><br class="">I added LOG statements to mutex.c printing the strerror from the<br class=""></blockquote></blockquote></blockquote>return values.<br class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><br class="">From<br class=""><br class="">ret = pthread_mutexattr_init(&ma); in line 570<br class=""><br class="">I got errorno 16 --> device busy.<br class=""><br class="">But due the manual of pthread_mutex_attr_init() there should only be<br class=""></blockquote></blockquote></blockquote>ENOMEM and EINVAL...<br class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><br class="">Doing gdb smbd -> run<br class="">Doe not reveal anything, but I never used gdb for debugging<br class=""></blockquote></blockquote></blockquote>anything. I just gave it a shot...<br class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><br class="">Sorry, looks a little like the libc/pthread_mutex support<br class="">on Solaris is broken. You might want to raise this on a<br class="">Solaris/Illumos forum.<br class=""><br class="">--<br class="">To unsubscribe from this list go to the following URL and read the<br class="">instructions:<br class=""></blockquote></blockquote>https://lists.samba.org/mailman/options/samba[https://lists.samba.org/mailman/options/samba]<br class=""><br class=""><blockquote type="cite" class=""><br class="">--<br class="">To unsubscribe from this list go to the following URL and read the<br class="">instructions:<br class=""></blockquote>https://lists.samba.org/mailman/options/samba[https://lists.samba.org/mailman/options/samba]<br class=""><br class=""><br class="">------------------------------------------<br class="">illumos: illumos-developer<br class="">Permalink:<br class="">https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M99a2afeb19714648c051ab5f<br class="">Delivery options:<br class="">https://illumos.topicbox.com/groups/developer/subscription<br class=""><br class="">*illumos <https://illumos.topicbox.com/latest>* / illumos-developer /<br class=""></blockquote></blockquote>see discussions <<a href="https://illumos.topicbox.com/groups/developer" class="">https://illumos.topicbox.com/groups/developer</a>> +<br class="">participants <<a href="https://illumos.topicbox.com/groups/developer/members" class="">https://illumos.topicbox.com/groups/developer/members</a>> + delivery<br class="">options <<a href="https://illumos.topicbox.com/groups/developer/subscription" class="">https://illumos.topicbox.com/groups/developer/subscription</a>><br class="">Permalink<br class=""><<a href="https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M2372380a78d4d5078f031f0c" class="">https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M2372380a78d4d5078f031f0c</a>><br class=""><br class=""><br class=""></blockquote></blockquote><br class="">------------------------------------------<br class="">illumos: illumos-developer<br class="">Permalink: <a href="https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M1048b12f29cda459a4de0893" class="">https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M1048b12f29cda459a4de0893</a><br class="">Delivery options: <a href="https://illumos.topicbox.com/groups/developer/subscription" class="">https://illumos.topicbox.com/groups/developer/subscription</a><br class=""></div></div></blockquote></div><br class=""></div></body></html>