<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div dir="ltr"></div><div dir="ltr">Well, the only thing I got was that there is the return code EBUSY on all mutexes, when I run unpatched and the issue only exists on illumos not linux. Using debuggers etc i did not succeed due to my lacking knowledge.</div><div dir="ltr"><br>Am 21.04.2019 um 14:28 schrieb Richard Lowe <<a href="mailto:richlowe@richlowe.net">richlowe@richlowe.net</a>>:<br><br></div><blockquote type="cite"><div dir="ltr"><div dir="auto">You could check whether the mutex already looks initialized to us using a debugger or logging statements added to the code.<div dir="auto"><br></div><div dir="auto">If that's the case, we can think again about relaxing the check I added in 9959.</div><div dir="auto"><br></div><div dir="auto"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Apr 21, 2019, 08:20 Rouven WEILER <<a href="mailto:Rouven_Weiler@gmx.net">Rouven_Weiler@gmx.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div dir="ltr"></div><div dir="ltr">Any chance to get to know what happens?</div><div dir="ltr"><br>Am 21.04.2019 um 14:12 schrieb Richard Lowe <<a href="mailto:richlowe@richlowe.net" target="_blank" rel="noreferrer">richlowe@richlowe.net</a>>:<br><br></div><blockquote type="cite"><div dir="ltr"><div dir="auto">It's possible there's a bug. It's also possible you're getting really unlucky and the uninitialized mutex looks valid because the magic is valid.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Apr 21, 2019, 06:10 Rouven WEILER <<a href="mailto:Rouven_Weiler@gmx.net" target="_blank" rel="noreferrer">Rouven_Weiler@gmx.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space">Dear all,<div><br></div><div>trying to make samba work and getting rid of my error message </div><div><pre style="white-space:pre-wrap;background-color:rgb(255,255,255)">[2019/04/01 09:51:41.112485, 0] ../../lib/tdb_wrap/tdb_wrap.c:64(tdb_wrap_log)
tdb(/var/samba/lock/gencache.tdb): tdb_open_ex: tdb_mutex_init failed for /var/samba/lock/gencache.tdb: Device busy</pre></div><div><br></div><div>I followed the info of Youzhong. It definitely works. So thankx.</div><div>BUT: The OS (OpenIndiana; state of April,19th) is patched against the illumos issue 9959</div><div><br></div><div>So there might be some mechanism (maybe MUTEX_MAGIC) that avoids the initialization of the mutex before init.</div><div>I am really not deep in all that pthread stuff…</div><div><br></div><div>Is there the possiblity that there is a bug in the code base of issue 9959?</div><div><br></div><div><br></div><div>Best regards</div><div><br></div><div>Rouven</div><div><br><div><br><blockquote type="cite"><div>Am 18.04.2019 um 19:20 schrieb Robert Mustacchi <<a href="mailto:rm@joyent.com" rel="noreferrer noreferrer" target="_blank">rm@joyent.com</a>>:</div><br class="m_6658004297779278959m_-8980092141119419513Apple-interchange-newline"><div><div>On 4/18/19 10:12 , Youzhong Yang wrote:<br><blockquote type="cite">Little bit confused. The page <a href="https://www.openindiana.org/download/" rel="noreferrer noreferrer" target="_blank">https://www.openindiana.org/download/</a> shows<br>latest ISO is dated 2018.10, but 9959 fix was submitted on 11/26/2018, I<br>guess Hipster doesn't have it.<br></blockquote><br>The ISO is just the install image. Generally, after using that to<br>bootstrap, then folks will update to latest bits via IPS. As a result,<br>the ISO isn't updated (and generally doesn't have to be) quite as often.<br><br>Robert<br><br><blockquote type="cite">On Thu, Apr 18, 2019 at 12:38 PM Rouven WEILER <<a href="mailto:Rouven_Weiler@gmx.net" rel="noreferrer noreferrer" target="_blank">Rouven_Weiler@gmx.net</a>><br>wrote:<br><br><blockquote type="cite">On my system the issue is solved and the sources are patched accordingly.<br>Thus, the issue seems to be somewhere else.<br><br>Am 18.04.2019 um 18:27 schrieb Rouven WEILER <<a href="mailto:Rouven_Weiler@gmx.net" rel="noreferrer noreferrer" target="_blank">Rouven_Weiler@gmx.net</a>>:<br><br>concerning openindiana I am using the most recent one. I right now asked<br>if the issue is solved in the latest version.<br><br>another interesting thing is that older samba versions do not throw EBUSY,<br>although on the same underlying system.<br><br><br><br>Am 18.04.2019 um 18:07 schrieb Youzhong Yang <<a href="mailto:youzhong@gmail.com" rel="noreferrer noreferrer" target="_blank">youzhong@gmail.com</a>>:<br><br>That is interesting. I am curious about how pthread_mutexattr_init ends up<br>returning EBUSY. What verison of Openindiana are you running? is there a<br>link pointing to the fix that it has for 9959?<br><br>On Thu, Apr 18, 2019 at 11:53 AM Rouven WEILER <<a href="mailto:Rouven_Weiler@gmx.net" rel="noreferrer noreferrer" target="_blank">Rouven_Weiler@gmx.net</a>><br>wrote:<br><br><blockquote type="cite">I understand and see that it works, but as written below<br>pthread_mutexattr_init throws EBUSY, not pthread_mutex_init.<br><br>Also Openindiana seems to include the patch for issue 9959 mentioned. I<br>will clarify that to make sure.<br><br>I just want to make sure that the patch you suggest aims for the issue<br>and is not a workaround.<br><br><br>Am 18.04.2019 um 16:09 schrieb Youzhong Yang <<a href="mailto:youzhong@gmail.com" rel="noreferrer noreferrer" target="_blank">youzhong@gmail.com</a>>:<br><br>I reproduced the issue on a smartos version which does not have the fix<br>for <a href="https://www.illumos.org/issues/9959" rel="noreferrer noreferrer" target="_blank">https://www.illumos.org/issues/9959</a>.<br><br>The pthread_mutex_init() at line 590 returns EBUSY:<br>ret = pthread_mutex_init(chain, &ma);<br><br><a href="https://git.samba.org/?p=samba.git;a=blob;f=lib/tdb/common/mutex.c;h=8a122d5d6a2ec321bce25cf06062acc808f1e126;hb=HEAD#l590" rel="noreferrer noreferrer" target="_blank">https://git.samba.org/?p=samba.git;a=blob;f=lib/tdb/common/mutex.c;h=8a122d5d6a2ec321bce25cf06062acc808f1e126;hb=HEAD#l590</a><br><br><br>Then I applied the following patch to lib/tdb/common/mutex.c:<br><br># git grep -B 1 -n pthread_mutex_init lib/tdb/common/mutex.c<br>lib/tdb/common/mutex.c-590- bzero(chain,<br>sizeof(pthread_mutex_t));<br>lib/tdb/common/mutex.c:591: ret = pthread_mutex_init(chain,<br>&ma);<br>--<br>lib/tdb/common/mutex.c-599- bzero(&m->allrecord_mutex,<br>sizeof(pthread_mutex_t));<br>lib/tdb/common/mutex.c:600: ret =<br>pthread_mutex_init(&m->allrecord_mutex, &ma);<br>--<br>lib/tdb/common/mutex.c-693- bzero(&m, sizeof(pthread_mutex_t));<br>lib/tdb/common/mutex.c:694: ret = pthread_mutex_init(&m, &ma);<br>--<br>lib/tdb/common/mutex.c-891- bzero(ptr, sizeof(pthread_mutex_t));<br>lib/tdb/common/mutex.c:892: ret = pthread_mutex_init(ptr, &ma);<br><br>It works like a charm. Fair enough?<br><br>So again I recommend either of the following two:<br>- apply 9959 fix to your OS image, or use a newer version which has 9959<br>fix<br>- bzero pthread_mutex_t struct before calling pthread_mutex_init().<br><br>In my environment I applied bzero patch to all pthread_mutex_init() calls<br>in the entire samba code base, you know, who knows what is going to happen.<br><br><br><br>On Thu, Apr 18, 2019 at 4:29 AM Rouven WEILER <<a href="mailto:Rouven_Weiler@gmx.net" rel="noreferrer noreferrer" target="_blank">Rouven_Weiler@gmx.net</a>><br>wrote:<br><br><blockquote type="cite">That sounds reasonable!<br>Could you help me to dtrace that in a feasible way? Up to now I am not<br>used to dtrace. Except knowing what is and that its powerful.<br><br>Am 18.04.2019 um 09:29 schrieb Jason King <<a href="mailto:jason.brian.king@gmail.com" rel="noreferrer noreferrer" target="_blank">jason.brian.king@gmail.com</a>>:<br><br>It appears that the robust mutexes are mmapped from the .tdb file.<br>EBUSY means that the mutex has already been intialized. It also appears<br>that if the TDB_CLEAR_IF_FIRST flag is _not_ set when opening the tdb, it<br>will try to initialize the mutexes. I’m wondering if the tdb file gets<br>created, the memory image of the initialized mutexes is getting saved, then<br>when it’s opened again, if that flag is not set, it tries to re-initialize<br>the mutexes and fails (because they are already constructed) legitimately<br>returning EBUSY. If the samba code is in fact trying to reinitialize an<br>already initialized mutex (dtrace might be able to help determine this — at<br>this point it’s merely a supposition), that would explain the error.<br><br><br><br><br>From: Rouven WEILER <<a href="mailto:rouven_weiler@gmx.net" rel="noreferrer noreferrer" target="_blank">rouven_weiler@gmx.net</a>> <<a href="mailto:rouven_weiler@gmx.net" rel="noreferrer noreferrer" target="_blank">rouven_weiler@gmx.net</a>><br>Reply: illumos-developer <<a href="mailto:developer@lists.illumos.org" rel="noreferrer noreferrer" target="_blank">developer@lists.illumos.org</a>><br><<a href="mailto:developer@lists.illumos.org" rel="noreferrer noreferrer" target="_blank">developer@lists.illumos.org</a>><br>Date: April 18, 2019 at 12:48:48 AM<br>To: Youzhong Yang <<a href="mailto:youzhong@gmail.com" rel="noreferrer noreferrer" target="_blank">youzhong@gmail.com</a>> <<a href="mailto:youzhong@gmail.com" rel="noreferrer noreferrer" target="_blank">youzhong@gmail.com</a>><br>Cc: Robert Mustacchi <<a href="mailto:rm@joyent.com" rel="noreferrer noreferrer" target="_blank">rm@joyent.com</a>> <<a href="mailto:rm@joyent.com" rel="noreferrer noreferrer" target="_blank">rm@joyent.com</a>>, Toomas Soome via<br>illumos-developer <<a href="mailto:developer@lists.illumos.org" rel="noreferrer noreferrer" target="_blank">developer@lists.illumos.org</a>><br><<a href="mailto:developer@lists.illumos.org" rel="noreferrer noreferrer" target="_blank">developer@lists.illumos.org</a>><br>Subject: [developer] Aw: Re: [Samba] gencache.tdb: device busy<br><br>You are totally right. That's the same I got from the man pages.<br><br>The responsible code lines of samba also seems to bear no issue and also<br>this issue arises on samba version 4.10.x not on earlier ones.<br>In addition, there seems to be only a problem with<br>illumos/openindiana...<br>That's why I do not know how to get that fixed. Up to now I hadn't time<br>to dtrace.<br><br>Only thing I got so far: When the mutexattr is already set the return<br>value will result on undefined behaviour.<br>But why only after version 4.10.x on openindiana/illumos?<br><br>Just for completion here the codelines of samba:<br><a href="https://git.samba.org/?p=samba.git;a=blob;f=lib/tdb/common/mutex.c;h=8a122d5d6a2ec321bce25cf06062acc808f1e126;hb=HEAD#l570" rel="noreferrer noreferrer" target="_blank">https://git.samba.org/?p=samba.git;a=blob;f=lib/tdb/common/mutex.c;h=8a122d5d6a2ec321bce25cf06062acc808f1e126;hb=HEAD#l570</a><br><br><br><br>Gesendet: Mittwoch, 17. April 2019 um 19:19 Uhr<br>Von: "Youzhong Yang" <<a href="mailto:youzhong@gmail.com" rel="noreferrer noreferrer" target="_blank">youzhong@gmail.com</a>><br>An: "Rouven WEILER" <<a href="mailto:Rouven_Weiler@gmx.net" rel="noreferrer noreferrer" target="_blank">Rouven_Weiler@gmx.net</a>><br>Cc: "Robert Mustacchi" <<a href="mailto:rm@joyent.com" rel="noreferrer noreferrer" target="_blank">rm@joyent.com</a>>, "Toomas Soome via<br>illumos-developer" <<a href="mailto:developer@lists.illumos.org" rel="noreferrer noreferrer" target="_blank">developer@lists.illumos.org</a>><br>Betreff: Re: [Samba] gencache.tdb: device busy<br><br>Unless I am totally wrong, pthread_mutexattr_init() shouldn't return<br>EBUSY:<br><br><a href="http://src.illumos.org/source/xref/illumos-gate/usr/src/lib/libc/port/threads/pthr_mutex.c#37" rel="noreferrer noreferrer" target="_blank">http://src.illumos.org/source/xref/illumos-gate/usr/src/lib/libc/port/threads/pthr_mutex.c#37</a><br><br><br>On Tue, Apr 16, 2019 at 2:45 PM Rouven WEILER <<a href="mailto:Rouven_Weiler@gmx.net" rel="noreferrer noreferrer" target="_blank">Rouven_Weiler@gmx.net</a>[<br>mailto:<a href="mailto:Rouven_Weiler@gmx.net" rel="noreferrer noreferrer" target="_blank">Rouven_Weiler@gmx.net</a> <<a href="mailto:Rouven_Weiler@gmx.net" rel="noreferrer noreferrer" target="_blank">Rouven_Weiler@gmx.net</a>>]> wrote:Hi Robert<br>& Youzhong,<br><br>I got a response from the Samba-Devs.<br>Would the issue mentioned by Youzhong also hot pthread_mutexattr_t?<br><br>Concerning dtrace I hadn’t the time yet but I will check for that.<br>Is there maybe a way to trace all forks of smbd? Usually the issue<br>arises after the fork…<br><br>FIND BELOW THE STATEMENT FROM <a href="http://samba.org" rel="noreferrer noreferrer" target="_blank">samba.org</a>[<a href="http://samba.org" rel="noreferrer noreferrer" target="_blank">http://samba.org</a>]:<br><br>This doesn't look exactly the same. In your<br>log below you say:<br><br>ret = pthread_mutexattr_init(&ma); in line 570<br>I got errorno 16 --> device busy.<br><br>But the bug report is about needing the value<br>passed to pthread_mutex_init() needing to be<br>zero, not the value passed to pthread_mutexattr_init().<br><br>Can you just try adding '= {0}' to all the<br>places where we declare:<br><br>pthread_mutexattr_t ma<br><br>(i.e. make it say)<br><br>pthread_mutexattr_t ma = {0};<br><br>and see if it makes your problem go away ?<br><br>END OF STATEMENT<br><br><br><blockquote type="cite">Anfang der weitergeleiteten Nachricht:<br><br>Von: Jeremy Allison via samba <<a href="mailto:samba@lists.samba.org" rel="noreferrer noreferrer" target="_blank">samba@lists.samba.org</a>[<br></blockquote>mailto:<a href="mailto:samba@lists.samba.org" rel="noreferrer noreferrer" target="_blank">samba@lists.samba.org</a> <<a href="mailto:samba@lists.samba.org" rel="noreferrer noreferrer" target="_blank">samba@lists.samba.org</a>>]><br><blockquote type="cite">Betreff: Aw: [Samba] gencache.tdb: device busy<br>Datum: 16. April 2019 um 02:07:05 MESZ<br>An: Rouven WEILER <<a href="mailto:Rouven_Weiler@gmx.net" rel="noreferrer noreferrer" target="_blank">Rouven_Weiler@gmx.net</a>[mailto:<a href="mailto:Rouven_Weiler@gmx.net" rel="noreferrer noreferrer" target="_blank">Rouven_Weiler@gmx.net</a><br></blockquote><<a href="mailto:Rouven_Weiler@gmx.net" rel="noreferrer noreferrer" target="_blank">Rouven_Weiler@gmx.net</a>>]><br><blockquote type="cite">Kopie: <a href="mailto:samba@lists.samba.org" rel="noreferrer noreferrer" target="_blank">samba@lists.samba.org</a>[mailto:<a href="mailto:samba@lists.samba.org" rel="noreferrer noreferrer" target="_blank">samba@lists.samba.org</a><br></blockquote><<a href="mailto:samba@lists.samba.org" rel="noreferrer noreferrer" target="_blank">samba@lists.samba.org</a>>]<br><blockquote type="cite">Antwort an: Jeremy Allison <<a href="mailto:jra@samba.org" rel="noreferrer noreferrer" target="_blank">jra@samba.org</a>[mailto:<a href="mailto:jra@samba.org" rel="noreferrer noreferrer" target="_blank">jra@samba.org</a><br></blockquote><<a href="mailto:jra@samba.org" rel="noreferrer noreferrer" target="_blank">jra@samba.org</a>>]><br><blockquote type="cite"><br>On Fri, Apr 12, 2019 at 09:46:04AM +0200, Rouven WEILER via samba<br></blockquote>wrote:<br><blockquote type="cite"><blockquote type="cite"><br><br>Hi Jeremy,<br><br>I got some info on that topic from the illumos devs:<br><br><br><blockquote type="cite">It's a sporadic issue, you're lucky enough to not encounter it on<br></blockquote></blockquote></blockquote>4.9.5.<br><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br>I confirmed in 4.10.2, it happens:<br><br>winbindd.log: tdb(/tmw-nas-3p/samba/var/lock/gencache.tdb):<br></blockquote></blockquote></blockquote>tdb_open_ex: tdb_mutex_init failed for<br>/tmw-nas-3p/samba/var/lock/gencache.tdb: Device busy<br><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br>So either apply OS fix, or apply a software fix (zeroing<br></blockquote></blockquote></blockquote>pthread_mutex_t before pthread_mutex_init()).<br><blockquote type="cite"><blockquote type="cite"><br>This all should be based on the following issue:<br><br><br></blockquote></blockquote><a href="https://www.illumos.org/issues/9959%5Bhttps://www.illumos.org/issues/9959%5D" rel="noreferrer noreferrer" target="_blank">https://www.illumos.org/issues/9959[https://www.illumos.org/issues/9959]</a><br><br><blockquote type="cite"><blockquote type="cite"><br>As far as I understand the issue making an software fix in samba for<br></blockquote></blockquote>the OS seems more reasonable since the problem can occur sporadic and thus<br>again...<br><blockquote type="cite"><blockquote type="cite">Could you tell me how to help to fix that bug?<br></blockquote><br>This doesn't look exactly the same. In your<br>log below you say:<br><br>ret = pthread_mutexattr_init(&ma); in line 570<br>I got errorno 16 --> device busy.<br><br>But the bug report is about needing the value<br>passed to pthread_mutex_init() needing to be<br>zero, not the value passed to pthread_mutexattr_init().<br><br>Can you just try adding '= {0}' to all the<br>places where we declare:<br><br>pthread_mutexattr_t ma<br><br>(i.e. make it say)<br><br>pthread_mutexattr_t ma = {0};<br><br>and see if it makes your problem go away ?<br><br><br><blockquote type="cite">Gesendet: Dienstag, 02. April 2019 um 18:59 Uhr<br>Von: "Jeremy Allison" <<a href="mailto:jra@samba.org" rel="noreferrer noreferrer" target="_blank">jra@samba.org</a>[mailto:<a href="mailto:jra@samba.org" rel="noreferrer noreferrer" target="_blank">jra@samba.org</a><br></blockquote></blockquote><<a href="mailto:jra@samba.org" rel="noreferrer noreferrer" target="_blank">jra@samba.org</a>>]><br><blockquote type="cite"><blockquote type="cite">An: "Rouven WEILER" <<a href="mailto:Rouven_Weiler@gmx.net" rel="noreferrer noreferrer" target="_blank">Rouven_Weiler@gmx.net</a>[<br></blockquote></blockquote>mailto:<a href="mailto:Rouven_Weiler@gmx.net" rel="noreferrer noreferrer" target="_blank">Rouven_Weiler@gmx.net</a> <<a href="mailto:Rouven_Weiler@gmx.net" rel="noreferrer noreferrer" target="_blank">Rouven_Weiler@gmx.net</a>>]><br><blockquote type="cite"><blockquote type="cite">Cc: <a href="mailto:samba@lists.samba.org" rel="noreferrer noreferrer" target="_blank">samba@lists.samba.org</a>[mailto:<a href="mailto:samba@lists.samba.org" rel="noreferrer noreferrer" target="_blank">samba@lists.samba.org</a><br></blockquote></blockquote><<a href="mailto:samba@lists.samba.org" rel="noreferrer noreferrer" target="_blank">samba@lists.samba.org</a>>]<br><blockquote type="cite"><blockquote type="cite">Betreff: Re: Re: [Samba] gencache.tdb: device busy<br>On Tue, Apr 02, 2019 at 01:24:19PM +0200, Rouven WEILER wrote:<br><blockquote type="cite"><br><br>I added LOG statements to mutex.c printing the strerror from the<br></blockquote></blockquote></blockquote>return values.<br><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br>From<br><br>ret = pthread_mutexattr_init(&ma); in line 570<br><br>I got errorno 16 --> device busy.<br><br>But due the manual of pthread_mutex_attr_init() there should only be<br></blockquote></blockquote></blockquote>ENOMEM and EINVAL...<br><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br>Doing gdb smbd -> run<br>Doe not reveal anything, but I never used gdb for debugging<br></blockquote></blockquote></blockquote>anything. I just gave it a shot...<br><blockquote type="cite"><blockquote type="cite"><br>Sorry, looks a little like the libc/pthread_mutex support<br>on Solaris is broken. You might want to raise this on a<br>Solaris/Illumos forum.<br><br>--<br>To unsubscribe from this list go to the following URL and read the<br>instructions:<br></blockquote></blockquote><a href="https://lists.samba.org/mailman/options/samba%5Bhttps://lists.samba.org/mailman/options/samba%5D" rel="noreferrer noreferrer" target="_blank">https://lists.samba.org/mailman/options/samba[https://lists.samba.org/mailman/options/samba]</a><br><br><blockquote type="cite"><br>--<br>To unsubscribe from this list go to the following URL and read the<br>instructions:<br></blockquote><a href="https://lists.samba.org/mailman/options/samba%5Bhttps://lists.samba.org/mailman/options/samba%5D" rel="noreferrer noreferrer" target="_blank">https://lists.samba.org/mailman/options/samba[https://lists.samba.org/mailman/options/samba]</a><br><br><br>------------------------------------------<br>illumos: illumos-developer<br>Permalink:<br><a href="https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M99a2afeb19714648c051ab5f" rel="noreferrer noreferrer" target="_blank">https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M99a2afeb19714648c051ab5f</a><br>Delivery options:<br><a href="https://illumos.topicbox.com/groups/developer/subscription" rel="noreferrer noreferrer" target="_blank">https://illumos.topicbox.com/groups/developer/subscription</a><br><br>*illumos <<a href="https://illumos.topicbox.com/latest" rel="noreferrer noreferrer" target="_blank">https://illumos.topicbox.com/latest</a>>* / illumos-developer /<br></blockquote></blockquote>see discussions <<a href="https://illumos.topicbox.com/groups/developer" rel="noreferrer noreferrer" target="_blank">https://illumos.topicbox.com/groups/developer</a>> +<br>participants <<a href="https://illumos.topicbox.com/groups/developer/members" rel="noreferrer noreferrer" target="_blank">https://illumos.topicbox.com/groups/developer/members</a>> + delivery<br>options <<a href="https://illumos.topicbox.com/groups/developer/subscription" rel="noreferrer noreferrer" target="_blank">https://illumos.topicbox.com/groups/developer/subscription</a>><br>Permalink<br><<a href="https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M2372380a78d4d5078f031f0c" rel="noreferrer noreferrer" target="_blank">https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M2372380a78d4d5078f031f0c</a>><br><br><br></blockquote></blockquote><br>------------------------------------------<br>illumos: illumos-developer<br>Permalink: <a href="https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M1048b12f29cda459a4de0893" rel="noreferrer noreferrer" target="_blank">https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M1048b12f29cda459a4de0893</a><br>Delivery options: <a href="https://illumos.topicbox.com/groups/developer/subscription" rel="noreferrer noreferrer" target="_blank">https://illumos.topicbox.com/groups/developer/subscription</a><br></div></div></blockquote></div><br></div></div>_______________________________________________<br>
oi-dev mailing list<br>
<a href="mailto:oi-dev@openindiana.org" rel="noreferrer noreferrer" target="_blank">oi-dev@openindiana.org</a><br>
<a href="https://openindiana.org/mailman/listinfo/oi-dev" rel="noreferrer noreferrer noreferrer" target="_blank">https://openindiana.org/mailman/listinfo/oi-dev</a><br>
</blockquote></div>
</div></blockquote><blockquote type="cite"><div dir="ltr"><span>_______________________________________________</span><br><span>oi-dev mailing list</span><br><span><a href="mailto:oi-dev@openindiana.org" target="_blank" rel="noreferrer">oi-dev@openindiana.org</a></span><br><span><a href="https://openindiana.org/mailman/listinfo/oi-dev" target="_blank" rel="noreferrer">https://openindiana.org/mailman/listinfo/oi-dev</a></span><br></div></blockquote></div>_______________________________________________<br>
oi-dev mailing list<br>
<a href="mailto:oi-dev@openindiana.org" target="_blank" rel="noreferrer">oi-dev@openindiana.org</a><br>
<a href="https://openindiana.org/mailman/listinfo/oi-dev" rel="noreferrer noreferrer" target="_blank">https://openindiana.org/mailman/listinfo/oi-dev</a><br>
</blockquote></div>
</div></blockquote><blockquote type="cite"><div dir="ltr"><span>_______________________________________________</span><br><span>oi-dev mailing list</span><br><span><a href="mailto:oi-dev@openindiana.org">oi-dev@openindiana.org</a></span><br><span><a href="https://openindiana.org/mailman/listinfo/oi-dev">https://openindiana.org/mailman/listinfo/oi-dev</a></span><br></div></blockquote></body></html>