[oi-dev] Opening executable with write flags

[Andreas Wacknitz]

Am 14.07.25 um 11:39 schrieb Paul Floyd via oi-dev:
> Hi
>
> I'm working on an issue related to an exe opening its own binary file 
> with flags like O_WRONLY|O_CREAT|O_TRUNC. Any writing is going to be 
> bad, and truncation even worse.
>
> On FreeBSD and Linux this is not allowed. 'open' will fail and set 
> errno to ETXTBSY.
>
> I haven't yet tried this on Solaris.  My testcase just opens and 
> truncates the exe and core dumps on the first call to open.
If I read open's explanation in the man page (man -s 2 open) I cannot 
find a hint that what you do is forbidden on illumos. So I think what 
you see is the expected behavior. Maybe nobody thought about your 
special case when it was designed.


Andreas

>
> Under gdb I get
>
> Dwarf Error: wrong version in compilation unit header (is 0, should be 
> 2, 3, 4 or 5) [in module 
> /export/home/paulf/scratch/valgrind/memcheck/tests/solaris/open_client]
>
> I guess that;s because it's failing to rread the Dwarf from the 0 byte 
> file.
>
> Here is my testcase, works fine on FreeBSD
>
> // For Bug 505673
> // Valgrind crashes with an internal error and SIGBUS when the guest 
> tries to open its own file with O_WRONLY|O_CREAT|O_TRUNC
> #include <fcntl.h>
> #include <cerrno>
> #include <stdexcept>
> #include <vector>
> #include <unistd.h>
> #include <sys/syscall.h>
> #include <unistd.h>
>
> int main(int argc, char** argv)
> {
>     std::vector<int> flags{O_WRONLY|O_CREAT|O_TRUNC, O_WRONLY, O_RDWR};
>
>     // On FreeBSD libc open uses syscall openat (at least on 14.2)
>     for (auto f : flags)
>     {
>         int res = open(argv[0], f, 0666);
>         if (-1 != res)
>         {
>             throw std::runtime_error("open should have failed");
>         }
>         else
>         {
>             if (errno != ETXTBSY)
>             {
>                 throw std::runtime_error("errno should be ETXTBSY");
>             }
>         }
>     }
>
>     // repeat the above, but with syscall(SYS_OPEN
>     for (auto f : flags)
>     {
>         int res = syscall(SYS_open, argv[0], f, 0666);
>         if (-1 != res)
>         {
>             throw std::runtime_error("open should have failed");
>         }
>         else
>         {
>             if (errno != ETXTBSY)
>             {
>                 throw std::runtime_error("errno should be ETXTBSY");
>             }
>         }
>     }
>
>     chdir("..");
>
>     // check that relative paths work
>     for (auto f : flags)
>     {
>         int res = open("solaris/open_client", f, 0666);
>         if (-1 != res)
>         {
>             throw std::runtime_error("open should have failed");
>         }
>         else
>         {
>             if (errno != ETXTBSY)
>             {
>                 throw std::runtime_error("errno should be ETXTBSY");
>             }
>         }
>     }
>
>     for (auto f : flags)
>     {
>         int res = syscall(SYS_open, "solaris/open_client", f, 0666);
>         if (-1 != res)
>         {
>             throw std::runtime_error("open should have failed");
>         }
>         else
>         {
>             if (errno != ETXTBSY)
>             {
>                 throw std::runtime_error("errno should be ETXTBSY");
>             }
>         }
>     }
> }
>
> A+
>
> Paul
>
>
>
> _______________________________________________
> oi-dev mailing list
> oi-dev at openindiana.org
> https://openindiana.org/mailman/listinfo/oi-dev