[oi-dev] Opening executable with write flags

[Paul Floyd]

Hi

I'm working on an issue related to an exe opening its own binary file 
with flags like O_WRONLY|O_CREAT|O_TRUNC. Any writing is going to be 
bad, and truncation even worse.

On FreeBSD and Linux this is not allowed. 'open' will fail and set errno 
to ETXTBSY.

I haven't yet tried this on Solaris.  My testcase just opens and 
truncates the exe and core dumps on the first call to open.

Under gdb I get

Dwarf Error: wrong version in compilation unit header (is 0, should be 
2, 3, 4 or 5) [in module 
/export/home/paulf/scratch/valgrind/memcheck/tests/solaris/open_client]

I guess that;s because it's failing to rread the Dwarf from the 0 byte file.

Here is my testcase, works fine on FreeBSD

// For Bug 505673
// Valgrind crashes with an internal error and SIGBUS when the guest 
tries to open its own file with O_WRONLY|O_CREAT|O_TRUNC
#include <fcntl.h>
#include <cerrno>
#include <stdexcept>
#include <vector>
#include <unistd.h>
#include <sys/syscall.h>
#include <unistd.h>

int main(int argc, char** argv)
{
     std::vector<int> flags{O_WRONLY|O_CREAT|O_TRUNC, O_WRONLY, O_RDWR};

     // On FreeBSD libc open uses syscall openat (at least on 14.2)
     for (auto f : flags)
     {
         int res = open(argv[0], f, 0666);
         if (-1 != res)
         {
             throw std::runtime_error("open should have failed");
         }
         else
         {
             if (errno != ETXTBSY)
             {
                 throw std::runtime_error("errno should be ETXTBSY");
             }
         }
     }

     // repeat the above, but with syscall(SYS_OPEN
     for (auto f : flags)
     {
         int res = syscall(SYS_open, argv[0], f, 0666);
         if (-1 != res)
         {
             throw std::runtime_error("open should have failed");
         }
         else
         {
             if (errno != ETXTBSY)
             {
                 throw std::runtime_error("errno should be ETXTBSY");
             }
         }
     }

     chdir("..");

     // check that relative paths work
     for (auto f : flags)
     {
         int res = open("solaris/open_client", f, 0666);
         if (-1 != res)
         {
             throw std::runtime_error("open should have failed");
         }
         else
         {
             if (errno != ETXTBSY)
             {
                 throw std::runtime_error("errno should be ETXTBSY");
             }
         }
     }

     for (auto f : flags)
     {
         int res = syscall(SYS_open, "solaris/open_client", f, 0666);
         if (-1 != res)
         {
             throw std::runtime_error("open should have failed");
         }
         else
         {
             if (errno != ETXTBSY)
             {
                 throw std::runtime_error("errno should be ETXTBSY");
             }
         }
     }
}

A+

Paul