diff mbox series

[v1,01/12] configure: add --enable-tsan flag + fiber annotations for coroutine-ucontext

Message ID 20200529132341.755-1-robert.foley@linaro.org
State Superseded
Headers show
Series Add Thread Sanitizer support to QEMU | expand

Commit Message

Robert Foley May 29, 2020, 1:23 p.m. UTC
From: Lingfeng Yang <lfy@google.com>


We tried running QEMU under tsan in 2016, but tsan's lack of support for
longjmp-based fibers was a blocker:
  https://groups.google.com/forum/#!topic/thread-sanitizer/se0YuzfWazw

Fortunately, thread sanitizer gained fiber support in early 2019:
  https://reviews.llvm.org/D54889

This patch brings tsan support upstream by importing the patch that annotated
QEMU's coroutines as tsan fibers in Android's QEMU fork:
  https://android-review.googlesource.com/c/platform/external/qemu/+/844675

Tested with '--enable-tsan --cc=clang-9 --cxx=clang++-9 --disable-werror'
configure flags.

Signed-off-by: Lingfeng Yang <lfy@google.com>

Signed-off-by: Emilio G. Cota <cota@braap.org>

[cota: minor modifications + configure changes]
Signed-off-by: Robert Foley <robert.foley@linaro.org>

[RF: Error out in configure if tsan not available, fix checkpatch warnings]
---
 configure                 | 41 +++++++++++++++++
 util/coroutine-ucontext.c | 97 +++++++++++++++++++++++++++++++++++----
 2 files changed, 129 insertions(+), 9 deletions(-)

-- 
2.17.1

Comments

Eric Blake May 29, 2020, 1:51 p.m. UTC | #1
On 5/29/20 8:23 AM, Robert Foley wrote:
> From: Lingfeng Yang <lfy@google.com>

> 

> We tried running QEMU under tsan in 2016, but tsan's lack of support for

> longjmp-based fibers was a blocker:

>    https://groups.google.com/forum/#!topic/thread-sanitizer/se0YuzfWazw


[meta-comment]

This message lacked References: and In-Reply-To: headers, making it the 
start of a new thread for patches 1-11.  But I also see you sent a 0/12 
message with Message-Id: <20200529132143.702-1-robert.foley@linaro.org>, 
as well as an unthreaded 12/12 with Message-Id: 
<20200529132438.837-1-robert.foley@linaro.org>.  You may want to figure 
out why your sending workflow is not preserving desired threading.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
Robert Foley May 29, 2020, 2:56 p.m. UTC | #2
On Fri, 29 May 2020 at 09:51, Eric Blake <eblake@redhat.com> wrote:
>

> On 5/29/20 8:23 AM, Robert Foley wrote:

> > From: Lingfeng Yang <lfy@google.com>

> >

> > We tried running QEMU under tsan in 2016, but tsan's lack of support for

> > longjmp-based fibers was a blocker:

> >    https://groups.google.com/forum/#!topic/thread-sanitizer/se0YuzfWazw

>

> [meta-comment]

>

> This message lacked References: and In-Reply-To: headers, making it the

> start of a new thread for patches 1-11.  But I also see you sent a 0/12

> message with Message-Id: <20200529132143.702-1-robert.foley@linaro.org>,

> as well as an unthreaded 12/12 with Message-Id:

> <20200529132438.837-1-robert.foley@linaro.org>.  You may want to figure

> out why your sending workflow is not preserving desired threading.


Thanks for pointing this out.  I see what happened here and will fix
it for next time.

Thanks,
-Rob
> --

> Eric Blake, Principal Software Engineer

> Red Hat, Inc.           +1-919-301-3226

> Virtualization:  qemu.org | libvirt.org

>
Alex Bennée June 2, 2020, 7:22 p.m. UTC | #3
Robert Foley <robert.foley@linaro.org> writes:

> From: Lingfeng Yang <lfy@google.com>

>

> We tried running QEMU under tsan in 2016, but tsan's lack of support for

> longjmp-based fibers was a blocker:

>   https://groups.google.com/forum/#!topic/thread-sanitizer/se0YuzfWazw

>

> Fortunately, thread sanitizer gained fiber support in early 2019:

>   https://reviews.llvm.org/D54889

>

> This patch brings tsan support upstream by importing the patch that annotated

> QEMU's coroutines as tsan fibers in Android's QEMU fork:

>   https://android-review.googlesource.com/c/platform/external/qemu/+/844675

>

> Tested with '--enable-tsan --cc=clang-9 --cxx=clang++-9 --disable-werror'

> configure flags.

>

> Signed-off-by: Lingfeng Yang <lfy@google.com>

> Signed-off-by: Emilio G. Cota <cota@braap.org>

> [cota: minor modifications + configure changes]

> Signed-off-by: Robert Foley <robert.foley@linaro.org>

> [RF: Error out in configure if tsan not available, fix checkpatch warnings]

> ---

>  configure                 | 41 +++++++++++++++++

>  util/coroutine-ucontext.c | 97 +++++++++++++++++++++++++++++++++++----

>  2 files changed, 129 insertions(+), 9 deletions(-)

>

> diff --git a/configure b/configure

> index b969dee675..c18efae65e 100755

> --- a/configure

> +++ b/configure

> @@ -395,6 +395,7 @@ gprof="no"

>  debug_tcg="no"

>  debug="no"

>  sanitizers="no"

> +tsan="no"

>  fortify_source=""

>  strip_opt="yes"

>  tcg_interpreter="no"

> @@ -1150,6 +1151,10 @@ for opt do

>    ;;

>    --disable-sanitizers) sanitizers="no"

>    ;;

> +  --enable-tsan) tsan="yes"

> +  ;;

> +  --disable-tsan) tsan="no"

> +  ;;

>    --enable-sparse) sparse="yes"

>    ;;

>    --disable-sparse) sparse="no"

> @@ -1750,6 +1755,7 @@ Advanced options (experts only):

>    --with-pkgversion=VERS   use specified string as sub-version of the package

>    --enable-debug           enable common debug build options

>    --enable-sanitizers      enable default sanitizers

> +  --enable-tsan            enable thread sanitizer

>    --disable-strip          disable stripping binaries

>    --disable-werror         disable compilation abort on warning

>    --disable-stack-protector disable compiler-provided stack protection

> @@ -6192,6 +6198,27 @@ if test "$fuzzing" = "yes" ; then

>    fi

>  fi

>  

> +# Thread sanitizer is, for now, much noisier than the other sanitizers;

> +# keep it separate until that is not the case.


I think we also need to stop both being enabled at once. On my clang-9
setup I get:

  make: *** [qapi/qobject-output-visitor.o] Error 1
  clang: error: invalid argument '-fsanitize=address' not allowed with '-fsanitize=thread'
  clang: error: invalid argument '-fsanitize=address' not allowed with '-fsanitize=thread'
  clang: errorclang: : errorinvalid argument '-fsanitize=address' not allowed with '-fsanitize=thread':
  invalid argument '-fsanitize=address' not allowed with '-fsanitize=thread'
  clang: error: invalid argument '-fsanitize=address' not allowed with '-fsanitize=thread'

> +have_tsan=no

> +have_tsan_iface_fiber=no

> +if test "$tsan" = "yes" ; then

> +  write_c_skeleton

> +  if compile_prog "$CPU_CFLAGS -Werror -fsanitize=thread" "" ; then

> +      have_tsan=yes

> +  fi

> +  cat > $TMPC << EOF

> +#include <sanitizer/tsan_interface.h>

> +int main(void) {

> +  __tsan_create_fiber(0);

> +  return 0;

> +}

> +EOF

> +  if compile_prog "$CPU_CFLAGS -Werror -fsanitize=thread" "" ; then

> +      have_tsan_iface_fiber=yes

> +  fi

> +fi

> +

>  ##########################################

>  # check for libpmem

>  

> @@ -6293,6 +6320,16 @@ if test "$have_asan" = "yes"; then

>             "Without code annotation, the report may be inferior."

>    fi

>  fi

> +if test "$have_tsan" = "yes" ; then

> +  if test "$have_tsan_iface_fiber" = "yes" ; then

> +    QEMU_CFLAGS="-fsanitize=thread $QEMU_CFLAGS"

> +    QEMU_LDFLAGS="-fsanitize=thread $QEMU_LDFLAGS"

> +  else

> +    error_exit "Cannot enable TSAN due to missing fiber annotation interface."

> +  fi

> +elif test "$tsan" = "yes" ; then

> +  error_exit "Cannot enable TSAN due to missing sanitize thread interface."

> +fi

>  if test "$have_ubsan" = "yes"; then

>    QEMU_CFLAGS="-fsanitize=undefined $QEMU_CFLAGS"

>    QEMU_LDFLAGS="-fsanitize=undefined $QEMU_LDFLAGS"

> @@ -7382,6 +7419,10 @@ if test "$have_asan_iface_fiber" = "yes" ; then

>      echo "CONFIG_ASAN_IFACE_FIBER=y" >> $config_host_mak

>  fi



Are we missing any LDFLAGS? On Ubuntu 18.04 with clang-9 I'm seeing:

    LINK    qemu-ga
  /usr/lib/llvm-9/lib/clang/9.0.0/lib/linux/libclang_rt.tsan-x86_64.a(tsan_rtl_amd64.S.o): warning: common of `__interception::real_setjmp' overridden by definition
  /usr/lib/llvm-9/lib/clang/9.0.0/lib/linux/libclang_rt.tsan-x86_64.a(tsan_interceptors.cc.o): warning: defined here
  /usr/lib/llvm-9/lib/clang/9.0.0/lib/linux/libclang_rt.tsan-x86_64.a(tsan_rtl_amd64.S.o): warning: common of `__interception::real__setjmp' overridden by definition
  /usr/lib/llvm-9/lib/clang/9.0.0/lib/linux/libclang_rt.tsan-x86_64.a(tsan_interceptors.cc.o): warning: defined here
  /usr/lib/llvm-9/lib/clang/9.0.0/lib/linux/libclang_rt.tsan-x86_64.a(tsan_rtl_amd64.S.o): warning: common of `__interception::real_sigsetjmp' overridden by definition
  /usr/lib/llvm-9/lib/clang/9.0.0/lib/linux/libclang_rt.tsan-x86_64.a(tsan_interceptors.cc.o): warning: defined here
  /usr/lib/llvm-9/lib/clang/9.0.0/lib/linux/libclang_rt.tsan-x86_64.a(tsan_rtl_amd64.S.o): warning: common of `__interception::real___sigsetjmp' overridden by definition
  /usr/lib/llvm-9/lib/clang/9.0.0/lib/linux/libclang_rt.tsan-x86_64.a(tsan_interceptors.cc.o): warning: defined here
  libqemuutil.a(osdep.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:41: multiple definition of `__tsan_mutex_linker_init'
  libqemuutil.a(control.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:41: first defined here
  libqemuutil.a(osdep.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:50: multiple definition of `__tsan_mutex_not_static'
  libqemuutil.a(control.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:50: first defined here
  libqemuutil.a(osdep.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:55: multiple definition of `__tsan_mutex_read_lock'
  libqemuutil.a(control.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:55: first defined here
  libqemuutil.a(osdep.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:45: multiple definition of `__tsan_mutex_read_reentrant'
  libqemuutil.a(control.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:45: first defined here
  libqemuutil.a(osdep.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:64: multiple definition of `__tsan_mutex_recursive_lock'
  libqemuutil.a(control.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:64: first defined here
  libqemuutil.a(osdep.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:68: multiple definition of `__tsan_mutex_recursive_unlock'

<snip>

-- 
Alex Bennée
Robert Foley June 3, 2020, 4:51 p.m. UTC | #4
On Tue, 2 Jun 2020 at 15:22, Alex Bennée <alex.bennee@linaro.org> wrote:
>

>

> Robert Foley <robert.foley@linaro.org> writes:

>

> > From: Lingfeng Yang <lfy@google.com>

<snip>
> >

> > +# Thread sanitizer is, for now, much noisier than the other sanitizers;

> > +# keep it separate until that is not the case.

>

> I think we also need to stop both being enabled at once. On my clang-9

> setup I get:

>

>   make: *** [qapi/qobject-output-visitor.o] Error 1

>   clang: error: invalid argument '-fsanitize=address' not allowed with '-fsanitize=thread'

>   clang: error: invalid argument '-fsanitize=address' not allowed with '-fsanitize=thread'

>   clang: errorclang: : errorinvalid argument '-fsanitize=address' not allowed with '-fsanitize=thread':

>   invalid argument '-fsanitize=address' not allowed with '-fsanitize=thread'

>   clang: error: invalid argument '-fsanitize=address' not allowed with '-fsanitize=thread'


Good point.  I see a similar error if I use --enable-sanitizers and
--enable-tsan.
Will look to add an error check for this.  Wondering if there are any
other interactions
we need to avoid?  Will poke around a bit here.

<snip>

>

> Are we missing any LDFLAGS? On Ubuntu 18.04 with clang-9 I'm seeing:

>

>     LINK    qemu-ga

>   /usr/lib/llvm-9/lib/clang/9.0.0/lib/linux/libclang_rt.tsan-x86_64.a(tsan_rtl_amd64.S.o): warning: common of `__interception::real_setjmp' overridden by definition

>   /usr/lib/llvm-9/lib/clang/9.0.0/lib/linux/libclang_rt.tsan-x86_64.a(tsan_interceptors.cc.o): warning: defined here


Looks like these warnings come from use of --warn-common, so I think we need to
exclude this when using TSan if we want to silence these warnings from
the TSan libraries.

Thanks & Regards,
-Rob

>   /usr/lib/llvm-9/lib/clang/9.0.0/lib/linux/libclang_rt.tsan-x86_64.a(tsan_rtl_amd64.S.o): warning: common of `__interception::real__setjmp' overridden by definition

>   /usr/lib/llvm-9/lib/clang/9.0.0/lib/linux/libclang_rt.tsan-x86_64.a(tsan_interceptors.cc.o): warning: defined here

>   /usr/lib/llvm-9/lib/clang/9.0.0/lib/linux/libclang_rt.tsan-x86_64.a(tsan_rtl_amd64.S.o): warning: common of `__interception::real_sigsetjmp' overridden by definition

>   /usr/lib/llvm-9/lib/clang/9.0.0/lib/linux/libclang_rt.tsan-x86_64.a(tsan_interceptors.cc.o): warning: defined here

>   /usr/lib/llvm-9/lib/clang/9.0.0/lib/linux/libclang_rt.tsan-x86_64.a(tsan_rtl_amd64.S.o): warning: common of `__interception::real___sigsetjmp' overridden by definition

>   /usr/lib/llvm-9/lib/clang/9.0.0/lib/linux/libclang_rt.tsan-x86_64.a(tsan_interceptors.cc.o): warning: defined here

>   libqemuutil.a(osdep.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:41: multiple definition of `__tsan_mutex_linker_init'

>   libqemuutil.a(control.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:41: first defined here

>   libqemuutil.a(osdep.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:50: multiple definition of `__tsan_mutex_not_static'

>   libqemuutil.a(control.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:50: first defined here

>   libqemuutil.a(osdep.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:55: multiple definition of `__tsan_mutex_read_lock'

>   libqemuutil.a(control.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:55: first defined here

>   libqemuutil.a(osdep.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:45: multiple definition of `__tsan_mutex_read_reentrant'

>   libqemuutil.a(control.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:45: first defined here

>   libqemuutil.a(osdep.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:64: multiple definition of `__tsan_mutex_recursive_lock'

>   libqemuutil.a(control.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:64: first defined here

>   libqemuutil.a(osdep.o):/usr/lib/llvm-9/lib/clang/9.0.0/include/sanitizer/tsan_interface.h:68: multiple definition of `__tsan_mutex_recursive_unlock'

>

> <snip>

>

> --

> Alex Bennée
diff mbox series

Patch

diff --git a/configure b/configure
index b969dee675..c18efae65e 100755
--- a/configure
+++ b/configure
@@ -395,6 +395,7 @@  gprof="no"
 debug_tcg="no"
 debug="no"
 sanitizers="no"
+tsan="no"
 fortify_source=""
 strip_opt="yes"
 tcg_interpreter="no"
@@ -1150,6 +1151,10 @@  for opt do
   ;;
   --disable-sanitizers) sanitizers="no"
   ;;
+  --enable-tsan) tsan="yes"
+  ;;
+  --disable-tsan) tsan="no"
+  ;;
   --enable-sparse) sparse="yes"
   ;;
   --disable-sparse) sparse="no"
@@ -1750,6 +1755,7 @@  Advanced options (experts only):
   --with-pkgversion=VERS   use specified string as sub-version of the package
   --enable-debug           enable common debug build options
   --enable-sanitizers      enable default sanitizers
+  --enable-tsan            enable thread sanitizer
   --disable-strip          disable stripping binaries
   --disable-werror         disable compilation abort on warning
   --disable-stack-protector disable compiler-provided stack protection
@@ -6192,6 +6198,27 @@  if test "$fuzzing" = "yes" ; then
   fi
 fi
 
+# Thread sanitizer is, for now, much noisier than the other sanitizers;
+# keep it separate until that is not the case.
+have_tsan=no
+have_tsan_iface_fiber=no
+if test "$tsan" = "yes" ; then
+  write_c_skeleton
+  if compile_prog "$CPU_CFLAGS -Werror -fsanitize=thread" "" ; then
+      have_tsan=yes
+  fi
+  cat > $TMPC << EOF
+#include <sanitizer/tsan_interface.h>
+int main(void) {
+  __tsan_create_fiber(0);
+  return 0;
+}
+EOF
+  if compile_prog "$CPU_CFLAGS -Werror -fsanitize=thread" "" ; then
+      have_tsan_iface_fiber=yes
+  fi
+fi
+
 ##########################################
 # check for libpmem
 
@@ -6293,6 +6320,16 @@  if test "$have_asan" = "yes"; then
            "Without code annotation, the report may be inferior."
   fi
 fi
+if test "$have_tsan" = "yes" ; then
+  if test "$have_tsan_iface_fiber" = "yes" ; then
+    QEMU_CFLAGS="-fsanitize=thread $QEMU_CFLAGS"
+    QEMU_LDFLAGS="-fsanitize=thread $QEMU_LDFLAGS"
+  else
+    error_exit "Cannot enable TSAN due to missing fiber annotation interface."
+  fi
+elif test "$tsan" = "yes" ; then
+  error_exit "Cannot enable TSAN due to missing sanitize thread interface."
+fi
 if test "$have_ubsan" = "yes"; then
   QEMU_CFLAGS="-fsanitize=undefined $QEMU_CFLAGS"
   QEMU_LDFLAGS="-fsanitize=undefined $QEMU_LDFLAGS"
@@ -7382,6 +7419,10 @@  if test "$have_asan_iface_fiber" = "yes" ; then
     echo "CONFIG_ASAN_IFACE_FIBER=y" >> $config_host_mak
 fi
 
+if test "$have_tsan" = "yes" && test "$have_tsan_iface_fiber" = "yes" ; then
+    echo "CONFIG_TSAN=y" >> $config_host_mak
+fi
+
 if test "$has_environ" = "yes" ; then
   echo "CONFIG_HAS_ENVIRON=y" >> $config_host_mak
 fi
diff --git a/util/coroutine-ucontext.c b/util/coroutine-ucontext.c
index bd593e61bc..a3dc78e67a 100644
--- a/util/coroutine-ucontext.c
+++ b/util/coroutine-ucontext.c
@@ -37,18 +37,33 @@ 
 #endif
 #endif
 
+#ifdef CONFIG_TSAN
+#include <sanitizer/tsan_interface.h>
+#endif
+
 typedef struct {
     Coroutine base;
     void *stack;
     size_t stack_size;
     sigjmp_buf env;
 
+    void *tsan_co_fiber;
+    void *tsan_caller_fiber;
+
 #ifdef CONFIG_VALGRIND_H
     unsigned int valgrind_stack_id;
 #endif
 
 } CoroutineUContext;
 
+#define UC_DEBUG 0
+#if UC_DEBUG && defined(CONFIG_TSAN)
+#define UC_TRACE(fmt, ...) fprintf(stderr, "%s:%d:%p " fmt "\n", \
+    __func__, __LINE__, __tsan_get_current_fiber(), ##__VA_ARGS__);
+#else
+#define UC_TRACE(fmt, ...)
+#endif
+
 /**
  * Per-thread coroutine bookkeeping
  */
@@ -65,7 +80,20 @@  union cc_arg {
     int i[2];
 };
 
-static void finish_switch_fiber(void *fake_stack_save)
+/* QEMU_ALWAYS_INLINE only does so if __OPTIMIZE__, so we cannot use it. */
+static inline __attribute__((always_inline))
+void on_new_fiber(CoroutineUContext *co)
+{
+#ifdef CONFIG_TSAN
+    co->tsan_co_fiber = __tsan_create_fiber(0); /* flags: sync on switch */
+    co->tsan_caller_fiber = __tsan_get_current_fiber();
+    UC_TRACE("Create new TSAN co fiber. co: %p co fiber: %p caller fiber: %p ",
+             co, co->tsan_co_fiber, co->tsan_caller_fiber);
+#endif
+}
+
+static inline __attribute__((always_inline))
+void finish_switch_fiber(void *fake_stack_save)
 {
 #ifdef CONFIG_ASAN
     const void *bottom_old;
@@ -78,18 +106,40 @@  static void finish_switch_fiber(void *fake_stack_save)
         leader.stack_size = size_old;
     }
 #endif
+#ifdef CONFIG_TSAN
+    if (fake_stack_save) {
+        __tsan_release(fake_stack_save);
+        __tsan_switch_to_fiber(fake_stack_save, 0);  /* 0=synchronize */
+    }
+#endif
 }
 
-static void start_switch_fiber(void **fake_stack_save,
-                               const void *bottom, size_t size)
+static inline __attribute__((always_inline)) void start_switch_fiber(
+    CoroutineAction action, void **fake_stack_save,
+    const void *bottom, size_t size, void *new_fiber)
 {
 #ifdef CONFIG_ASAN
-    __sanitizer_start_switch_fiber(fake_stack_save, bottom, size);
+    if (action == COROUTINE_TERMINATE) {
+        __sanitizer_start_switch_fiber(
+            action == COROUTINE_TERMINATE ? NULL : fake_stack_save,
+            bottom, size);
+    }
+#endif
+#ifdef CONFIG_TSAN
+    void *curr_fiber =
+        __tsan_get_current_fiber();
+    __tsan_acquire(curr_fiber);
+
+    UC_TRACE("Current fiber: %p.", curr_fiber);
+    *fake_stack_save = curr_fiber;
+    UC_TRACE("Switch to fiber %p", new_fiber);
+    __tsan_switch_to_fiber(new_fiber, 0);  /* 0=synchronize */
 #endif
 }
 
 static void coroutine_trampoline(int i0, int i1)
 {
+    UC_TRACE("Start trampoline");
     union cc_arg arg;
     CoroutineUContext *self;
     Coroutine *co;
@@ -104,21 +154,34 @@  static void coroutine_trampoline(int i0, int i1)
 
     /* Initialize longjmp environment and switch back the caller */
     if (!sigsetjmp(self->env, 0)) {
-        start_switch_fiber(&fake_stack_save,
-                           leader.stack, leader.stack_size);
+        UC_TRACE("Current fiber: %p. Set co %p to env 0x%lx",
+                 __tsan_get_current_fiber(), self, (unsigned long)self->env);
+        start_switch_fiber(
+            COROUTINE_YIELD,
+            &fake_stack_save,
+            leader.stack,
+            leader.stack_size,
+            self->tsan_caller_fiber);
+        UC_TRACE("Jump to co %p caller fiber %p env 0x%lx",
+                 co, self->tsan_caller_fiber, *(unsigned long *)co->entry_arg);
         siglongjmp(*(sigjmp_buf *)co->entry_arg, 1);
     }
 
+    UC_TRACE("After first siglongjmp");
+
     finish_switch_fiber(fake_stack_save);
 
     while (true) {
         co->entry(co->entry_arg);
+        UC_TRACE("switch from co %p to caller co %p fiber %p\n",
+                 co, co->caller, self->tsan_caller_fiber);
         qemu_coroutine_switch(co, co->caller, COROUTINE_TERMINATE);
     }
 }
 
 Coroutine *qemu_coroutine_new(void)
 {
+    UC_TRACE("Start new coroutine");
     CoroutineUContext *co;
     ucontext_t old_uc, uc;
     sigjmp_buf old_env;
@@ -154,12 +217,16 @@  Coroutine *qemu_coroutine_new(void)
 
     arg.p = co;
 
+    on_new_fiber(co);
     makecontext(&uc, (void (*)(void))coroutine_trampoline,
                 2, arg.i[0], arg.i[1]);
 
     /* swapcontext() in, siglongjmp() back out */
     if (!sigsetjmp(old_env, 0)) {
-        start_switch_fiber(&fake_stack_save, co->stack, co->stack_size);
+        start_switch_fiber(
+            COROUTINE_YIELD,
+            &fake_stack_save,
+            co->stack, co->stack_size, co->tsan_co_fiber);
         swapcontext(&old_uc, &uc);
     }
 
@@ -185,6 +252,7 @@  static inline void valgrind_stack_deregister(CoroutineUContext *co)
 
 void qemu_coroutine_delete(Coroutine *co_)
 {
+    UC_TRACE("Nuking co %p from orbit", co_);
     CoroutineUContext *co = DO_UPCAST(CoroutineUContext, base, co_);
 
 #ifdef CONFIG_VALGRIND_H
@@ -209,6 +277,10 @@  qemu_coroutine_switch(Coroutine *from_, Coroutine *to_,
 {
     CoroutineUContext *from = DO_UPCAST(CoroutineUContext, base, from_);
     CoroutineUContext *to = DO_UPCAST(CoroutineUContext, base, to_);
+    UC_TRACE("from to: %p %p uc: %p %p. fibers: %p %p caller fibers: %p %p\n",
+            from_, to_, from, to,
+            from->tsan_co_fiber, to->tsan_co_fiber,
+            from->tsan_caller_fiber, to->tsan_caller_fiber);
     int ret;
     void *fake_stack_save = NULL;
 
@@ -216,8 +288,8 @@  qemu_coroutine_switch(Coroutine *from_, Coroutine *to_,
 
     ret = sigsetjmp(from->env, 0);
     if (ret == 0) {
-        start_switch_fiber(action == COROUTINE_TERMINATE ?
-                           NULL : &fake_stack_save, to->stack, to->stack_size);
+        start_switch_fiber(action, &fake_stack_save,
+                           to->stack, to->stack_size, to->tsan_co_fiber);
         siglongjmp(to->env, action);
     }
 
@@ -231,6 +303,13 @@  Coroutine *qemu_coroutine_self(void)
     if (!current) {
         current = &leader.base;
     }
+#ifdef CONFIG_TSAN
+    if (!leader.tsan_co_fiber) {
+        leader.tsan_co_fiber = __tsan_get_current_fiber();
+        UC_TRACE("For co %p set leader co fiber to %p",
+                 current, leader.tsan_co_fiber);
+    }
+#endif
     return current;
 }