[malloc dispatch] Install dispatch over the top of GWP-ASan.

When enabled, GWP-ASan sets the current dispatch table. Then, when a
shim layer (malloc_debug, malloc_hooks, heapprofd) comes along, they
should (by design) overwrite the current dispatch table.

Currently, these shim layers check to see whether malloc_limit is
installed by checking the current dispatch table against nullptr.
Because GWP-ASan owns the current dispatch table, the shim thinks that
malloc_limit is installed and falls back to only use the default
dispatch, thinking that malloc_limit will call them. This is not the
case, and they should take over the current dispatch pointer.

Bug: 135634846
Test: atest bionic

Change-Id: Ifb6f8864a15af9ac7f20d9364c40f73c5dd9d870

libc/bionic/malloc_common_dynamic.cpp

@@ -344,7 +344,7 @@ bool FinishInstallHooks(libc_globals* globals, const char* options, const char*
   // Do a pointer swap so that all of the functions become valid at once to
   // avoid any initialization order problems.
   atomic_store(&globals->default_dispatch_table, &globals->malloc_dispatch_table);
-  if (GetDispatchTable() == nullptr) {
+  if (!MallocLimitInstalled()) {
     atomic_store(&globals->current_dispatch_table, &globals->malloc_dispatch_table);
   }
 

libc/bionic/malloc_heapprofd.cpp

@@ -45,6 +45,7 @@
 #include "malloc_common.h"
 #include "malloc_common_dynamic.h"
 #include "malloc_heapprofd.h"
+#include "malloc_limit.h"
 
 static constexpr char kHeapprofdSharedLib[] = "heapprofd_client.so";
 static constexpr char kHeapprofdPrefix[] = "heapprofd";
@@ -189,7 +190,7 @@ void HandleHeapprofdSignal() {
       __libc_globals.mutate([](libc_globals* globals) {
         atomic_store(&globals->default_dispatch_table, &__heapprofd_init_dispatch);
         auto dispatch_table = GetDispatchTable();
-        if (dispatch_table == nullptr || dispatch_table == &globals->malloc_dispatch_table) {
+        if (!MallocLimitInstalled() || dispatch_table == &globals->malloc_dispatch_table) {
           atomic_store(&globals->current_dispatch_table, &__heapprofd_init_dispatch);
         }
       });

libc/bionic/malloc_limit.cpp

@@ -253,6 +253,10 @@ static void* LimitValloc(size_t bytes) {
 }
 #endif
 
+bool MallocLimitInstalled() {
+  return GetDispatchTable() == &__limit_dispatch;
+}
+
 #if defined(LIBC_STATIC)
 static bool EnableLimitDispatchTable() {
   // This is the only valid way to modify the dispatch tables for a

libc/bionic/malloc_limit.h

@@ -32,3 +32,7 @@
 
 // Function prototypes.
 bool LimitEnable(void* arg, size_t arg_size);
+
+// Returns true if malloc_limit is installed (by checking the current dispatch
+// table).
+bool MallocLimitInstalled();