Canaries and Redzones for C++ Memory Safety

What are Canaries and Redzones?

The two terms are related to memory safety for prevention and detection of memory areas. Redzones are regions of bytes around a memory block that are marked as invalid or "poisoned" for use. Canary values are a special type of redzone, with a single value, which is examined to see if it has changed.

There are various other terms used for these two approaches. Redzones are also called memory poisoning, memory tainting, memory tagging, memory coloring, and I've probably missed a few. Canaries are sometimes called sentinel values or guard values. The general techniques are referred to as memory safety or buffer overflow protection.

The main usage of redzones and canaries is to detect buffer overflows that result in array bounds violations, which are a common C++ bug and also a security vulnerability. These types of array buffer overflow attacks are more likely to be a security vulnerability if they occur in stack memory (rather than the heap), because the program stack can be corrupted intentionally. However, never underestimate human creativity, and many other memory errors can also be used as an attack vector. Surprisingly, one of the other major vulnerabilities is by abusing "dangling pointers" that arise from use-after-free errors.

What are Array Bounds Violations?

There are a lot of imprecise names used for basically the same thing:

• Array bounds violation
• Array overflow or array underflow
• Buffer overrun or underrun
• Buffer overwrite

Enough with the terminology; let's look at code! An example with string buffers on the stack looks like:

    char buf[3];
    strcpy(buf, "abcd");  // Boom! (buffer overflow)

An example of a buffer overwrite or overflow with an array looks like this:

    int arr[10];
    arr[10] = 0;    // Write error
    val = arr[55];  // Read error

And this is an array "underflow" error:

    arr[-1] = 0;  // Boom!

Note that watching for changes in canary values can only detect "write" array bounds errors, rather than reads, but more advanced methods with redzones can also detect some read accesses to redzone memory.

Text Buffer Last Byte Canaries

One of the simplest methods of using a canary is useful for text buffers. The very last byte of a memory block containing a text string can be used as a canary. This last byte must either be the null byte, if the string buffer is full, or an unused byte if the string is shorter. Hence, there is a trick where we set the last byte of a text buffer to the null byte, even if it's not going to be used. Then this last byte is a canary, where a non-zero value being found afterwards means it has overflowed at some previous point (i.e., a bounds overflow write error). Here's a raw example of how it works:

    char buf[100] = "";
    buf[99] = 0;  // Set up canary null value in last byte
    // ... Do stuff with buf
    if (buf[99] != 0) {  // Check at the end
        // Canary squawks! 
        // Text buffer has bounds-violate
    }

The advantage of this method is that it has no extra memory overhead, and only two fast single-byte operations (null byte assignment and testing for the null byte). However, it only works for text string with a null byte at the end, rather than for other types of arrays, and can only detect write errors (not reads). This use of the last byte of the text buffers as a canary was examined fully in a previous article on text buffer array bounds checking.

Array Extra Element Canaries

The idea of using the last element in text buffers as a canary can be generalized to non-text arrays. An array overflow error would look like this:

    int arr[10];
    arr[10] = 0;  // Error

To add a canary, we need to do this:

• Allocate one more element for the array.
• Set it to a magic value at the start.
• Check it still has the magic value at the end.

Here's the basic hand-coded idea:

    const int sz = 10;
    int arr[sz + 1];  // +1 for the canary

    // Set up the canary (last element)
    const unsigned magic = 0x12345678;
    arr[sz] = magic;

    // Do stuff....
    arr[10] = 0;  // Error

    // Check canary afterwards
    if (arr[sz] != magic) {
        // Overflow write error detected!
    }

Note the features of this canary technique:

• Works for any basic data type.
• Works for any array memory type (e.g., heap, stack, global, etc.)
• Canary value can be checked multiple times, rather than only at the very end.

The disadvantages include:

• After-the-fact detection of the overwrite, rather than immediately.
• Memory overhead is one extra array element per array.
• Time cost is setting one array element, and then checking it later.
• Need to disable this trick, or use a poisoning API, when running a sanitizer, because it interferes with their checks.

Redzones and Canaries for Memory Allocation Overflows

Array buffer overflows are the main reason to use redzones or canary values. These occur where an array access goes beyond the end of a valid array block, whether for a read access or a write access. Canary values can only detect writes, because they rely on the code changing the canary value, but redzones can also be used to detect reads.

The general idea is to add some extra memory to the end of an allocated block. We can intercept malloc or new memory primitives and replace them with wrapper versions that set aside some extra memory for use in error checking. Then we can check for modifications to these redzone or canary bytes, in which case an array write has occurred that is a bounds violation.

Hence, the basic steps are:

• Macro-intercepts of malloc, calloc, and free (also strdup and realloc, amongst others).
• Linker intercepts of new and delete (four versions with two basic and two array versions).
• Add extra bytes to be allocated in the memory block we return from our wrapper versions.
• Fill these extra redzone bytes with a special value.
• Detect uses of these special bytes later.

In advanced implementations, we can mark these redzone bytes with binary instrumentation or hardware-assisted pointer tagging.

Detection of Heap Underflows

Checking for underflows of heap addresses, such as addr[-1], is trickier because we cannot just add more memory to the start of the block. The region prior to an allocated heap block contains a system header block, which is used by the system allocator (i.e., the system's malloc or new primitives). Hence, we cannot just write a canary value to addr[-1], because doing so would be an underflow write error in itslef, which will trigger a crash. Our technique is supposed to prevent memory glitches, not cause them!

The tricky way to detect underflow is to allocate extra memory for this underflow redzone, but not in the system header block. The idea is to take a simple allocation:

    char *str = (char*)malloc(100);
    // Do stuff with 'str'
    free(str);

Instead, we allocated more memory, say 16 bytes, like this:

    char *mem = (char*)malloc(100 + 16);
    // Set up the redzone mem[0]..mem[15]
    char *str = mem + 16;  
    // Do stuff with 'str' ...
    // Check the redzone mem[0]..mem[15]
    free(str - 16);

This is messy, because we need to keep adding and subtracting the size of the redzone block (i.e., 16 bytes) but that's the overall idea. The first 16 bytes of the larger block are the redzone for underflow checking. The original code is passed a pointer to the middle of the block for use with the original code.

More generally, we can do this in a debug memory allocation library. As usual with this approach, the code needs to macro-intercept malloc and free, and link-intercept new and delete operators. There are several problems that make this plan difficult:

• Memory alignment of addresses
• Calls to free need to be offset (or it crashes!)
• new and delete cannot be used in manual code sequences like the above.
• Non-intercepted calls to malloc in third-party linked libraries will not have redzones and are thus problematic to deallocate.

Memory Read Errors

Read errors are those that access memory, but don't change the value. Some examples of memory safety concerns with read accesses include:

• Uninitialized memory usage
• Array bound overflow reads
• Array bound underflow reads
• Use-after-free errors
• Use-after-delete errors

Superficially, it might seem that these reads are less likely to be dangerous than writes. However, it's not really the case, because read errors can still be important to detect and prevent because they can be:

(a) crashes — e.g., segmentation faults.

(b) invalid results — e.g., reading the wrong values.

(c) attack vectors — use-after-free exploits are a major category of vulnerabilities.

Redzones can be used to detect read errors if it is possible to intercept read operations on an invalid block. There are several techniques in detecting memory read errors including uninitialized memory and use-after-free:

• Instrumentation of assembly or binary code
• Memory tagging (pointer tagging)
• Hardware-assisted exceptions
• Shadow memory

The technologies for hardware-assisted memory management include:

• ARM Memory Tagging Extension (ARM MTE)
• Intel Memory Protection Extensions (MPX)
• Sparc Application Data Integrity (ADI)

All of these techniques are somewhat beyond a basic DIY memory safety technique. These are the types of methods used in runtime memory checker tools such as Valgrind and AddressSanitizer. The basic idea is to set aside a redzone area of unused memory around every memory block, such as heap and stack memory, and then various methods are used to check every memory access for an invalid redzone address.

Prevention Versus Detection

Most of the above DIY techniques are about detecting memory safety issues, rather than preventing crashes or blocking security attackers using exploits. Full prevention requires instrumentation or hardware-assisted shadow memory, as done by sanitizers, but then the code tends to run too slow for use in production.

However, these techniques are great to use continually during development and testing. Some of the simpler methods are also fast enough to leave in production code, or at least when shipping to beta customers.

The idea is to find as many of these issues as possible. Hence, canary and redzone techniques should be combined with fuzzing and other types of stress testing, such as passing invalid or very long inputs to the code. And these methods are complementary to sanitizers, which should still be run in nightly builds of the regression test suites, and also sometimes combined with fuzzing and other longer tests.

Limitations of Canaries and Redzones

The canary and redzone techniques are not perfect, and won't do as well as a real runtime sanitizer tool. Some of the problems include:

• Canary value checks only detect prior failures (not immediately).
• Redzone techniques to detect overflows immediately are too difficult for DIY.
• Extra memory overhead to store the canary values and redzone bytes.
• Extra time cost of setting up canaries/redzones, and then later testing them.
• Read errors are much harder to detect than writes (almost impossible in DIY techniques).
• Crashes and memory corruption are not actually prevented (in most cases).
• Bounds violations further away than the length of the redzone will be missed.
• Security attacks are not actually prevented (and redzones can be worked around anyway).

Nevertheless, the goal of DIY canaries and redzones is to add some checking that detects a subset of failures, but is much faster than sanitizers, so it can be run 100% of the time, maybe even in production for customers.

Related Memory Safety Blog Articles

See also these articles:

• DIY Preventive C++ Memory Safety
• Use-After-Free Memory Errors in C++
• Array Bounds Violations and Memory Safe C++
• Poisoning Memory Blocks for Safer C++
• Uninitialized Memory Safety in C++
• DIY Memory Safety in C++
• CUDA C++ Floating Point Exceptions
• Memory Safe C++ Library Functions
• Smart Stack Buffers for Memory Safe C++
• Safe C++ Text Buffers with snprintf

Safe C++ Book

The new Safe C++ coding book by David Spuler: Memory Safety Rust versus C++ The Safe C++ Standard Pragmatic Memory Safety Get your copy from Amazon: Safe C++: Fixing Memory Safety Issues

More AI Research Topics

Read more about:

• « Aussie AI Blog Index
• Inference Optimizations
• List of Optimization Techniques
• AI Research Home