41. Self-Testing Code

“Knowing yourself is the beginning of all wisdom.”

— Aristotle.

 

 

Instead of doing work yourself, get a machine to do it. Who would have ever thought of that?

Getting your code to test itself means you can go get a cup of coffee and still be billing hours. The basic techniques include:

• Unit tests
• Regression tests
• Assertions
• Self-testing code blocks
• Debug wrapper functions

The simplest of these is unit tests, which aim to build quality brick-by-brick from the bottom of the code hierarchy.

AI Engine Automated Testing

Automated testing can be applied to AI engines and incorporated into every build. There are two main types of testing, unit tests and regression tests, although there's some overlap between them.

Unit Testing. Unit tests have saved my bacon so many times. And yet, I have also often neglected to unit test a function, only to regret it later. The more unit tests you add, the better. When I express the unpopular view that rewriting code adds technical debt, rather than reducing it, the absence of unit tests is one of the big issues.

The benefit from unit testing and regression testing is just as much for an AI Engine as any other large project. An AI engine's unit tests would be things like testing a vector dot product kernel, whereas a regression test would be running an entire query through all the layers of the model to get an output response.

Unit Testing Harness. The unit testing harness is the code that runs multiple unit tests. The JUnit tool for Java was a major advance at the time, leapfrogging past C++, but there are now a variety of free C++ test harness libraries including:

• Google Test (BSD-3-clause license)
• Boost.Test (Boost Software License)
• CppUnit (a port of Java's JUnit; LGPLv2 license)

Alternatively, you can create one yourself. I often build my own simple unit test API with basic features including:

• Test API versions for Boolean, integer, and float types.
• Emits error context information on failures (e.g. filename, line).
• Tracks a count of failures.
• Tracks the total number of tests done.
• Reports success or failure at the end.

Regression Testing. Unit tests are distinct from “regression tests.” The idea with unit tests is to call very low-level C++ functions to ensure they work in a basic way. Regression tests are much higher-level and test whether the whole component is running as desired.

A typical regression test for an AI engine is to pass it a query and check that its answer is the same as it was last Tuesday. To do so, you need a batch method to run a prompt through the model, such as a command-line interface.

In order to successfully run a full regression test through an AI engine, you need to control everything. This involves managing various aspects of the code to ensure that the regression test case will still produce identical output, including:

• Random number seed
• Time-specific code
• RAG component (“retriever”)
• Inference cache component
• Data source integrations
• Tool interfaces
• Model file

Randomness in the top-k decoding algorithm is the enemy of regression tests. But you can simply curtail the model's creativity by using a fixed random number seed. This can be done either by allowing the regression test to specify a seed, or having a special “testing” mode where a hard-coded fixed seed is used.

Another area that can change is the current time. The AI engine probably interfaces with a “tool” that helps it answer time-specific questions. To make this consistent in regression tests, you also need to intercept these API calls in a way that the test harness can specify what time to use.

To the extent that the AI engine relies on other major components, you need to control their interactions for regression testing. For example, a regression test of the engine itself needs to provide a fixed input from the RAG retrieval component. You may need to alter the RAG component so that it logs its answers to a file, which can then be re-used in regression tests.

Similarly, anything else that alters the conditions of a test needs to be made fixed. We probably want to disable any caching components, except when we're trying to test the cache module itself. Integrations with data sources and tools are also areas where the AI engine sends requests and receives data, so this needs to be predictable in order for regression tests to work correctly.

The simplest idea is that the AI engine itself could simply log its own input prompts and output results for later usage in testing. It would also need to log its configuration settings (e.g. temperature), random number seed, and time-related data. In this way, a huge test suite of prompt-response pairs with fixed configuration settings can be generated for later re-testing.

Finally, note that we don't change the model file for regression testing. The goal of regression testing is not to test the model; that's called “model evaluation” and is a separate discipline. Instead, regression testing aims to re-check the C++ in the Transformer engine's kernels, and actually works best if we keep the old model files around and re-use them. An updated model is unlikely to output exactly the same output on a large range of inputs (nor would we want it to!), so you need to build a new regression test suite for a new model.

Advanced regression testing. The above section assumes you are sending the same inputs and testing for the exact output string. Another more flexible way is to test the results approximately using techniques such as:

• Substring match (e.g. factual questions get an answer containing the right words).
• Vector match (i.e., the answer's vector embeddings are “close enough” semantically to the expected result).
• Disallowed words (e.g., substring match to ensure “non-safe” words are absent).

Many of these methods are going beyond basic regression testing and into the area of “model evaluation.” Running a suite of test prompts through a new model is one part of evaluating its smartness and safety.

Test Coverage

Test coverage is the concept of checking that your testing code is executing all the features of the program. The idea is to run your entire suite of unit tests and regression tests, and then get a report showing which parts of the code didn't get executed.

There are multiple levels of test coverage:

• Function coverage. Which functions didn't get called?
• Line coverage. Which lines of code didn't get executed?
• Branch coverage. Which sections of loops or if-else tests didn't get run through?
• Condition coverage. Which if-tests and loop-conditions didn't check both the true and false sections?

It's hard to get to 100% coverage on any of the metrics, let alone all of them. Your goal should probably to have these metrics increasing over time, rather than seeking perfection as the goal. Note that you can't get to 100% coverage easily even with function coverage because much of the code will be exception handling. There are special ways to test exception handling, such as by intentionally injecting failures (e.g. memory allocation failure), but they add to the testing workload.

GPROF Test Coverage Script

There are many software development tools available to do testing coverage. But here's an easy way to do function-level test coverage using GCC and GPROF on Linux. GPROF is a performance profiling tool that's intended to help you identify CPU usage costs across your code, but it can also be repurposed to help with test coverage.

The method is basically:

• Re-compile your program with g++ using the “-pg” options (debug mode)
• Run your program in test mode (i.e. runs the unit test harness)
• After execution, there's a binary file “gmon.out” in the current directory.
• Run gprof to process “gmon.out” and generate a profiling report (in human-readable text format)
• Use a script to extract the “uncalled functions” section in the report (e.g. use “awk”)
• Then you have a list of functions not “covered” by unit tests.

Personally, I build all of these steps into a Makefile, so I can run my own command like “make coverage”, but you can also script it separately.

Assertions

Of all the self-testing code techniques, my favorite one is definitely assertions. They're just so easy to add!

The use of assertions in AI programs is not much different to any other C++ coding task. Assertions can be a very valuable part of improving the quality of your work over the long term. I find them especially useful in getting rid of obvious glitches when I'm writing new code, but then I usually leave them in there.

The standard C++ library has had an “assert” macro since back when it was called C. The assert macro is one convenient method of performing simple tests, and larger code fragments can be used for more complicated tests. The basic usage is illustrated to check the inputs of a vector function:

    #include <assert.h>
    ...
    float vector_sum_assert_example1(float v[], int n)
    {
        assert(v != NULL);  // Easy!
        float sum = 0;
        for (int i = 0; i < n; i++) {
            sum += v[i];
        }
        return sum;
    }

static_assert

Runtime assertions have been a staple of C++ code reliability since the beginning of time. However, there's often been a disagreement over whether or not to leave the assertions in production code, because they inherently slow things down.

The modern answer to this conundrum is the C++ “static_assert” directive. This is like a runtime assertion, but it is fully evaluated at compile-time, so it's super-fast. Failure of the assertion triggers a compile-time error, preventing execution, and the code completely disappears at run-time.

Unfortunately, there really aren't that many things you can assert at compile-time. Most computations are dynamic and stored in variables at runtime. However, the static_assert statement can be useful for things like blocking inappropriate use of template instantiation code, or for portability checking such as:

    static_assert(sizeof(float) == 4, "float is not 32 bits");

This statement is an elegant and language-standardized method to prevent compilation on a platform where a “float” data type is 64-bits, alerting you to a portability problem.

BYO assertion macros

An important point about the default “assert” macro is that when the condition fails, the default C++ library crashes your program by calling the standard “abort” function, which triggers a fatal exception on Windows or a core dump on Linux. That is fine for debugging, but it isn't what you want for production code, so most professional C++ programmers declare their own assertion macros instead.

For example, here's my own “yassert” macro in a “yasssert.h” header file:

    #define yassert(cond) ( (cond) || aussie_yassert_fail(#cond, __FILE__, __LINE__) )

This tricky macro uses the short-circuiting of the “||” operator, which has a meaning like “or-else”. So, think of it this way: the condition is true, or else we call the failure function. The effect is similar to an if-else statement, but an expression is cleaner in a macro.

The __FILE__ and __LINE__ preprocessor macros expand to the current filename and line number. The filename is a string constant, whereas the line number is an integer constant.

Function names: Note that you can add “__func__” to also report the current function name if you wish. There's also an older non-standard __FUNCTION__ version of the macro. Note that the need for all these macros goes away once there is widespread C++ support for std::stacktrace, as standardized in C++23, in which case a failing assertion could simply report its own call stack in an error message.

When Assertions Fail. This yassert macro relies on a function that is called only when an assertion has failed. And the function has to have a dummy return type of “bool” so that it can be used as an operand of the || operator, whereas a “void” return type would give a compilation error. Hence, the declaration is:

    bool aussie_yassert_fail(char* str, char* fname, int ln);  // Assertion failed

And here's the definition of the function:

    bool aussie_yassert_fail(char* str, char* fname, int ln)  
    {
        // Assertion failure has occurred...
        g_aussie_assert_failure_count++;
        fprintf(stderr, "AUSSIE ASSERTION FAILURE: %s, %s:%d\n", str, fname, ln);
        return false;  // Always fails
    }

This assertion failure function must always return “false” so that the assertion macro can be used in an if-statement condition.

Assertion Failure Extra Message

The typical assertion macro will report a stringized version of the condition argument, plus the source code filename, line number, and function name. This can be a little cryptic, so a more human-friendly extra message is often added. The longstanding hack to do this has been:

    yassert(fp != NULL && "File open failed");   // Works

The trick is that a string constant has a non-null address, so && on a string constant is like doing “and true” (and is hopefully optimized out). This gives the extra message in the assertion failure because the string constant is stringized into the condition (although you'll also see the “&&” and the double quotes, too). Note that an attempt to be tricky with comma operator fails:

    yassert(fp != NULL, "File open failed");   // Bug

There are two problems. Firstly, it doesn't compile because it's not the comma operator, but two arguments to the yassert macro. Even if this worked, or if we wrapped it in double-parentheses, there's a runtime problem: this assertion condition will never fail. The result of the comma operator is the string literal address, which is never false.

Optional Assertion Failure Extra Message: The above hacks motivate us to see if we could allow an optional second parameter to assertions. We need two usages, similar to how “static_assert” currently works in C++:

    yassert(fp != NULL);
    yassert(fp != NULL, "File open failed");

Obviously, we can do this if “yassert” was a function, using basic C++ function default arguments or function overloading. If you have faith in your C++ compiler, just declare the functions “inline” and go get lunch. But if we don't want to call a function just to check a condition, we can also use C++ variadic macros.

Variadic Macro Assertions

C++ allows #define preprocessor macros to have variable arguments using the “...” and “__VA_ARG__” special tokens. Our yassert macro changes to:

    #define yassert(cond, ...) \
       ( (cond) || \
          aussie_yassert_fail(#cond, __FILE__, __LINE__, __VA__ARG__) )

And we change our “aussie_yassert_fail” to have an extra optional “message” parameter.

    bool aussie_yassert_fail(char* str, char* fname, int ln, char *mesg=0);

This all works fine if the yassert macro has 2 arguments (condition and extra message) but we get a bizarre compilation error if we omit the extra message (i.e. just a basic assertion with a condition). The problem is that __VA_ARG__ expands to nothing (because there's no optional extra message argument), and the replacement text then has an extra “,” just hanging there at the end of the argument list, causing a syntax error.

Fortunately, the deities who define C++ standards noticed this problem and added a solution in C++17. There's a dare-I-say “hackish” way to fix it with the __VA__OPT__ special token. This is a special token whose only purpose is to disappear along with its arguments if there's zero arguments to __VA_ARG__ (i.e. it takes the ball and goes home if there's no-one else to play with). Hence, we can hide the comma from the syntax parser by putting it inside __VA_OPT__ parentheses. The final version becomes:

    #define yassert(cond, ...) \
       ( (cond) || \
          aussie_yassert_fail(#cond, __FILE__, __LINE__ \
              __VA_OPT__(,) __VA__ARG__) )

Note that the comma after __LINE__ is now inside of a __VA_OPT__ special macro. Actually, that's not the final, final version. We really should add “__func__” in there, too, to report the function name. Heck, why not add __DATE__ and __TIME__ while we're at it? Why isn't there a standard __DEVELOPER__ macro that adds my name?

I really need someone from the C++ committee to wash my mouth out with soap. I mean, __VA_OPT__ is not a hack; it's an internationally standardized feature. Sorry, my mistake!

Assertless Production Code

Not everyone likes assertions, and coincidentally some people wear sweaters with reindeer on them. If you want to compile out all of the assertions from the production code, you can use this:

   #define yassert(cond)  // nothing

But this is not perfect, and has an insidious bug that occurs rarely (if you forget the semicolon). A more professional version is to use “0” and this works by itself, but even better is a “0” that has been typecast to type “void” so it cannot be accidentally used in any expression:

    #define yassert(cond) ( (void)0 )

The method to remove calls to the yassert variadic macro version uses the “...” token:

    #define yassert(cond, ...) ( (void)0 )

Personally, I don't recommend doing this at all, as I think that assertions should be left in the production code for improved supportability. I mean, come on, recycle and reuse, remember? Far too many perfectly good assertions get sent to landfill every year.

Assertion Return Value Usage

Some programmers like to use an assertion style that tests the return code of their assert macro:

    if (assert(ptr != NULL)) {  // Risky
        // Normal code
        ptr->count++;
    }
    else {
        // Assertion failed
    }

This assertion style can be used if you like it, but I don't particularly recommend it, because it has a few risks:

1. The hidden assert failure function must return “false” so that “if” test fails when the assertion fails.

2. Embedding assertions deeply into the main code expressions increases the temptation to use side effects like “++” in the condition, which can quietly disappear if you ever remove the assertions from a production build:

    if (assert(++i >= 0))  // Risky

3. The usual assertion removal method of “((void)0)” will fail with compilation errors in an if statement. Also using a dummy replacement value of “0” is incorrect, and even “1” is not a great option, since the “if(assert(ptr!=NULL))” test becomes the unsafe “if(1)”. A safer removal method is a macro:

    #define assert(cond) (cond)

Or you can use an inline function:

    inline void assert(bool cond) { } // Empty

This avoids crashes, but may still leave debug code running (i.e. a slug, not a bug). It relies on the optimizer to remove any assertions that are not inside an “if” condition, which just leave a null-effect condition sitting there. Note also that this removal method with “(cond)” is also safer because keeping the condition also retains any side-effects in that condition (i.e. the optimizer won't remove those!).

Generalized Assertions

Once you've used assertions for a while, you start to hate them a little bit. They start to fail a lot, especially during initial module development and unit testing of new code. And that's the first time they get annoying, because the assertion failure reports don't actually give you enough information to help debug the problem. However, you can set a breakpoint on the assertion failure code, so that's usually good enough.

And the second time that assertions are annoying is when you ship the product. That's when you see assertion failures in the logs as an annoying reminder of your own imperfections. Again, there's often not enough information to reproduce the bug.

So, for your own sanity, and for improved supportability, consider extending your own assertion library into a kind of simplified unit-testing library. The extensions you should consider:

• Add std::stacktrace capabilities if you can, or use Boost Stacktrace or GCC backtrace as a backup. Printing the whole stacktrace on an assertion failure is a win.
• Add extra assertion messages as a second argument.
• Add __func__ to show the function name, if you haven't already.

And you can also generalize assertions to cover some other common code failings.

• Unreachable code assertion
• “Null pointer” assertion
• Integer value assertions
• Floating-point value assertions
• Range value assertions

Creating specialized assertion macros for these special cases also means the error messages become more specific.

Unreachable code assertion

This is an assertion failure that triggers when code that should be unreachable actually got executed somehow. The simple way that programmers have done this in the past is:

   yassert(0);  // unreachable

And you can finesse that a little with just a better name:

    #define yassert_not_reached()   ( yassert(false) )
    ...
    yassert_not_reached(); // unreachable

Here's a nicer version with a better error message:

    #define yassert_not_reached() \
        ( aussie_yassert_fail("Unreachable code was reached", __FILE__, __LINE__) )

Once-only execution assertion

Want to ensure that code is never executed twice? A function that should only ever be called once? Here's an idea for an assertion that triggers on the second execution of a code pathway, by using its own hidden “static” call counter local variable:

    #define yassert_once()  do { \
            static int s_count = 0; \
                ++s_count; \
                if (s_count > 1) { \
                aussie_yassert_fail("Code executed twice", \
                                __FILE__, __LINE__); \
                } \
       } while(0)

Restricting any block of code to once-only execution is as simple as adding a statement like this:

    yassert_once();   // Not twice!

This can be added at the start of a function, or inside any if statement or else clause, or at the top of a loop body (although why is it coded as a loop if you only want it executed once?). Note that this macro won't detect the case where the code is never executed. Also note that you could customize this macro to return an error code, or throw a different type of exception, or other exception handling method when it detects double-executed code.

Function Call Counting

The idea of once-only code assertions can be generalized to a count. For example, if you want to ensure a function isn't called too many times, use this code:

    yassert_N_times(1000);

Here's the macro, similar to yassert_once, but with a parameter:

    #define yassert_N_times(ntimes)  do { \
            static int s_count = 0; \
                ++s_count; \
                if (s_count > (ntimes)) { \
                aussie_yassert_fail( \
               "Code executed more than " #ntimes " times", \
                                __FILE__, __LINE__); \
                } \
       } while(0)

This checks for too many invocations of the code block. Checking for “too few” is a little trickier, and would need a static smart counter object with a destructor.

Detecting Spinning Loops

Note that the above call-counting macro doesn't work for checking that a loop isn't spinning. It might seem that we can use the above macro at the top of the loop body to avoid a loop iterating more than 1,000 times. But it doesn't work, because it will count multiple times that the loop is entered, not just a single time. If we want to track a loop call count, the counter should not be a “static” variable, and it's more difficult to do in a macro. The simplest method is to hand-code the test:

    int loopcount = 0;
    while (...) {
        if (++loopcount > 1000) {  // Spinning?
                // Warn...
        }
    }

Generalized Variable-Value Assertions

Various generalized assertion macros can not only check values of variables, but also print out the value when the assertion fails. The basic method doesn't print out the variable's value:

    yassert(n == 10);

A better way is:

    yassertieq(n, 10);  // n == 10

The assertion macro looks like:

   #define yassertieq(x,y) \
        (( (x) == (y)) || \
         aussie_yassert_fail_int(#x "==" #y, \
                 (x), "==", (y), \
                __FILE__, __LINE__))

The assertion failure function has extra parameters for the variables and operator string:

    bool aussie_yassert_fail_int(char* str, int x, char *opstr, int y, char* fname, int ln)  
    {
        // Assert failure has occurred...
        g_aussie_assert_failure_count++;
        fprintf(stderr, "AUSSIE INT ASSERT FAILURE: %s, %d %s %d, %s:%d\n", str, x, opstr, y, fname, ln);
        return false;  // Always fails
    }

If you don't mind lots of assertion macros with similar names, then you can define named versions for each operator, such as:

• yassertneq — !=
• yassertgtr — >
• yassertgeq — >=
• yassertlss — <
• yassertleq — <=

If you don't mind ugly syntax, you can generalize this to specify an operator as a parameter:

   yassertiop(n, ==, 10);

The macro with an “op” parameter is:

   #define yassertiop(x, op, y) \
        (( (x) op (y)) || \
         aussie_yassert_fail_int(#x #op #y, \
                 (x), #op, (y), \
                __FILE__, __LINE__))

And finally, you have to duplicate all of this to change from int to float type variables. For example, there's macros named “yassertfeq”, “yassertfop”, and a failure function named “aussie_yassert_fail_float”. There's probably a fancy way to avoid this using C++ templates and compile-time type traits, but only if you're smarter than me.

Assertions for Function Parameter Validation

Assertions and toleration of exceptions have some tricky overlaps. Consider the modified version of vector summation with my own “yassert” macro instead:

    float vector_sum_assert_example2(float v[], int n)
    {
        yassert(v != NULL);
        float sum = 0;
        for (int i = 0; i < n; i++) {
                sum += v[i];
        }
        return sum;
    }

This still isn't great in production because it crashes if the assertion fails. Once control flow returns from the failing “yassert” macro, then the rest of the code has “v==NULL” and it will immediately crash with a null-pointer dereference.

Hence, the above code works fine only if your “yassert” assertion macro throws an exception. This requires that you have a robust exception handling mechanism in place above it, which is a significant amount of work.

The alternative is to both assert and handle the error in the same place, which makes for a complex block of code:

   yassert(v != NULL); 
   if (v == NULL) {
            return 0.0;  // Tolerate
    }

Slightly more micro-efficient is to only test once:

    if (v == NULL) {
            yassert(v != NULL); // Always triggers
            return 0.0;  // Tolerate
    }

This is a lot of code that can get repeated all over the place. Various copy-paste coding errors are inevitable.

Assert Parameter and Return

An improved solution is an assertion macro that captures the logic “check parameter and return zero” in one place. Such a macro first tests a function parameter and if it fails, the macro will not only emit an assertion failure message, but will also tolerate the error by returning a specified default value from the function.

Here's a generic version for any condition:

    #define yassert_and_return(cond,retval) \
        if (cond) {} else { \
            aussie_yassert_fail(#cond " == NULL", __FILE__, __LINE__); \
            return (retval); \
        }

The usage of this function is:

    float aussie_vector_something(float v[], int n) 
    {
            yassert_and_return(v != NULL, 0.0f);
        ...
    }

The above version works for any condition. Here's another version specifically for testing an incoming function parameter for a NULL value:

    #define yassert_param_tolerate_null(var,retval) \
        if ((var) != NULL) {} else { \
            aussie_yassert_fail(#var " == NULL", __FILE__, __LINE__); \
            return (retval); \
        }

The usage of this function is:

    yassert_param_tolerate_null(v, 0.0f);

If you want to be picky, a slightly better version wraps the “if-else” logic inside a “do-while(0)” trick. This is a well-known trick to make a macro act more function-like in all statement structures.

    #define yassert_param_tolerate_null2(var,retval) \
        do { if ((var) != NULL) {} else { \
            aussie_yassert_fail(#var " == NULL", __FILE__, __LINE__); \
            return (retval); \
        }} while(0)

The idea of this macro is to avoid lots of parameter-checking boilerplate that will be laborious and error-prone. But it's also an odd style to hide a return statement inside a function-like preprocessor macro, so this is not a method that will suit everyone.

Next-Level Assertion Extensions

Here are some final thoughts on how to further improve your assertions:

• Change any often-triggered assertions into proper error messages with fault tolerance. Users don't like seeing assertion messages. They're kinda like gibberish to ordinary mortals.
• Add extra context information in the assertion message (i.e. add an extra information string). This is much easier to read than a stringized expression, filename with line number, or multi-line stack trace.
• Add unique codes to assertion messages for increased supportability. Although, maybe not, because any assertion that's triggering often enough to need a code, probably shouldn't remain an assertion!
• Inlined assertion function? Why use macros? Maybe these assertions should instead be an inline function in modern C++? All I can say is that old habits die hard, and I still don't trust the optimizer to actually optimize much.

Laziness and Assertion Macros

No, I'm not talking about you and I know that you're not a lazy C++ programmer. I'm talking about your friends at that old company where you used to work.

The downside of assertions is mainly that they make you lazy as a programmer because they're so easy to add. But sometimes no matter how good they seem, you have to throw an assertion into the fires of Mordor. The pitfalls include:

• Don't use assertions instead of user input validation.
• Don't use assertions to check program configurations.
• Don't use assertions as unit tests (it works, but bypasses the test harness statistics).
• Don't use assertions to check if a file opened.

You need to step up and actually code the checks of input and configurations as part of proper exception handling. For example, it has to check the values, and then emit a useful error code if they've failed, and ideally it's got a unique error code as part of the message, so that users can give a code to support if they need. You really don't want users to see the dirty laundry of an assertion message with its source file, function name, and line number.

Debug Wrapper Functions

Assertions are wonderful but also laborious because you have to add them everywhere. However, one way to avoid assertions to check arguments at every call to a library function is to use a debug wrapper function instead.

It can be helpful during debugging to wrap some standard library function calls with your own versions, so as to add additional parameter validation and self-checking code. These self-tests can be as simple as parameter null tests or as complex as detecting memory stomp overwrites with your own custom code.

Some of the functions which you might consider wrapping include:

• malloc, calloc, strdup
• memset
• memcpy
• strcpy, etc.

Note that you can wrap the C++ “new” and “delete” operators at the linker level by defining your own versions, but not as macro intercepts. You can also intercept the “new[]” and “delete[]” array allocation versions.

There are different approaches to consider when wrapping system calls, which we examine using memset as an example:

• Leave “memset” calls in your code (auto-intercepts)
• Use “memset_wrapper” in your code instead (manual intercepts)

Macro auto-intercepts: You might want to leave your code unchanged using memset. To leave “memset” in your code, but have it automatically call “memset_wrapper” you can use a macro intercept in a header file.

    #undef memset // ensure no prior definition
    #define memset memset_wrapper  // Intercept

Note that you can also use preprocessor macros to add context information to the debug wrapper functions. For example, you could add extra parameters to “memset_wrapper” such as:

    #define memset(x,y,z)  memset_wrapper((x),(y),(z),__FILE__,__LINE__,__func__)

Note that in the above version, the macro parameters must be parenthesized even between commas, because there's a C++ comma operator that could occur in a passed-in expression. Also note that these context macros (e.g. __FILE__) aren't necessary if you have a C++ stacktrace library, such as std::stacktrace, on your platform.

Variadic preprocessor macros: Note also that there is varargs support in C++ #define macros. If you want to track variable-argument functions like sprintf, printf, or fprintf, or other C++ overloaded functions, you can use “...” and “__VA_ARGS__” in preprocessor macros as follows.

    #define sprintf(fmt,...)  sprintf_wrapper((fmt),__FILE__,__LINE__,__func__, __VA_ARGS__ )

Manual Wrapping: Alternatively, you might want to individually change the calls to memset to call memset_wrapper without hiding it behind a macro. If you'd rather have to control whether or not the wrapper is called, then you can use both in the program, wrapped or non-wrapped. Or if you want them all changed, but want the intercept to be less hidden (e.g. later during code maintenance), then you might consider adding a helpful reminder instead:

    #undef memset
    #define memset dont_use_memset_please

This trick will give you a compilation error at every call to memset that hasn't been changed to memset_wrapper.

Example: memset Wrapper Self-Checks

Here's an example of what you can do in a wrapper function called “memset_wrapper” from one of the Aussie AI projects:

    void *memset_wrapper(void *dest, int val, int sz)  // Wrap memset
    {
        if (dest == NULL) {
                yassert2(dest != NULL, "memset null dest");
                return NULL;
        }
        if (sz < 0) {
                // Why we have "int sz" not "size_t sz" above
                yassert2(sz >= 0, "memset size negative");
                return dest;  // fail
        }
        if (sz == 0) {
                yassert2(sz != 0, "memset zero size (reorder params?)");
                return dest;
        }
        if (sz <= sizeof(void*)) {
                // Suspiciously small size
                yassert2(sz > sizeof(void*), "memset with sizeof array parameter?");
                // Allow it, keep going
        }
        if (val >= 256) {
                yassert2(val < 256, "memset value not char");
                return dest; // fail
        }
        void* sret = ::memset(dest, val, sz);  // Call real one!
        return sret;
    }

It's a judgement call whether or not to leave the debug wrappers in place, in the vein of speed versus safety. Do you prefer sprinting to make your flight, or arriving two hours early? Here's one way to remove the wrapper functions completely with the preprocessor if you've been manually changing them to the wrapper names:

    #if YDEBUG
        // Debug mode, leave wrappers..
    #else // Production (remove them all)
        #define memset_wrapper memset
        //... others
    #endif

Compile-time self-testing macro wrappers

Here's an idea for combining the runtime debug wrapper function idea with some additional compile-time tests using static_assert.

    #define memset_wrapper(addr,ch,n) ( \
        static_assert(n != 0), \
        static_assert(ch == 0), \
        memset_wrapper((addr),(ch),(n),__FILE__,__LINE__,__func__))

The idea is interesting, but it doesn't really work, because not all calls to the memset wrapper will have constant arguments for the character or the number of bytes, so the static_assert commands will fail in that case. You could use standard assertions, but this adds runtime cost. Note that it's a self-referential macro, but that C++ guarantees it only gets expanded once (i.e., there's no infinite recursion of preprocessor macros).

Generalized Self-Testing Debug Wrappers

The technique of debug wrappers can be extended to offer a variety of self-testing and debug capabilities. The types of messages that can be emitted by debug wrappers include:

• Input parameter validation failures (e.g. non-null)
• Failure returns (e.g. allocation failures)
• Common error usages
• Informational tracing messages
• Statistical tracking (e.g. call counts)

Personally, I've built some quite extensive debug wrapping layers over the years. It always surprises me that this can be beneficial, because it would be easier if it were done fully by the standard libraries of compiler vendors. The level of debugging checks has been increasing significantly (e.g. in GCC), but I still find value in adding my own wrappers.

There are several major areas where you can really self-check for a lot of problems with runtime debug wrappers:

• File operations
• Memory allocation
• String operations

Self-Testing Code Block

Sometimes an assertion, unit test, or debug tracing printout is too small to check everything. Then you have to write a bigger chunk of self-testing code. The traditional way to do this in code is to wrap it in a preprocessor macro:

    #if YDEBUG
       ... // block of test code
    #endif

Another reason to use a different type of self-testing code than assertions is that you've probably decided to leave the simpler assertions in production code. A simple test like this is probably fine for production:

    yassert(ptr != NULL);  // Fast

But a bigger amount of arithmetic may be something that's not for production:

    yassert(aussie_vector_sum(v, n) == 0.0);  // Slow

So, you probably want to have macros and preprocessor settings for both production and debug-only assertions and self-testing code blocks. The simple way looks like this:

    #if YDEBUG
    yassert(aussie_vector_sum(v, n) == 0.0);
    #endif

Or you could have your own debug-only version of assertions that are skipped for production mode:

    yassert_debug(aussie_vector_sum(v, n) == 0.0);

The definition of “yassert_debug” then looks like this in the header file:

    #if YDEBUG
    #define yassert_debug(cond) yassert(cond)  // Debug mode
    #else 
    #define yassert_debug(cond)  // nothing in production
    #endif

This makes the “yassert_debug” macro a normal assertion in debug mode, but the whole coded expression disappears to nothing in production build mode. The above example assumes a separate set of build flags for a production build.

Self-test Code Block Macro

An alternative formulation of a macro for installing self-testing code using a block-style, rather than a function-like macro, is as follows:

    SELFTEST {
        // block of debug or self-test statements
    }

The definition of the SELFTEST macro looks like:

    #if YDEBUG
    #define SELFTEST // nothing (enables!)
    #else
    #define SELFTEST if(1) {} else // disabled
    #endif

This method relies on the C++ optimizer to fix the non-debug version, by noticing that “if(1)” invalidates the else clause, so as to remove the block of unreachable self-testing code that's not ever executed.

Note also that SELFTEST is not function-like, so we don't have the “forgotten semicolon” risk when removing SELFTEST as “nothing”. In fact, the nothing version is actually when SELFTEST code is enabled, which is the opposite situation of that earlier problem. Furthermore, we cannot use the “do-while(0)” trick in this different syntax formulation.

Self-Test Block Macro with Debug Flags

The compile-time on/off decision about self-testing code is not the most flexible method. The block version of SELFTEST can also have levels or debug flag areas. One natural extension is to implement a “flags” idiom for areas, to allow configuration of what areas of self-testing code are executed for a particular run (e.g. a decoding algorithm flag, a normalization flag, a MatMul flag, etc.). One Boolean flag is set for each debugging area, which controls whether or not the self-testing code in that module is enabled or not.

A macro definition of SELFTEST(flagarea) can be hooked into the run-time configuration library for debugging output. In this way, it has both a compile-out setting (YDEBUG==0) and dynamic runtime “areas” for self-testing code. Here's the definition of the self-testing code areas:

    enum selftest_areas {
        SELFTEST_NORMALIZATION,
        SELFTEST_MATMUL,
        SELFTEST_SOFTMAX,
        // ... more
    };

A use of the SELFTEST method with areas looks like:

    YSELFTEST(SELFTEST_NORMALIZATION) {
        // ... self-test code
    }

The SELFTEST macro definition with area flags looks like:

    
    extern bool g_aussie_debug_enabled;  // Global override
    extern bool YDEBUG_FLAGS[100];     // Area flags

    #if YDEBUG
        #define SELFTEST(flagarea) \
            if(g_aussie_debug_enabled == 0 || YDEBUG_FLAGS[flagarea] == 0) \
            { /* do nothing */ } else
    #else
    #define SELFTEST if(1) {} else // disabled completely
    #endif

This uses a “debug flags” array idea as for the debugging output commands, rather than a single “level” of debugging. Naturally, a better implementation would allow separation of the areas for debug trace output and self-testing code, with two different sets of levels/flags, but this is left as an extension for the reader.

 

• Next: Chapter 42. Debugging • Up: Table of Contents • Buy: Generative AI in C++: Coding Transformers and LLMs

The new AI programming book by Aussie AI co-founders: AI coding in C++ Transformer engine speedups LLM models Phone and desktop AI Code examples Research citations Get your copy from Amazon: Generative AI in C++