Memory Reduction in C++

Memory Reduction in C++

This chapter discusses the general techniques for reducing the memory requirements of a C++ program. The more general AI memory management issues of reducing the size of an AI model (e.g. model compression, quantization, pruning, etc.) or improving the memory access bottleneck in AI models (e.g. pipelining and marshaling data for a GPU) are discussed in a separate chapter.

These techniques herein aim to reduce memory usage of a program so that:

(a) your C++ does not waste too much time on memory management activity, such as allocating too much memory, and

(b) your C++ code can execute on a low-memory platform, such as an IoT embedded device.

In these days of cheap gigabytes of memory in every PC, memory reduction techniques are perhaps not as important as those for increasing speed. However, there are certainly situations when reducing space requirements is far more important than increasing the speed of a program. This section discusses a number of general techniques for reducing C++ memory requirements.

Unfortunately, reducing space requirements can also lead to loss of speed. There is often a trade-off between space efficiency and time efficiency. Every C++ program uses memory for a number of different purposes, and each of these areas needs to be attacked separately. The memory usage of the program can be divided into the following memory sections:

• Executable instructions
• Static storage
• Stack storage
• Heap storage

The executable instructions for a program are usually stored in one contiguous block of memory. Static storage refers to memory used by global and local static variables, string constants and (possibly) floating-point constants. Stack storage refers to the dynamic storage of non-static local variables. Heap storage refers to the memory that is dynamically allocated using the new/delete operators and the malloc/calloc/free standard library functions.

The memory requirements for the executable instructions are largely independent of the other memory areas, whereas the techniques for reducing the memory required for the other three areas are often similar. However, care must be taken that applying a technique to reduce data space does not increase the amount of C++ code too greatly, thus increasing the executable size.

• Next: • Up: Table of Contents • Buy: Generative AI in C++: Coding Transformers and LLMs

The new AI programming book by Aussie AI co-founders: AI coding in C++ Transformer engine speedups LLM models Phone and desktop AI Code examples Research citations Get your copy from Amazon: Generative AI in C++