Preface

Welcome to Generative AI in C++ and the wonderful world of Transformers and LLMs. I hope you enjoy the show!

Why a Book on AI and C++?

I was chatting with my pal GPT about what I should do on a rainy Sunday afternoon, when it said, “Hey! I haven't read a great book on my gizzards yet!”

So, I prompted it, “Please write me an advanced professional book on Generative AI in C++.”

Voila! We were finished in 35 milliseconds.

Well, immediately, you know that's false, right? I mean, the “35 milliseconds” part is what gave it away, because AI takes a lot longer than that to do anything, because it's slooooowwww!

And yet, AI is mostly written with oodles of C++ in its undercarriage, which is a blazingly fast programming language, as everybody who's anybody has known since the 1970s. All of the C++ AI engines should be super-fast, too.

What gives? Well, hopefully this book has all the answers for you.

Who This Book is For

If you know how to code C++, but have no idea how it's used in AI, then this book is for you. The short answer: C++ is underneath everything in AI, doing all the hard yards. More specifically, this book is for people who are intermediate to advanced C++ programmers, and:

• Dream of building your own AI Engine in C++.
• Want to understand, extend, or modify any of the open source C++ AI engines.
• Want to learn the state-of-the-art of AI algorithms with C++ code and research paper citations.
• Know C++ and want the challenge of coding AI kernels at a big AI company.
• Are a student, academic or industry researcher wanting to research novel AI algorithms, code them, and test them.
• Are already a deep AI developer, but want to polish your C++ and catch up on the latest AI research ideas.

This book is probably not for you if:

• You prefer Python to C++.
• You're happy to use TensorFlow or PyTorch without understanding what they're doing.
• You'd rather sit in meetings chatting about processes than cut code.
• You prefer going slowly to speeding really fast without a seat-belt.

How This Book is Organized

Part I: AI Projects in C++ is an introduction to various AI concepts and terminology for a C++ programmer.

Part II: Basic C++ Optimizations presents a number of chapters on basic C++ optimization techniques for normal sequential execution. A lot of this material is general C++ programming that is needed for later AI optimization techniques.

Part III: Parallel C++ Optimizations looks at more advanced optimizations of C++ that add parallelism. These are the basic techniques needed to optimize AI algorithms for CPU or GPU acceleration.

Part IV: Transformer Components in C++ introduces all of the major components that make up a Transformer like GPT. This shows simple C++ for many of these parts of the AI engine.

Part V: Optimizing Transformers in C++ looks at highly advanced optimization techniques that are used to speed up an AI engine. This includes optimizations to the model (e.g. making it smaller) and speedups to the C++ in the engine internals (e.g. hardware vectorization).

Part VI: Enterprise AI in C++ looks at enterprise software engineering issues on an AI project, such as tuning, debugging, and reliability.

Part VII: Research on AI Optimization examines the state-of-the-art in AI research papers, with a particular focus on the myriad ways to speed up an AI engine. This section is mainly for researchers and advanced developers looking to create new features for AI engines.

Value in this Book

It is my profound hope that the value to you of the ideas herein will far outweigh your investment in buying this book. If AI and C++ are your livelihood, then even one good idea is valuable, and I have strived hard to ensure that there are many on offer in these pages. Via the words on these pages, I keenly seek to extend your professional knowledge and advance your career.

Novel Features of this Book

There are a lot of books on AI, but mostly they use that snake-ish language which isn't as good as C++. The areas where I think this book makes a contribution include:

• Detailed examination of Transformer components in C++.
• Real C++ code for hard algorithms, but starting with easy examples.
• Lots of general C++ expertise applied to AI.
• Numerous ways to speed up AI Engine algorithms to run on a PC or smartphone.
• A helpful guide to AI for C++ experts.
• A full research literature review of the state-of-the-art ideas in published papers on engine optimizations.

Print versus E-book Formats

Electrons are infinite, whereas trees are not. Hence, due to space constraints, the paperback printed copy of this book is slightly shorter than the e-book (they'd call it “abridged” in the publishing trade). Some of the research paper lists are shortened, appendices may be removed, and some low-priority sections are hiding. These are available in the online bonus materials, but hopefully you'll never notice the difference, because nobody reads the Preface anyway. On the upside, this means that the e-book is both longer and cheaper.

How to Use This Book

    bool GenAICPPBook::reader(
        float iq=200.0f, 
        bool ontoilet=false)
    {
        pick_it_up(*this, std::hands::left);
        do {
            for (int i = 0; i < n; i += 8) {
                dst = __mm256_read_words(&pages[i]);
                __mm256_store_ideas(dst, &brainmem[i]);
            }
        } while (!posix_asleep() && !std::brainfull());
        wake_up_again(time(NULL) + 8*SECS_PER_HOUR);
        if (this->good_book()) 
            [[likely]] write_review();
        else 
            [[unlikely]] throw Exception::grumpy();
    }

About Aussie AI

Aussie AI is a suite of AI-based writing and editing tools, with a special focus on fiction writing. Our software already has an extensive range of reports and error checks for both fiction and non-fiction writing, from a full-length novel to a short report. Please try it out and let us know what you think: https://www.aussieai.com

Our AI Research

The primary focus of research at Aussie AI is on optimizing LLM inference algorithms (i.e. “running” the model after training or fine-tuning), and our research is toward the following aims:

• Fast on-device model inference algorithms, specifically for smartphones and AI PCs.
• Scaling inference algorithms to large volumes of requests.
• Efficient GPU inference algorithms (hardware acceleration).
• Non-GPU inference optimization algorithms (i.e. software methods).

C++ Source Code

Most of the source code examples are excerpts from the Aussie AI C++ library, in many of the C++ source code examples. Details about source code availability can be found in the Aussie AI research area: https://www.aussieai.com/research/overview.

Some code examples are abridged with various code statements removed for brevity or elucidation. For example, assertions, self-checking code or function argument validation tests have sometimes been removed.

Most code has been tested on Microsoft Visual Studio C++ on a Windows laptop with an AMD Ryzen CPU and on Linux CentOS and/or Linux Ubuntu running on Intel Xeon processors. Much of the C++ code is platform-independent and should run elsewhere, except for hardware-specific coding such as the x86 AVX intrinsics.

Disclosure: Minimal AI Authorship

Despite my being involved in the AI industry, there was almost no AI engine usage in creating this book's text or its coding examples. Some text has been analyzed and reviewed using Aussie AI's editing tools, but not even one paragraph was auto-created by any generative AI engine. All of the C++ code is also human-written, without involvement of any AI coding copilot tools.

However, AI was used in several ways. AI-assisted search tools, such as “Bing Chat with GPT-4”, were very useful in brainstorming topics and researching some of the technical issues. The main cover art image of a baby dragon was AI-generated using Google ImageFX, followed by human editing by Mike Trent, a professional cover designer.

More Generative AI in C++

A whole book on generative AI in C++ isn't enough for you? Well, alrighty then.

Updates and Additions: Additional book materials, updates and errata will be made available over time online at the Aussie AI website. Visit this URL: https://www.aussieai.com/book/overview.

Errata: Any bugs or slugs that we learn about in this work will be posted online on the Aussie AI website in the Errata section of Aussie AI research. Visit this URL to view these details: https://www.aussieai.com/book/errata

AI Research Literature Review: Ongoing updates to the AI research literature review are found in the Aussie AI Research pages, categorized by topic, starting at the entry page: https://www.aussieai.com/research/overview. If you have a correction to a citation or a paper to suggest for a category, please email research@aussieai.com.

Blog & Newsletter: Add a regular dose of Generative AI in C++ to your feed. Review the Aussie AI blog at https://www.aussieai.com/blog/ To join the book's newsletter visit https://www.aussieai.com/book/newsletter.

Future Editions: Please get in touch with any contributions or corrections as future editions of the book are planned. I welcome suggestions for improvement or information on any errors you find in the book.

Disclaimers

Although I hope the information is useful to you, neither the content nor code in this work is guaranteed for any particular purpose. Nothing herein is intended to be personal, medical, financial or legal advice. You should make your own enquiries to confirm the appropriateness to your situation of any information. Many code examples are simplistic and have been included for explanatory or educational benefit, and are therefore lacking in terms of correctness, quality, functionality, or reliability. For example, some of the examples are not good at handling the special floating-point values such as negative zero, NaN, or Inf.

Oh, and sometimes I'm being sarcastic, or making a joke, but it's hard to know when, because there's also a saying that “Truth is often said in jest!” Your AI engine certainly won't be able to help you sort out that conundrum.

Third-Party License Notices

Except where expressly noted, all content and code is written by David Spuler (or the contributors), with rights owned by David Spuler and/or Aussie AI.

Additional information, acknowledgments and legal notices in relation to this book, the C++ source code, or other Aussie AI software, can be found on the Aussie AI Legal Notices page: https://www.aussieai.com/admin/legal-notices.

Acknowledgements

This book would not have been possible without the help of others. Thank you to Michael Sharpe who lent his AI and C++ expertise to the project with industry guidance and technical reviews. Data scientist and architecture expert Cameron Gregory also provided much assistance with many contributions to various chapters on coding, architecture, and DevOps.

I would like to acknowledge the many AI researchers and open source contributors who have made the AI revolution possible. In particular, the advanced coding skills shown in the many C++ projects are acknowledged with both admiration and appreciation.

Please Leave a Review

I hope you enjoy the book! Please consider leaving a review on the website where you purchased the book or on Goodreads. Since few readers do this, each review is important to me, and I read them all personally. Five-star reviews leave me coding up a storm for days, and for one-star reviews, I go home and cuddle my robot dog.

Feedback and Contacts

Feedback from readers is welcome. Please feel free to tell us what you think of the book, the literature review, or our Aussie AI software. Contact us by email via research@aussieai.com.