AI Quality

AI Quality

A quality AI would predict my wishes and wash my dishes. While we wait for that to happen, the desirable qualities of an AI engine include:

• Accuracy
• Sensitivity
• Empathy
• Predictability
• Alignment

Much as I like code, a lot of the “smartness” of the LLM starts with the training data. Garbage in, garbage out! Finding enough quality data that is ratified to use for model fine-tuning or a RAG database is one of the hurdles that delays business deployment of AI applications. Another problem with data quality is that new models are starting to be trained using the outputs of other models, and this “synthetic data” is leading to degradation in these downstream models.

At the other end of the quality spectrum, we've seen the headlines about the various types of malfeasance that a low-quality AI engine could perform, such as:

• Bias
• Toxicity
• Inappropriateness
• Hallucinations (i.e., fake answers)
• Wrong answers (e.g., from inaccurate training data)
• Dangerous answers (e.g., mushroom collecting techniques)
• Going “rogue”

And some of the technical limitations and problems that have been seen in various AI applications include:

• Lack of common sense
• Difficulty with mathematical reasoning
• Explainability/attribution difficulty
• Overconfidence
• Model drift (declining accuracy over time)
• Catastrophic forgetting (esp. in long texts)
• Lack of a “world view”
• Training cut-off dates
• Difficulty with time-related queries (e.g., “What is significant about today?”)
• Problems handling tabular input data (e.g., spreadsheets)
• Banal writing that lacks emotion and “heart” (it's a robot!)

If you ask me, almost the exact same list would apply to any human toddler, although at least ChatGPT doesn't pour sand in your ear or explain enthusiastically that “Dad likes wine” during show-and-tell. Personally, I think it's still a long road to Artificial General Intelligence (AGI).

Unfortunately, every single bullet point in the above paragraphs is a whole research area in itself. Everyone's trying to find methods to improve the smartness and reduce the dumbness. There's another whole book in that list, so I'm going to stick to the code.

The remainder of this chapter is primarily around the quality issues that you have as an engineer of a C++ Transformer, such as ensuring it never crashes and responds fast enough. Many of the issues are generic to any kind of C++ application, but there are also some AI-specific aspects to software quality.

• Next: • Up: Table of Contents • Buy: Generative AI in C++: Coding Transformers and LLMs

The new AI programming book by Aussie AI co-founders: AI coding in C++ Transformer engine speedups LLM models Phone and desktop AI Code examples Research citations Get your copy from Amazon: Generative AI in C++