Aussie AI

Tool Usage in AI Architectures

  • Last Updated 30 August, 2025
  • by David Spuler, Ph.D.

What are Tools?

This section is about tools being used by LLMs, rather than tools being used by humans to create LLMs. There's a big difference!

Why LLM Tools?

LLMs require tools to do more advanced things, just like humans. For example, if someone asks you the time, you look at your watch (or your phone). If you ask an LLM "What is the time?" there is nothing in its training data set that could possibly answer this correctly. The only way is to use a clock that's integrated iinto the LLM, and executed by the AI Engine as part of answering your query.

Types of LLM Tools

There are several types of tools that can be integrated:

  • Data sources (e.g. real estate listings)
  • Dynamic calculations
  • Action tools in agent architectures (e.g. an API to send an email).

Types of dynamic calculation tools include:

  • Clocks
  • Calculators (arithmetic)
  • Converters (e.g. pounds to kilograms)
  • Calendars (date or day calculations)

And many more...

How are Dynamic Tools Integrated?

Like humans, an AI needs to learn to look at its watch if someone asks the time. Specific training data sets are required that tell the AI what tool to use, and when.

The AI engine has to recognize in the LLM output that a tool must be executed. There are a variety of ways to do this:

  • Tool-specific tokens — i.e., the LLM can emit a "trigger" token to run a tool. Note that PEFT could be used here to fine-tune new tool capabilities, by only adding a few new tool-triggering tokens to the vocabulary.)
  • Placeholder patterns — i.e., output something like an "--insert current time here--" special pattern is another way, and the engine then looks for these patterns, which avoids adding tool tokens to the vocabulary, but is inefficient in that there are multiple text tokens in the output).
  • Code generation — there are various AI models that will generate code, such as in Python, that can be executed to generate the answer. This is a general solution, because Python can call various submodules and can thereby generate many tools.
  • Multi-level planning — the AI first generates a plan of how to answer the query, including what tools to use, and then runs any tools, and then does another inference query to collate it into a final answer.

Research on AI Tool Integrations

Tool integration papers:

Tool-Augmented Language Models (TALM)

Reserch papers on TALM:

LLM Screen Access

LLM Computer Usage

AI Books from Aussie AI



The Sweetest Lesson: Your Brain Versus AI The Sweetest Lesson: Your Brain Versus AI: new book on AI intelligence theory:
  • Your brain is 50 times bigger than the best AI engines.
  • Truly intelligent AI will require more compute!
  • Another case of the bitter lesson?
  • Maybe it's the opposite of that: the sweetest lesson.

Get your copy from Amazon: The Sweetest Lesson



RAG Optimization RAG Optimization: Accurate and Efficient LLM Applications: new book on RAG architectures:
  • Smarter RAG
  • Faster RAG
  • Cheaper RAG
  • Agentic RAG
  • RAG reasoning

Get your copy from Amazon: RAG Optimization



Generative AI in C++ Generative AI Applications book:
  • Deciding on your AI project
  • Planning for success and safety
  • Designs and LLM architectures
  • Expediting development
  • Implementation and deployment

Get your copy from Amazon: Generative AI Applications



Generative AI in C++ Generative AI programming book:
  • Generative AI coding in C++
  • Transformer engine speedups
  • LLM models
  • Phone and desktop AI
  • Code examples
  • Research citations

Get your copy from Amazon: Generative AI in C++



CUDA C++ Optimization CUDA C++ Optimization book:
  • Faster CUDA C++ kernels
  • Optimization tools & techniques
  • Compute optimization
  • Memory optimization

Get your copy from Amazon: CUDA C++ Optimization



CUDA C++ Optimization CUDA C++ Debugging book:
  • Debugging CUDA C++ kernels
  • Tools & techniques
  • Self-testing & reliability
  • Common GPU kernel bugs

Get your copy from Amazon: CUDA C++ Debugging

More AI Research

Read more about: