Aussie AI

Tool Usage in AI Architectures

  • Last Updated 1 April, 2025
  • by David Spuler, Ph.D.

This section is about tools being used by LLMs, rather than tools being used by humans to create LLMs. There's a big difference!

Why LLM Tools?

LLMs require tools to do more advanced things, just like humans. For example, if someone asks you the time, you look at your watch (or your phone). If you ask an LLM "What is the time?" there is nothing in its training data set that could possibly answer this correctly. The only way is to use a clock that's integrated iinto the LLM, and executed by the AI Engine as part of answering your query.

Types of LLM Tools

There are several types of tools that can be integrated:

  • Data sources (e.g. real estate listings)
  • Dynamic calculations
  • Action tools in agent architectures (e.g. an API to send an email).

Types of dynamic calculation tools include:

  • Clocks
  • Calculators (arithmetic)
  • Converters (e.g. pounds to kilograms)
  • Calendars (date or day calculations)

And many more...

How are Dynamic Tools Integrated?

Like humans, an AI needs to learn to look at its watch if someone asks the time. Specific training data sets are required that tell the AI what tool to use, and when.

The AI engine has to recognize in the LLM output that a tool must be executed. There are a variety of ways to do this:

  • Tool-specific tokens — i.e., the LLM can emit a "trigger" token to run a tool. Note that PEFT could be used here to fine-tune new tool capabilities, by only adding a few new tool-triggering tokens to the vocabulary.)
  • Placeholder patterns — i.e., output something like an "--insert current time here--" special pattern is another way, and the engine then looks for these patterns, which avoids adding tool tokens to the vocabulary, but is inefficient in that there are multiple text tokens in the output).
  • Code generation — there are various AI models that will generate code, such as in Python, that can be executed to generate the answer. This is a general solution, because Python can call various submodules and can thereby generate many tools.
  • Multi-level planning — the AI first generates a plan of how to answer the query, including what tools to use, and then runs any tools, and then does another inference query to collate it into a final answer.

Research on AI Tool Integrations

Tool integration papers:

Tool-Augmented Language Models (TALM)

Reserch papers on TALM:

LLM Screen Access

LLM Computer Usage

More AI Research

Read more about: