Working with the LLM API

You have an API key. Now what?

Every demo makes it look easy: one line of code, the model replies, the crowd cheers. Then you open the docs and drown in roles, streaming chunks, tool schemas, token counts, and a 429 error you have never seen before. The chatbot you typed into hid all of this. The API does not.

Here is the good news: under the surface, almost every modern LLM provider speaks the same dialect. Learn the shape once and you can move between providers by swapping a base URL and a model name. This article is that shape, the handful of moves you will use in literally every AI feature you ever build.

The one mental model: it is just a function over a list

An LLM API call is a pure-ish function: you hand it the entire conversation so far, and it returns the next message. It remembers nothing between calls, you resend the history every time.

That last sentence trips up everyone. There is no session, no server-side memory of your last question. You are the database. The "conversation" is just an array you keep appending to and resending. Once that clicks, the rest is detail.

The request/response lifecycle

Before any code, hold the picture in your head. Your app builds a messages array, sends it to the API, and gets tokens back, usually streamed one chunk at a time. Sometimes the reply is plain text. Sometimes the model branches: instead of answering, it asks you to run a tool, you run it, and you loop the result back in.

1. 1

   Assemble the messages

   Start with a system message (the rules), add the running history, then append the new user message. This array is the entire input.

2. 2

   Send the request

   POST the messages, the model name, and parameters like temperature and max_tokens to the chat endpoint with your API key in the header.

3. 3

   Receive the reply

   Either wait for the full response, or stream it, the server pushes partial tokens so you can render text as it is generated.

4. 4

   Branch if needed

   If you provided tools and the model decides to use one, it returns a tool-call request instead of text. You run the tool and append its output to the messages.

5. 5

   Loop or finish

   After a tool result you call the API again with the extended history. When the model returns plain text, you append it and you are done, ready for the next user turn.

The four message roles

Every entry in the messages array has a role. There are only four you need, and each has a distinct job. Get the roles right and the model behaves; mix them up and you get confused, off-topic replies.

A basic chat request

Here is the whole thing in Python, using the OpenAI-style SDK that most providers are compatible with. The same code points at a different provider by changing base_url and model.

from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from the environment

messages = [
    {"role": "system", "content": "You are a concise assistant. Answer in one sentence."},
    {"role": "user", "content": "What is a vector database, in plain English?"},
]

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    temperature=0.2,   # lower = more deterministic
    max_tokens=200,    # cap the reply length (and your bill)
)

reply = response.choices[0].message.content
print(reply)

# Crucial: to continue the conversation, append the reply
# and the next user turn, then call create() again.
messages.append({"role": "assistant", "content": reply})

# Token accounting comes back on every response:
usage = response.usage
print(f"prompt={usage.prompt_tokens} "
      f"completion={usage.completion_tokens} "
      f"total={usage.total_tokens}")

Two things to internalize. First, the reply lives at choices[0].message.content, a nested path because the API can return multiple choices. Second, you pay per token, both for what you send (prompt_tokens) and what you get back (completion_tokens). A token is roughly 4 characters of English, or about ¾ of a word. Long histories get expensive fast, which is why trimming old turns matters.

Streaming the response

Waiting ten seconds for a full paragraph feels broken. Streaming flips a flag and the server pushes tokens as they are generated, so text appears word-by-word, the experience you know from every chat UI. You assemble the chunks yourself.

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    stream=True,   # the only change
)

full_reply = []
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:                # some chunks carry no text (role, finish)
        print(delta, end="", flush=True)
        full_reply.append(delta)

# Re-join the pieces to store the complete turn in history.
reply = "".join(full_reply)
messages.append({"role": "assistant", "content": reply})

Getting structured JSON output

Free-form prose is great for humans and terrible for code. When the next step is data["price"], you need guaranteed, parseable JSON, not "Sure! Here is the JSON: ..." wrapped in markdown. Modern APIs let you pin the output to a JSON schema, and the model is constrained to produce exactly that shape.

import json

schema = {
    "type": "object",
    "properties": {
        "sentiment": {"type": "string", "enum": ["positive", "neutral", "negative"]},
        "confidence": {"type": "number"},
        "keywords": {"type": "array", "items": {"type": "string"}},
    },
    "required": ["sentiment", "confidence", "keywords"],
    "additionalProperties": False,
}

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "Classify the review. Reply only with JSON."},
        {"role": "user", "content": "The battery dies in two hours but the screen is gorgeous."},
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "review", "schema": schema, "strict": True},
    },
)

# Now it is safe to parse, the shape is guaranteed.
data = json.loads(response.choices[0].message.content)
print(data["sentiment"], data["confidence"])

The strict: True flag is what turns a polite request into a hard guarantee, the model physically cannot emit a token that breaks the schema. Still wrap json.loads in a try/except in production: a truncated response (you hit max_tokens mid-object) produces incomplete JSON. Want to go deeper on schemas and letting the model call your code? That is its own topic, see Structured Output & Tool Calling.

A first look at tool / function calling

Models cannot check today's weather, query your database, or do reliable arithmetic. Tool calling fixes that: you describe a function with a JSON schema, and when the model needs it, instead of answering it returns a structured request, "call get_weather with {city: 'Amsterdam'}". Your code runs the real function, appends the result as a tool message, and calls the API again. The model weaves the result into a natural answer.

1. 1

   Describe the tool

   Pass a `tools` list: each entry is a name, a description, and a JSON schema for its arguments. The description tells the model when to use it.

2. 2

   Model decides

   If the request needs the tool, the response contains a tool_call (name + JSON arguments) instead of normal content.

3. 3

   You execute

   Parse the arguments, run your actual function, and capture the return value. The provider never runs your code, you do.

4. 4

   Feed it back

   Append a message with role `tool` containing the result, then call the API again. The model now has the data it was missing and writes the final reply.

Errors, retries, and rate limits

Your happy-path code works on the first try in a notebook and falls over the moment real traffic hits it. LLM APIs are network calls to a busy, shared service, failure is normal, not exceptional. Plan for it.

• `429 Too Many Requests`, you hit a rate limit. Limits are usually measured in both requests-per-minute and tokens-per-minute. The fix is exponential backoff: wait 1s, then 2s, then 4s (plus a little random jitter) before retrying.
• `5x` server errors, transient overload on the provider's side. Safe to retry with the same backoff. Most official SDKs retry these automatically; check before rolling your own.
• `400 Bad Request`, your fault: malformed messages, a context window that is too long, or a bad schema. Do not retry, it will fail identically. Fix the input.
• `401 / 403`, bad or missing API key, or no access to that model. Retrying never helps.
• Context-length errors, your messages plus the requested completion exceed the model's window. Trim old turns, summarize history, or pick a model with a larger context.
• Timeouts, set a sane client timeout. A streamed request can legitimately run for many seconds; a non-streamed one that hangs for a minute is usually stuck, fail and retry.
• Always cap retries (e.g. 3–5 attempts) and respect any Retry-After header the API sends, it tells you exactly how long to wait.

Common mistakes that cost hours

1. Expecting the API to remember. It is stateless. If you do not resend the history, every turn starts from a blank slate.
2. Forgetting to append the assistant reply. Skip this and the model loses the thread of its own previous answers.
3. Hardcoding the API key in source. Read it from an environment variable or secret manager, never commit it, never ship it to the browser.
4. Ignoring token usage until the bill arrives. Long histories and verbose system prompts are billed on every single call. Trim ruthlessly.
5. Parsing JSON without a schema or a try/except. Without strict mode the model may wrap JSON in prose; even with it, truncation breaks json.loads.
6. Retrying `400`s and `401`s. They are deterministic failures. Retrying just burns time and quota.
7. Setting `max_tokens` too low, which silently truncates answers mid-sentence (or mid-JSON) instead of erroring loudly.
8. Firing hundreds of parallel requests and tripping the rate limit, instead of using a small concurrency limit with backoff.

Takeaways

Where to go next

You can now hold a real conversation with a model from code, stream it, force it into JSON, and survive a bad night on the provider's servers. Two directions build directly on this.

• Go deeper on schemas, multi-tool loops, and agent patterns in Structured Output & Tool Calling.
• Curious why any of this works at all, what "tokens" and "context windows" really are under the hood? Read How LLMs Actually Work.
• See where these skills fit in the bigger picture on the AI Engineer career path.