Nested AI agents: why our album chat has a search agent of its own

A chat assistant is built on a context window — the running transcript of everything said and done in the conversation. Every time the assistant takes another turn, the entire window is sent back to the model. That has a sneaky consequence: anything bulky that enters the conversation is paid for again on every later message.

Photo search is exactly that kind of bulk. Our earlier design had the chat pull raw photo listings straight into the conversation: for a 219-photo album, that meant 219 blocks of captions, dates, quality scores, and tags landing in the transcript. We measured a single turn of that conversation at hundreds of thousands of input tokens — most of it photo metadata the user would never see, re-sent with every subsequent request.

Big contexts aren’t just expensive. Models get measurably worse at following instructions when the important sentence is buried under a mountain of listings — the “lost in the middle” problem. Our assistant didn’t need a bigger memory. It needed a way to do bulky work without remembering it.

AI assistants act through tools — small functions like place this photo or apply this theme. A tool takes an input, returns a result, and the conversation moves on. The nested-agent insight is that a tool doesn’t have to be a simple function. A tool can be a whole other agent.

In Sustain, the album chat now has a tool we call dig_photos. When your request needs real digging, the chat hands your words — verbatim, in whatever language you wrote them — to a second agent: the dig agent, the same search brain that powers photo search in our iOS app. The dig agent runs its own private loop:

• Search — combining what’s actually in each picture (every photo is described in detail when it’s analyzed, so “beach” can match a photo nobody ever labelled) with tagged people, dates, and quality filters. It can search several times, narrowing as it learns what’s there.

• Verify — reading the AI-written descriptions of candidate photos to drop the near-misses: the beach photo where nobody is laughing.

• Refine — narrowing or re-ranking, keeping the best of burst duplicates.

• Finish — returning an ordered, vetted list with a one-line summary.

All of that reading and retrying happens in the dig agent’s own, throwaway context. When it finishes, its workspace is discarded. The only thing that enters the album chat’s conversation is the final list — a few hundred tokens instead of tens of thousands.

This is the benefit the whole pattern hangs on. The chat’s transcript stays lean — your messages, its answers, and compact results. The heavy reading is quarantined inside the sub-agent and thrown away when it’s done.

Three things follow directly:

• Cost stops compounding. Bulk that never enters the transcript is never re-sent. Counter-intuitively, running a second AI agent is often cheaper than the flat design, because the flat design kept re-paying for those photo listings on every turn.

• Quality goes up. Each agent reads a short, focused prompt about its own job instead of one mega-prompt about everything. The searcher searches better; the designer designs better.

• Long conversations stay sharp. A session that searches five times doesn’t degrade, because none of those searches left residue in the window.

Because the dig agent is a self-contained unit with a clean contract — query in, vetted list out — we point more than one product surface at it. The photo search on our iOS home screen and the album chat on the web now share the same search brain. When we teach it a new trick, every surface learns it at once.

This mirrors a principle we apply across Sustain: one engine per job. The same week we shipped nested search, we also unified photo placement — the chat’s “rebuild my album” now drives the exact same pipeline as the AI Enhance button, parameters and all, instead of a lookalike. Two engines that drift apart will eventually disagree; one engine can’t.

A nested agent is a contained blast radius. Ours runs at most a handful of reasoning rounds, on a small fast model suited to search, and its output is validated before it’s trusted — every photo id it returns is checked against the photos you actually own, and if it wanders off-script the system falls back to a plain filtered search rather than failing your message.

That containment is also what makes the pattern safe to extend. The dig agent can get smarter, chattier, or more thorough internally without any risk of flooding the album conversation — the interface between the two agents doesn’t change.

Nesting isn’t free — a sub-agent adds a round of thinking, which means a moment of latency. So the album chat uses a simple escalation rule:

• Deterministic lookups stay flat. “Photos from 2023,” “photos of Omri,” “unplaced portrait shots” — these are instant database filters. Wrapping them in an agent would just make them slower.

• Fuzzy, multi-constraint requests escalate. “Omri laughing at the beach,” “the best 20 from the Rome trip” — anything that needs visual search plus judgment goes to the dig agent.

The orchestration level should match the complexity of the question. A good rule of thumb from our experience: if a human assistant would need to open files and read before answering, nest it. If they’d just run a query, don’t.

Here’s an actual exchange from a travel album. The request is deliberately awkward — it describes a scene, not a keyword:

“find photos with mountain views”

A couple of seconds later the answer came back as pictures: a swipeable strip of the seven matching photos, each with its number. Scroll to the end and one is outlined in gold and tagged new — the only one of the seven not yet on a page, which is the single thing you might actually want to do something about. Nothing to decode, nothing to cross-reference.

Underneath that strip, the dig agent had run its own loop: searching the album, then reading the candidates’ descriptions to keep the real mountain views and drop the near-misses. All of that reading happened in the sub-agent’s context and was thrown away. The album chat’s transcript gained one short entry.

Then a second, narrower question in the same breath:

“and the photos of people on a beach at sunset?”

Three matches this time — few enough that each gets a full row: the thumbnail, what’s in it, and an Apply button to place it. Same agent, same search, different presentation, because three results and thirty results are not the same kind of answer.

The distinction the agent is making here — a beach at sunset with people in it versus an empty sunset over the ocean — exists in no tag or filter. It comes from a second agent having actually read what’s in the pictures, in a context you never have to pay for again.

None of this changes how you use Sustain — that’s rather the point. The assistant just gets faster to talk to, better at hard searches, and identical across devices, because the work happens in the right place. The next time it answers a needle-in-a-haystack request in a couple of seconds with a clean shortlist, you’ll know: there was a second agent in the room, and it already left.