As the context grows, the output (usually a structured function call) remains relatively short. This makes the ratio between prefilling and decoding highly skewed in agents compared to chatbots. The problem is that context engineering is still an emerging science, even though for agent systems, it's already essential. This is overall a really interesting post and makes me think of Chroma’s technical report on context rot: https://research.trychroma.com/context-rot
As the context grows, the output (usually a structured function call) remains relatively short. This makes the ratio between prefilling and decoding highly skewed in agents compared to chatbots. The problem is that context engineering is still an emerging science, even though for agent systems, it's already essential. This is overall a really interesting post and makes me think of Chroma’s technical report on context rot: https://research.trychroma.com/context-rot