HOW COME CLAUDE GETS

@arjay_mccandless

94.3K followers · usually 36.8K views a post

Views

130.0K

Likes

5.7K

Comments

47

Shares

245

SoundTikTok sound · original sound - arjay_mccandless

View velocity

+22.8K views over 23.9d · ~952/day · 4 observations

Why it worked

The video explains a technical concept in an accessible way using clear visuals and relatable analogies, addressing a common pain point for users of AI models.

Summary

The video explains the concept of "context rot" in large language models, where performance degrades over long conversations due to the accumulation of irrelevant information. It suggests that "compaction," a process of summarizing and retaining only essential information, can mitigate this issue, though it is a lossy operation.

Structure

1Introduction to context rot
2Explanation of how context rot affects LLMs
3Illustration of context window and compaction
4How compaction works to free up space
5What information should survive compaction
6Limitations of compaction

Product placement

Claude: acts - a large language model that is being discussed and analyzed for its performance characteristics. Removing it would change the subject of the video. It is not clear if this refers to Claude AI or Claude Opus, but the context suggests a specific LLM product. It is not clear if this is a specific product or a general reference to the model family. It is not clear if removing it would change what happens in the video, but it is the subject of the video. It is not clear if this is a specific product or a general reference to the model family.

Original caption

Context Rot: why LLMs get worse the longer a conversation goes. Your favorite AI like Claude Code, Codex, etc has some implementation of this to keep performance reasonable and prevent context from overflowing. #coding #programming #claudecode #ai

More from @arjay_mccandless

On-screen text

Output quality vs. context filled Rotation @ 65% CONTEXT USED --> Auto-compact @ 80% CONTEXT ROT CONTEXT ROT Part 1 Input User message Output Text response Part 2 Input User message Text response Output Text response Part 3 Input User message Text response User message Text response Output Text response Context window Compaction Compaction Compaction free WHAT SHOULD SURVIVE? -> Current goal -> Next task -> Tried solutions -> Important decisions JUST ASK A LARGE LANGUAGE MODEL TO SUMMARIZE THE CURRENT CONVERSATION AND ONLY KEEP THE RELEVANT INFO YOU NEED TO CONTINUE WORKING THINGS LIKE THE CURRENT GOAL THE IMMEDIATE NEXT TASKS THING YOU'VE TRIED THAT HAVE FAILED AND ANY IMPORTANT DECISIONS THAT WILL BE RELEVANT BUT YOU DON'T KEEP EVERY SINGLE TOOL CALL WEB SEARCHES THAT TURNED UP IRRELEVANT RESULTS OR THE EXACT WORD FOR WORD CONVERSATION THAT YOU MAY HAVE HAD SO DOES THIS LET LONG CONVERSATION SORT OF BUT COMPACTION IS A LOSSY OPERATION DO IT YOU LOSE THOSE EXACT DETAILS OF YOUR CONVERSATION THREE COMPACTIONS I LIKE TO JUST A NEW SESSION

Transcript

How come Claude gets dumber and dumber the longer I talk to it? That happens because of something called context rot. The longer a conversation goes, the more stuff that a large language model has to pay attention to at the beginning of a chat, it's very easy for it to always focus on the most relevant information because that's all that's there. But as it gets longer and longer, there's more and more irrelevant information getting passed back and forth to the model. Things like tool calls that may not be relevant anymore, or your initial couple prompts which aren't relevant to your current goal. That makes sense, but how do people get around that? Once your context window hits 70 or 80% utilization, you can perform something called compaction to get some space back. And how does that work? Well, you basically just ask a large language model to summarize the current conversation and only keep the relevant info you need to continue working. Things like the current goal, the immediate next tasks, things you've tried that have failed, and any important decisions that will be relevant going forward. But you don't keep every single tool call, web searches that turned up irrelevant results, or the exact word for word conversation that you may have had with it. So does this let long conversation sort of, but compaction is a lossy operation. Every single time you do it, you lose those exact details of your conversation. After two or three compactions, I like to just run clear or start a new session.