Anthropic engineer:

Hey everyone, I'm. uh I work at anthropological on Cloud Code. Uh before we get started, we have a tradition on Cloud Code where we take a selfie before it talk. So, if you don't mind, if you strike a pose with me, I'll uh take a quick selfie at AI engineer. Okay, incredible. Well, uh, yeah, to kick things off. Like we said, Fable is back. Um, we're rolling it out later today. Uh, keep stay tuned for exact timeline. Me and Cat Woo and Simon Wilson will be doing a fireside chat at 12:30. We might have some updates for you then. Um, but Fable is a model I'm just so, so excited about it's one of those anthropomorphic models where you just like you're just going to remember it like Sonic 3.5 new, OP 4, OP 4.5. It's a model that I just have a lot of like affection and excitement for. And the best way to describe Fable to me is like the the map is opening up. You know, like you were playing like an RPG and you've been on the tutorial and now you get to the point where the like you know the open world starts, right? And there's so much that you you can do and explore. Uh but there's also it's also a little bit intimidating and confusing, right? because there's some of you can do. And so what I wanted to do in this talk is give you guys a field guide to fable, right? How do you work with this new class of models. So I've got four parts to it. I've been working on this as a series of articles and blog post. Uh but you know, when we announced fable was coming out, I was like, okay, let me do uh all of this at once at the talk, uh, you know, uh speed run. So, the four parts, oning Cloud, finding your unknowns, dealing with the grief and being unreasonable. So, first, unholing Cloud. Um, I think something we say really often is that the models are grown not designed, right? We don't wake up and be like, we need 99% on S-bench, right? Like the models are, you know, something we we grow carefully. We give a data and feedback and compute. Um, but ultimately, it's, you know, something that we it's a little bit organic and we sort of figure out and learn with the model as we use it. And so, um, that what that also means that what contains them is us, right? The hardness we put them in and the way we prompt them is basically like a function of our understanding of , right? And by unhobbling it, I mean, how can we understand cloud better to unleash it? And we need to understand Fable more. So I think one of my points is that, you know, uh we're still so early and I think there's a lot more understanding in Fable uh to unlock. And uh I think I'll give you a quick example about how models get smarter because it's a little bit intuitive, right? Like there I saw this viral tweet, couple weeks ago being like, you know, why can't LLMs say which Pokemon end in AW? There are a thousand Pokemon, right? And turns out there are two who whose name is in AW and Redna, right? And it turns out if you ask like a normal chat model, it can't answer it. It's just kind of confusing because like, you know, they definitely know all the names of the Pokemon, right? But if you ask Cloud code can, right? Cuz what it does is that it fetches every Pokemon and writes a script to filter for AOW, right? And so this is what I mean by like unhobbling cloud, we call this capability overhang, right? Cloud gets smarter in spiky ways. So it doesn't just remember every Pokemon and reason through it, but if you give it the code execution tool, it can find the two Pokemon that end with AOW, right? And so this is I think part of the challenge with Fable is figuring out this capability overhang. What is now possible? And I think this is like discovered that I'm excited to go on with you. Uh to make this a little bit clearer, I'm going to talk about a few different examples of how models have progressed in the past. Um, one of the big examples obviously is like chat, you know, the chat models were had to be given context, right? Like maybe you paste in your code base and maybe naively you might have thought like, you know, the way we solve coding is by the context just gets really large and I can just paste in my entire code base, you know, it'll be a 100 million context window. But it turns out that in Instead, if you give it arms, like you give it the bash tool and ways to work with the environment, it can build and search its own context. And that's sort of like the insight that led to Cloud Code, right? And so again, spiky, like a new like innovation kind of, right? And how we think about and work with the model. And then recently we we rolled out cloud tag. Uh and what sort of unlocked cloud tag is it's ability to work proactively and multiplayer. Uh Cloud Code, you know, is something that you have to prompt. for it to do work, right? And uh this ability for cloud to wake itself up and do work is something that we think is unlocking the new wave of agents. But there's there's more here. So, for example, uh we recently removed 80% of the system prompt for cloud code, right? And this is one of the ways in which models, you know, and what they need uh changes over time. So originally, like, you know, maybe back in 3.5 new, the best practices for a system prompt a small system prompt, few tools and lots of examples, right? And then as the models get smarter, you can give them more information and more instructions and they start following them. And so it's a larger system prompt with lots of examples and many tools, right? But most recently we found this new class of models want fewer want a smaller system prompt. The examples tend to constrain it because it's actually more imaginative than the examples we give it. And so uh and we try to give it context and not just constraints. We really try and avoid being like, do not do this, um, which was really necessary for the previous models. Um, and so this is like uh a way that the system prompt is changing and and probably will continue to change. Uh, another feature I really like is the ask user question tool. This was something I worked on when I first got to Cloud code. And and it's uh when Cloud, you know, is is planning or wants to ask you a question, it can show you a multiple choice dialogue. Uh, for open before it could barely call it. I had to like really tweak the tool to make sure that it was uh that it would work, right? And then sometimes at OP 4.5, I was like, well, what if I asked it to like, you know, ask me 40 questions about this spec? It could start interviewing me, right? And so it's ability to ask questions jumped, right? And then most recently with OP 4.8 and Fable, I can now build a whole HTML report with the questions embedded inside of them. And uh It's just like a whole new way of interacting with uh with Cloud, right? And and so this progression of like how Cloud can get information from you is also changed. Um speaking of which uh Mark down and HML is something I've also talked a lot about. Um you know, it turned initially Mark down was a a good output for the model. Um you know, it could show a little bit of rich information. And then you know, with plan mode it started to be for you like you could understand what Cloud was about to do. Um and now you know, cloud can build you these in depth HTML reports, right? So again, a way of this the models getting smarter in a spiky way. I really like to emphasize that this is closer to a biology than a physics, right? It's still very empirical, very organic. Um, we don't know all the rules, but there is some sort of science behind it, right? Like there is an intuition to build as well. And so I really, you know, encourage you to treat fable like that. Uh one of my favorite papers uh that an anthropological that we're written is on the biology of a large language model. Um all of our research papers are meant to be read by, you know, people with various degrees of technical expertise, but this is one of my favorites. So, uh if you're looking to learn a little bit more, suggest you check it out. But so, uh yeah, we talked about unhobbling cloud. But it turns out when you're working with fable, you also need to unhobble yourself, right? And so one of the things that I think a lot about is that The map is not the territory, right? When I'm working on a coding problem, the plan and prompt and spec that I have in my mind is the map. Right? But the territory is the actual code base, the real world, the constraints that cloud needs to navigate, right? And whenever cloud runs into something in the territory that's not in the map, I call that an unknown, right? Cloud has to figure out what to do about it. It's a decision point that I haven't specified. And fable is one of the first models where I felt that like I really have to figure out my unknowns because if not it's going to traverse such a large area that like it's going to run into a lot of them. So how do you figure out your unknowns? Um, I I Fable's bottlestone my by my ability to match the map and the territory to find my unknowns. So a few um few ways to think about this. I like to think of it in a matrix. So like for any problem I have a bunch of known known. This is usually like what I write in my prompt. What do I want, right? Then I have known unknown. Things that like I know I haven't don't really know yet, but I just haven't figured it out yet. I can um uh yeah, then I've got unknown known. like what's so obvious that I just wouldn't write it down, you know, but I I know it when I see it, right? And then finally unknown unknown unknown. What haven't I considered it all? What do I not know, right? Like what is something that if I knew could change how it prompt cloud? And and luckily, you can use cloud, you can use fiber to find your unknown. So I'm going to go over a few examples of how I do that with fiber. Um, the first is I like to do what I call a blind spot pass. So I like to say something like, hey, I'm working on a new OS provider that I know nothing about uh like in this code base. Can you do a blind spot pass to help me figure out my relevant unknown and known and help me prompt better, right? And so this like might have cloud go through the the and figure out like, oh you know, this kind of like a very dead end that comes up a lot. Maybe search my get dif or slack. I might tell it where there's context, right? So that I can learn about, you know, all the. And and you can use this very broadly, right? You can use it to teach you about new fields. I I recently did this for color grading when doing video editing. Um I think it's really powerful and and is incredible at it. Um in many ways the model knows more about you know, almost everything then I do. I just need to get it out of it. Um, then I like to use brainstorms and prototypes. Uh this helps me figure out my unknown known, right? Things like especially for design, for me it's like know when you see it, right? So I might ask it to uh create a dashboard, um, and I tell it I have no visual taste. Uh make me an HTML page with four widely different design decisions so I can react to them, right? And and you know, you tweak this as you want, but like the idea is to sort of get an idea like what are the things that you you know you can't describe in words right and like work with the model to help figure that out. then then interviews so once I have an idea of like this is what I want to do there's probably still a lot of like unknow here right where I might not have considered something I might not have specified it and so I'll ask to interview me right and I'll give it a little bit more context any of these questions, like giving it a little bit more context about you and the work and the stage you're at like, hey, yeah, prioritize questions that would change the architecture is extremely helpful. Uh, then references. One of the best ways to give Cloud a map is to give it another map, right? So, instead of me writing out the spec, uh, I can just say, hey, here's some code that represents what I want to be done, right? It could be in a different uh, system or language, uh, but just read this code, understand it and then use that to start your work, right? And uh again, this can be in a lot of different ways if I'm making a a React component, I might have an HTML mock up that is my map, right? That I pass in as a reference. I think this is really, really powerful. A Firebase is really incredibly at it. Uh something else I've like really appreciated is implementation notes. So if uh while you're running Firebase uh and it runs into an unknown ask it to log it, right? So that um you uh you can see where the deviations happened and then you can sort of figure out why as well. You know, we'll usually give you some context about what happened. And then finally, I like to get a fable to quiz me about what happened. Uh just to make sure I understand what I'm doing and I can represent this work, you know, when I'm creating a PR or merging it. Um, this is a really great way of like making sure that you're like really in the loop with fable. And I think that's like one of the most important parts of fable is like staying in the loop and making sure that you uh you get what you want. So, um there's some of my tips for working with Fable. Uh I also want to say that the first time I used a Metos class model, uh I used Fable. I felt both a huge sense of like gain but also a sense of loss and I I wanted to talk a little bit about that, you know. Um when I think about coding before LLMs, it feels like a foreign country. You know, like I used to run a YC startup about 30 people and we were just constantly forced into tradeoffs because of how hard code was, right? Like we could make the the app fast or we could try prototyping a new feature and and this might take a month or this would take two months and so we had to choose. It was really, really hard. Um, and now I went back to that code base a couple weeks ago and I thought about some of the things that I wanted to do and uh it was just way easier. It was like the things that would have taken me weeks, I could do in hours, you know? And uh at some point it's like, yeah, like how can you not laugh? Also, how can you not cry on honestly? Like it's like one of these things where um I really really loved programming and writing code by hand. I love the feeling of like seeing the code base in my mind and like rotating it. I also remember just, you know, like staying up late nights, trying to debug, working on things for weeks without working, right? I just remember swimming in failure. I just remember that like the most of the projects I've ever worked on have failed. Most startups go bankrupt. You know, I think just overall programming and coding is extremely hard and like as much as I enjoy those high, I I cannot go back, right? And uh the way my reflection here is like the only way out is through, right? There's still a lot to learn with a gent to coding. There's a lot to learn with Fable. Uh but I think if we try really hard and if we like stay in the loop, we unhobble it, uh, we can get there, you know, and we can come out on the other side uh with just um, so much more. And so the last bit I wanted to talk about is is the so much more part, right? I call this being unreasonable. Um, one of my favorite parts of anthropological is that we believe that tradeoffs are not real. Um, like I think that very often I Like in my previous company, I was very used to being reasonable. So I'd like write down this list of priorities and I'd be like, well, I guess we can prioritize this against this, right? Um and uh like you know, that makes sense. So we'll we'll this will be our priority this quarter. But what if you uh just did all of it, you know, what if you force the reality to show you the trade-off, right? Um this is something I really valued at our culture and anthropological and my reflection going forward is that I'm going to be a lot less reasonable. Um I think one of this like the math of cloud and fable really changes how you think about tradeoffs and there are so many tradeoffs that you make implicitly in your head, right? Like good, fast, cheap, now it's pick three, right? Um, I think that like the best way to like do more ambitious work is to uh like reframe and make big make ourselves more ambitious. Because I think the only way to prove that agent's work is to do the best work of our lives faster than ever before. Um, you know, for example, I made this deck last night in about four hours with Fable. I feel like it's a it's a deck I really like and I I really enjoyed it, but I also um, you know, did it really fast. Uh, and I think that if you're here, you know, at I engineer, the world is kind of looking at you to prove that AI works, right? It's not just like a fad or something, but that it can make us more productive and also save us time. And and that's my resolution for this year is to work be more productive but work less and spend more time with people I really care about. Uh, I think it's also worth calling out that building is easier, but generating value is still hard. And I think this is something that we run into, you know, as AI engineers sometimes where we think so much about the process of building and our our setups, um, but the the point is to generate value, right? And uh, there it takes a lot of swings. It takes a lot of tries to find the valuable stuff. Uh but that really is the goal and that's like, you know, again what the world is looking to us to prove that AI can really transform it. So to to end, I just wanted to say like, go explore, make it real and uh yeah, be less reasonable. Thank you.