Elephants, Goldfish and the New Golden Age of Software Engineering – i-programmer.info

A new research paper from Google has some very interesting ideas about using AI effectively in software development based on experiences by the author, Dave Rensin, Distinguished Engineer at Google.
In the paper titled Elephants, Goldfish and the New Golden Age of Software Engineering, Rensin sets out the case for using both elephants and goldfish as useful models. He starts from the obvious argument that while AI speeds up coding, it can also quickly generate a lot of mistakes and messy code if it isn’t carefully managed by human oversight and clear processes. 

He says the first thing to do is “to treat AI as a Tool, Not a Toy,” arguing that people often get poor results by asking AI to do everything in a single prompt. Instead, users should have back-and-forth conversations with AI to question assumptions, set clear grading rules, and guide the research. This is where the elephant and the goldfish come in. Both ideas relate to the creation of an easily-read, well-written design document as being more important than the code itself. 
To achieve this, Rensin says what you need to do is a two-part process. First, you feed the elephant (which famously never forgets) with all the information about what you want to do, so it can help you write the design document. Then you show the resulting design document to a brand new AI session with zero knowledge of the project, and check that it is understandable. 
So the first phase is to create your elephant. You open a fresh session with your AI tool and point it at the relevant design docs or source tree in your organization. Ask it to give you a high-level description of what it learned. If it misunderstands your system, you correct it. You then tell the AI tool you don’t want code, you’re going to describe a feature and you want the tool to ask clarifying questions and challenge your assumptions.
Rensin advises:
Talk to the tool like a human. Have a back-and-forth conversation for 20–30 minutes. If the AI hallucinates, point it to the file that corrects it. Really argue with the AI. These models are excellent interrogators so make it really push your thinking about the description and edges of the problem you are trying to solve.
Rensin points out that because AI models are trained to be helpful, this often makes them insufferable sycophants.
If the AI stops asking hard questions and just says, “Great idea, Dave!”, you have to force it back into critic mode. Tell it: “You are not being helpful. Your highest and best use is to challenge my thinking.”
Next, you ask the AI to propose an implementation in prose and block diagrams. Rensin says the prompt he most commonly uses is:
“Based on your understanding of the codebase and what we discussed, I would like you to give me a first draft proposal of a technical implementation to actually make this feature happen. I’m not looking for code. I want prose from you that demonstrates your understanding of my system. Short blocks of pseudocode are fine if you think that will help, but I would strongly prefer clearly written text and block diagrams.”
The first draft is going to be (at least a little) wrong. Ask clarifying questions (e.g., “Why did you choose that pattern? I thought the system worked like X…”). Argue with the AI until you settle on a solid technical approach.
When the model hallucinates something about your system, point it to the file that will correct its understanding. “Nope. I don’t think it works that way. Please read auth.py. (or AUTH.md or whatever)”
Now you create the markdown document that will serve as the absolute source of truth and the guardrails for the code. Don’t ask the AI to write the whole doc at once because the current models have token limits on the outputs they will produce at once. Build it iteratively in the same chat session.
Now we get to the goldfish. To find out if your design doc is actually good, you test it against a goldfish – a brand new, empty AI session. Give it your markdown doc and say:
“Read this document and the files it references. Tell me what it’s trying to accomplish, and how my system currently works as it relates to this feature.”
If the Goldfish cannot explain your system based only on that document and the files it references, your doc is missing context. Add the details and repeat until it passes. Then you start another goldfish session, telling it:
“Assume the role of an expert technical reviewer. Read this design doc and all the files it references. Tell me all the things I missed, all the faulty assumptions, all the edge cases I’m missing, and things I should have considered but did not. Every mistake and ambiguity you find makes you more helpful and useful.”
Rensin says that anecdotally, he finds about 30% of its suggestions are highly valuable, which is more than enough to make this worthwhile. You then update the file accordingly, and repeat until the recommendations and questions start falling consistently into the category of “nit pick.”
At this stage, you start a new Goldfish and ask it:
“You are an experienced software engineer experienced with our codebase. Read this document and the files it references and tell me: Does it absolutely have all the information you would require to successfully implement this feature in your first pass?”
If it asks questions, answer them in the doc. Rinse and repeat until you are satisfied that there aren’t important missing items or unresolved ambiguities.
Finally, show it to a real human for review and approval. 
Rensin says other engineers usually say these ideas are fine for new projects, but when a codebase is millions of lines long, it looks too challenging to write design docs for all of it. Rensin argues that training an elephant still only takes about a week, and the trick is to feed the elephant small chunks – peanuts and hay – small, highly nutritious, compressed bites of context, by assigning each developer a set of directories within the project and getting the AI to generate a readme.md file explaining the purpose of the directory and the files contained in it. Once the leaves are done, move up one level. 
The full document is available on Medium.  
Elephants, Goldfish and the New Golden Age of Software Engineering
Get Started With Vibe Coding!
AI Breakthroughs But At A Cost
Google Introduces Developer Knowledge API

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Facebook or Linkedin.

Microsoft has released Coreutils for Windows, a set of Linux-like command line utilities that run natively on Windows. The announcement of the utilities was made at Microsoft Build last week.

We explore a combination that opens new ways of interacting with the Neo4j Graph database.

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info

source