LLMs reward expertise

The post argues that LLMs make many tasks accessible to non-experts, but that this does not remove the value of expertise. The key advantage in prompting is knowing the domain well enough to judge outputs, ask sharper questions, and steer the model toward a better solution.

Terence Tao’s ChatGPT conversation is used as the main example. Tao keeps messages short, signals that he is talking to a mathematician, pushes back when something looks off, and suggests his own directions. The author extends that pattern to software work: people who know their codebase can press an LLM harder, while people without domain knowledge can mostly only use it to get something workable.

Reading notes

#

• LLMs make everyone into a generalist, but that does not mean skill disappears
• The most important skill in prompting is expertise in the domain being prompted for
• Tao’s prompts are short and focused on the gist rather than point-by-point replies
• His questions signal that the model should answer as if talking to mathematicians
• He pushes back on responses that look wrong without directly contradicting them
• He also makes leaps and suggests alternate directions himself
• The technique depends on understanding the mathematics, not only on prompt style
• The same pattern applies to codebases and system design work
• Familiarity with concrete specifics matters more than generic software principles
• Without domain knowledge, LLMs are useful for getting something usable
• With domain knowledge, the same LLM can be steered much harder toward the desired result
• Human expertise remains a bottleneck because the hard part is communicating the right kind of solution