Large language models (LLMs) are rather ubiquitously presented as a technology so transformative that we are supposedly standing on the verge of the next industrial revolution. But the truth, with respect to application security at least, is a bit more pedestrian. The LLM value proposition for AppSec is, at best, unclear. We have seen some genuinely innovative and productive uses, but these case studies are usually presented in ways that discount the significant engineering effort necessary to make them work. Moreover, for each novel demo there are many more “solutions” that are nothing more than a chatbot duct taped to a preexisting tool. The strategy behind these sales pitches is to use blitzscaling to entrench LLMs as dependencies in other businesses before the venture capital dries up; before these services are no longer heavily subsidized. The message being all you need is a little “AI” pixie dust to help you find more exploitable vulnerabilities faster with little to no additional effort.

This sales pitch exploits our habit of guesstimating or anticipating technological progress based on our experience with the steady advancement of hardware over the past six or so decades (i.e., Moore’s Law). We think of computing ability as being limited by how “fast” or “powerful” any particular computer is because it’s intuitive. For example, how much RAM a computer has or the number of CPU cores. But the pace of machine learning advancement we’re living through is much more complex than the physical constraints of the computers hosting LLMs. Our understanding of how to implement intelligence is significantly more limited than, say, graphics or operating systems. Developing that understanding is a question of scientific research. And scientific research doesn’t proceed along neat eighteen month cycles. A contrasting example from computer graphics is ray tracing, a technique we have formally understood for decades. However, we’ve only seen hardware capable of real-time ray tracing in the last few years. This means there is no reason to feel at risk of falling behind because we want to adopt a mindful, deliberate approach to LLMs. Let’s start by demystifying the pixie dust and figure out how language models perform inference, the process of evaluating and responding to user input.

Language Models Aren’t Magic#

The inciting incident of the current AI boom was the 2017 research paper “Attention Is All You Need” by Google machine learning researchers. At a very high level, the key insight of this paper was the idea of diffusing meaning across the entire context window of an LLM session. The context window is the totality of input that is considered by a model when responding to a query, and it is composed of a series of tokens. When it reaches its maximum length it is truncated, commonly by dropping the oldest tokens. For our purposes, we can think of tokens as words though in reality they are more granular representing punctuation, affixes, pieces of compound words, etc. What’s important is that the actual meaning of each token is allowed to influence every other token in the context window.

Essentially, language model inference works by cross-pollinating meaning from each token to all the others. Unfortunately, this is a computationally expensive process which grows quadratically as the context window grows. We can even express this formally. In computer science we would describe the self-attention mechanism used to diffuse meaning as having a runtime complexity of 𝒪(𝓃²) where 𝓃 is the length, in tokens, of the context window. In more intuitive terms, this means a language model’s work roughly quadruples every time the context window doubles in size. In the figure above, the numbered nodes represent tokens, and the edges (or arcs between the nodes) illustrate an LLM calculating pairwise relationships, token by token. Each pairwise calculation must be performed on every query despite the fact that subsequent queries are added to an existing context window. This cost does not level-off or plateau with scale; there are no economies of scale to be had with 𝓃² algorithms. To fully cross-pollinate a context window of ten tokens requires an LLM to perform one hundred individual pairwise propagations; one hundred tokens would require ten thousand. This section so far is approaching 500 tokens depending on the model.

And tokens are not the irreducible, basic unit of language models. Each token is a high-dimensional vector (i.e., a long list of numbers) which embeds it in a semantic space enabling mathematical operations on tokens. A single token in a frontier model is represented by somewhere in the ballpark of tens of thousands of values. “High-dimensional” is something of an understatement. When a language model performs a pairwise propagation (each edge in the figure above) that task is comprised primarily of many multiplications, specifically it is computing the dot product of the vectors representing the two tokens. More plainly, each of those edges represents thousands or tens of thousands of multiplications. Now consider that an attention transformer does not simply apply self-attention once per query. Self-attention is computed on the entire context window tens or even hundreds of times per query (i.e., every time you press enter when conversing with a chatbot).

After all of those repeated steps, it is finally time for the LLM to start generating its response. As a result of its training, a language model is equipped with an unembedding matrix or a dictionary of known tokens and their meanings. The results of self-attention then allow the model to calculate which tokens in its dictionary have a high likelihood of being pertinent to the query. Intuitively, which tokens would be “nearby” (in the semantic space) to the last token in the context window. In order to prevent overly restricted output, selection is not as simple as picking the most likely next token. Instead selection involves statistical methods like softmax and temperature that strike a balance between relevance and novelty. The key take away is that the generation of tokens is probabilistic. Another thing to keep in mind is that generated tokens are appended to the context window since they are, well, contextually relevant. And, yes, context windows are prone to growing quite rapidly. This is a very high level understanding, but it should equip you with a useful scaffolding which you can use to build a more complete understanding as you need. 3Blue1Brown has an excellent series on machine learning if you’re looking for a good next step:

Mozilla Harnesses Mythos#

With Anthropic’s recent launch of Claude Mythos, they undertook a massive PR campaign to convince the world of its unprecedented capabilities. They recruited Mozilla to this effort by offering them free use of Mythos to analyze Firefox’s source code for security vulnerabilities. The results were genuinely impressive with some two hundred seventy one flaws identified and a claimed false positive rate of effectively zero percent. This was a remarkable outcome because Mozilla’s previous experiments with LLMs resulted in an avalanche of slop findings (findings that were plausible but unvalidated). Previous models had effectively automated the generation of such plausible findings, but determining if they were actual vulnerabilities or hallucinations (a phenomenon more accurately referred to as confabulation) required significant skilled human labor. So for Mythos to generate hundreds of meaningful findings with a vanishingly small false positive rate is legitimately impressive. However, this outcome is not quite the singularity it was presented as.

The results were presented as though Mythos had crossed some capability threshold finally allowing it to produce useful findings with little guidance. As though you could just point Mythos at a git repo, and let it do its thing. But a crucial aspect of Mozilla’s success was the use of an orchestration wrapper called an agentic harness. A harness is a piece of software that implements traditional task specific logic, and uses an LLM to drive its execution. This is accomplished by providing the LLM with a tokenized description of actions available in the harness. The harness can then feed data into the LLM allowing it to probabilistically direct the harness. Mozilla’s developers put significant engineering effort into building a harness highly tailored to the Firefox project which leveraged Mozilla’s preexisting testing infrastructure. The way Mozilla describes developing their harness, almost casually, undermines the Mythos PR narrative:

We began with small-scale experiments prompting the harness to look for sandbox escapes with Claude Opus 4.6. Even with this model, we identified an impressive amount of previously-unknown vulnerabilities which required complex reasoning over multiprocess browser engine code.

Behind the Scenes Hardening Firefox with Claude Mythos Preview

Mozilla Hacks

This calls into question just how vital Mythos, specifically, was to this entire exercise. By Mozilla’s own account, they were able to achieve massive improvements to the quality of findings with older, widely available models suggesting their agentic harness might be the component actually doing the heavy lifting. And if you’re familiar with the technique of static analysis called abstract interpretation, the question becomes all but unavoidable.

Abstract interpretation allows us to model a computer program’s behavior by strategically over approximating safe outcomes. We can then validate the program’s correctness by asking if it can be coerced outside of that safe approximation. An intuitive example from AppSec is authentication. A login event should either result in a successfully authenticated user granted additional functionality or a simple error (e.g., incorrect username or password). If specific inputs result in user-visible database errors, however, the web app’s output is not within the safe approximation; it is susceptible to SQL injection. When discussing how their harness works, Mozilla devs describe a methodology that bears an uncanny resemblance to abstract interpretation:

With these harnesses, so long as you can define a deterministic and clear success signal or task verification signal, you can just keep telling it to keep working. In our case when we’re looking for memory safety issues we have our sanitizer build of Firefox and if you make it crash you win.

Brian Grinstead, Mozilla Distinguished Engineer

curl, the widely used HTTP request utility and library, was also offered free access to Mythos to scan for vulnerabilities, and the results offer an instructive comparison. curl’s approach was much closer to a traditional static analysis than Firefox. That is Mythos only reviewed the source code and did not use a test harness to run test instances of curl. The scale of the findings was significantly smaller but still impressive (keeping in mind that curl is a significantly smaller code base that is also one of the most reviewed). curl had a similar outcome of very few false positives. Mythos reported finding five security vulnerabilities and about twenty non-security related bugs. If you have experience with more traditional static analysis tools, the lack of untold thousands of results is likely already impressive.

Upon review, the curl developers determined three of the five security findings to be false positives, one to be an additional non-security bug, and one low severity vulnerability. And of the twenty non-security bugs, Daniel Stenberg, lead developer of the curl project, reports almost no false positives. The point here is not to “disprove” the value of Mozilla’s exercise. Rather both efforts and the contrast between them lead us away from the marketable conclusion that LLMs suddenly outclass human hackers. Instead, they point us to an unglamorous reality: success requires domain expertise and hard work. And that unglamorous conclusion further leads to a few very compelling questions that simply aren’t being asked by the broader industry:

Skepticism Not Cynicism#

There is an unresolvable contradiction in the way LLMs are being sold. As developers, we’re told we must incorporate LLMs into our work because productivity is a metric of singular importance. If we’re not maximizing productivity, it is a professional failing. The same industry then tells us computer and software security are under critical threat because LLMs enable rapid vulnerability discovery meaning threat actors now have an insurmountable advantage. Of course, this obligates developers to also use LLMs to identify vulnerabilities first which is…convenient. But LLMs are trained on untold volumes of existing code. And software projects are filing record numbers of CVEs because of LLMs. Connecting those dots, LLMs are trained on the very same insecure code they’re finding vulnerabilities in at a record clip, and the same LLMs used for vulnerability discovery are used for code generation. Essentially, many devs have been compelled to seed security vulnerabilities far and wide in the service of maximizing their output volume.

We should pause to take stock of our history in computing and how past paradigm shifts have altered our profession. We do not have to proceed as though we have never been here before. Back before Unix, when assembly was a high level language, a relevant example comes to mind: the invention of the compiler. Today many developers pay them little mind, but at the time there was significant resistance to the very idea. There was no way a mindless computer program could generate acceptable, let alone good, assembly. Assembly was a craft. Human guile and insight were required to perform such feats. Besides, what would become of all the programmers? Would we not be obsolete? Grace Hopper, a pioneer of compiler construction, called out the narrowness of this view. She correctly predicted the compiler would vastly expand the capability of computers inducing far greater demand for developers rather than rendering us redundant.

Compilers, as we now know, did precisely that. Now that we no longer need devote our time to tedious activities, like juggling registers or timing read/write operations, we can get on with the business of solving problems with computers. Hopper’s position stands in stark contrast to today’s AI marketing narrative which actually embraces the old-fashioned, narrow view that developer skill is now obsolete. Like the compiler before it, the LLM cannot replace developers (or software testers, or UI/UX designers, or pentesters, etc.). To be leveraged effectively, language models demand developer skill as Mozilla’s experience with Mythos demonstrates. Good software must be designed not generated. And design is an abstract process. Not to mention the teensy, tiny, itty-bitty problem—hardly even worth mentioning—that you need domain expertise to spot LLM confabulation. Yes, LLMs can indeed free us from even more drudgery, but someone must still understand the machine being built even when it’s built with ones and zeros.

Successfully directing an LLM to produce working code requires hard won engineering expertise and, crucially, domain knowledge. Any code produced by an LLM must be fully reviewed and understood by human devs in order to identify bugs. Both bugs directly in the generated code and bugs that may arise from integrating with existing code. It is also worth noting serious flaws in human written code slip through in spite of code review having become standard practice. Now, we’re moving towards generating code at volumes well beyond what a human could write and expecting code reviews to catch all the bugs that may arise in and from that code? We haven’t solved the problem at human scale, but we’re already proceeding to machine scale.

Go Your Own Way#

So, what does all of this mean for you and your organization? Unfortunately, there are no simple or concrete action items that are broadly applicable. The landscape is simply too unsettled. Instead, remembering there are no easy answers will help you recognize when someone is trying to sell them to you wrapped in AI industry lingo. Common advice offered for software projects or IT infrastructure is that you might as well start using LLMs now before attackers beat you to it. But remember, a lot of open source projects are being given the necessary compute for free as part of the marketing campaigns of frontier labs. Unfortunately, the rest of us will be expected to pay those compute costs. And, rather notably, the actual costs of exercises like Firefox (above) have not been disclosed—a pattern that is suspiciously common when it comes to these highly publicized feats. For most of us, deploying an LLM almost certainly means trading one expense for another rather than eliminating or reducing an expense. The old adage “there is no free lunch” is clichéd for a reason: it’s true.

Another helpful heuristic is the standard consumer advice: make your purchasing decision based on what the product can do now. Never buy something because the vendor is promising incredible functionality in the future. This is particularly salient because LLMs, and their promised solutions, are not so inevitable as the marketing would have us believe. As we’ve seen, LLM inference is compute intensive, and training significantly more so. (As a side note, these compute demands are the why behind the sudden, need to build out data center capacity at unprecedented and accelerating scales.) Marketing around LLMs regularly employs fantastical tales of the future and inevitability as misdirection to conceal the prodigious costs of running frontier models. Those costs raise an altogether unsettled question. Are frontier models financially sustainable? If not, much smaller, highly specialized models will start to look far more appealing. And those specialist models are entirely capable of producing comparable results to frontier models but in a constrained space.

Language models are seductive because they seemingly offer relief from the tedious and routine. But often that promise is just another easy answer, a sort of computational cure-all, presented nebulously as “AI”. However, successful use of LLMs still requires us to do the work of evaluating their suitability to a given scenario, honestly assessing the costs and benefits, and finally implementing. Just like any other tool, we have to do our homework. Computing is a challenging profession that requires abstract problem solving. As alluring as it would be to have a pixie dust automate every labor-intensive task, trying to do so with statistical token generation amounts to nothing more than alchemy. During the frenetic stage of the hype cycle, taking the time to look past the smoke and mirrors is a vital aspect of our homework. How will AI shape the future of application security? As corny as it sounds, it really is up to us.

Remember the hacker ethos.