Large language models (LLMs) are rather ubiquitously presented as a technology so transformative that we are supposedly standing on the verge of the next industrial revolution. But the truth, with respect to application security at least, is a bit more pedestrian. The LLM value proposition for AppSec is, at best, unclear. We have seen some genuinely innovative and productive uses, but these case studies are usually presented in ways that discount the significant engineering effort necessary to make them work. Moreover, for each novel demo there are many more “solutions” that are nothing more than a chatbot duct taped to a preexisting tool. The strategy behind these sales pitches is to use blitzscaling to entrench LLMs as dependencies in other businesses before the venture capital dries up; before these services are no longer heavily subsidized. The message being all you need is a little “AI” pixie dust to help you find more exploitable vulnerabilities faster with little to no additional effort.

Application security penetration testing has stagnated. AppSec hasn’t had any developments that compare in breadth and scope to red teaming, the more holistic approach to network pentesting. Yes, we continue to work around the margins of the common vulnerabilities that are inherent to web and mobile applications, and new vulnerabilities will continue to impact server software and middleware. But we have not evolved our testing methodology alongside the development methodologies we have informed. And no, duct taping generative AI onto our existing tools in the vain hope a use case or two will fall out doesn’t count. At (Re)clarative, we believe it is time to pursue new AppSec testing strategies through the development of new tools.