Alan Nichol is Co-founder & CTO of Rasa, a conversational AI framework that has been downloaded over 50 million times.
Enterprise AI leaders want systems that work, not just in demos but in real-world operations. Large language models (LLMs) promise flexible automation, capable of reasoning through complex workflows with minimal programming effort. The idea was simple: Build a prompt, connect an API and let the model handle the rest. But as enterprises push AI into production, many are hitting the same roadblocks—unpredictable outputs, escalating costs and security concerns.
We’ve seen this challenge unfold repeatedly as enterprises attempt to scale AI. We hit a breaking point after six months of testing LLM-powered automation for our core business processes. The fully agentic approach, where AI autonomously handled tasks, proved too inconsistent for production use.
It became clear that we couldn’t achieve the reliability, security and cost-efficiency enterprises demanded without structured automation. That realization reshaped our approach to AI implementation and revealed what truly works at scale.
The Reliability Crisis In Enterprise AI
Unpredictable system behavior is the top reason enterprise AI projects fail. Gartner predicts that by the end of 2025, at least 30% of generative AI projects will be discontinued after the proof-of-concept stage due to challenges such as poor data quality, insufficient risk management, rising costs or a lack of clear business value.
This aligns with findings from the LangChain “State of AI Agents” report, where 41% of respondents cited performance quality as the biggest limitation to putting more agents into production. These failures often stem from what we call the “prompt-and-pray” model—where business logic is embedded entirely in LLM prompts, with developers hoping the model will consistently follow instructions.
This approach creates fundamentally unreliable systems. Across multiple tests, agentic AI assistants introduced inconsistencies in execution over 80% of the time, often misinterpreting requests, generating conflicting responses, or failing to follow business logic. This inconsistency is unacceptable for enterprises handling thousands of customer interactions daily.
Beyond reliability issues, cost is another significant factor. While LLM pricing fluctuates, enterprises must consider long-term cost efficiency at scale. Fully agentic approaches introduce unpredictable resource consumption, inefficient token usage and increased latency, all of which compound over millions of interactions. Structured automation significantly reduces these inefficiencies, ensuring AI systems remain cost-effective, scalable and operationally predictable.
The Three Paths To Enterprise AI Implementation
Through our experimentation, we’ve identified three distinct architectural approaches to integrating LLMs into enterprise systems:
1. Full Agentic Model: Business logic resides entirely in prompts, with LLMs making all decisions about execution paths. While this provides flexibility, it comes at the cost of reliability.
2. Hybrid Model: LLMs manage some decisions, while rule-based systems handle others. This setup improves consistency compared to fully agentic approaches but still relies on traditional logic for high-stakes decisions, limiting scalability and flexibility.
3. Structured Automation: This approach separates conversational ability from business logic execution. LLMs handle intent recognition and response generation, while predefined workflows execute business processes deterministically.
Our metrics show that structured automation consistently delivers better results across key performance indicators. By separating conversational ability from business logic, we reduced costs by up to 77% per interaction, decreased latency by a factor of four and achieved 99.8% execution consistency compared to 68% with purely agentic approaches.
Building For Enterprise-Grade Reliability
Structured automation acknowledges that LLMs excel at understanding natural language but struggle with consistent execution. By playing to these strengths and weaknesses, enterprises can build systems that combine conversational AI’s flexibility with traditional software’s predictability.
Key architectural considerations include:
• Using LLMs For Interpretation, Not Execution: LLMs should recognize intent and generate responses, but deterministic workflows should handle business logic. For example, an LLM can identify a customer’s request to change their subscription plan, but the execution of that request should be controlled by predefined system logic.
• Optimizing Data Operations To Reduce Token Usage: Every unnecessary token increases costs and latency. Optimized prompt structures in our testing reduced token consumption by over 60% compared to naive implementations.
• Implementing Robust Validation Layers: No matter how refined prompts become, LLMs will occasionally generate unexpected outputs. Validation layers prevent incorrect AI-generated actions from affecting production systems.
The Future Of Enterprise AI
As generative AI adoption matures, enterprises shift their focus from raw capability to operational reliability. Organizations that succeed in this phase will integrate LLMs effectively while maintaining enterprise-grade performance, security, and compliance standards.
This shift enhances AI’s transformative potential. By building AI systems that consistently deliver value rather than simply impress in isolated cases, enterprises can confidently deploy conversational AI at scale into mission-critical processes.
Structured automation is the foundation of this evolution. It allows AI systems to behave like traditional software (reliable, predictable and maintainable) while benefiting from modern language models’ breakthrough capabilities.
Conclusion
The prompt-and-pray era of enterprise AI is ending. As organizations move from experimental implementations to production systems, the focus is shifting toward structured automation as the key to reliable, efficient and scalable AI systems.
This transition mirrors the natural progression of other technological revolutions: early excitement around raw capabilities followed by a maturation period focused on harnessing those capabilities reliably at scale.
For enterprises navigating this shift, the key is finding the right balance—using LLMs for what they do best while using structured workflows to ensure consistent execution. By separating conversational ability from business logic, organizations can realize the promise of AI without compromising the reliability that enterprise applications demand.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
Read More
Alan Nichol