Building an AI feature that works in a demo is easy. Building one that works reliably in production — across diverse inputs, at load, with observable failure modes — requires deliberate design. These patterns describe what separates prototype code from production AI systems.
Pattern 1: Keep the LLM Call Behind an API
The most important architectural decision is to never call an LLM directly from your frontend or your user-facing backend layer. Every LLM call should go through an execution layer that can be independently monitored, rate-limited, and updated.
In Waiflo's model, this is explicit: your application calls POST /api/exec/step with the step name and inputs. The execution layer resolves the prompt template, injects the output schema, calls the LLM, validates the response, and returns the result. Your application never touches the prompt text or the LLM API directly.
This indirection means you can update a prompt without deploying a new version of your application. It means you can switch LLM providers without changing integration code. And it means you have a single place to add logging, cost tracking, and error handling.
Pattern 2: Stream, Don't Block
LLM calls can take seconds for complex prompts. Blocking the user's experience for the full duration is unnecessary and damages perceived performance. Server-Sent Events (SSE) let you stream tokens to the client as they are generated:
const source = new EventSource('/api/exec/step');
source.onmessage = e => appendToken(e.data);
source.onerror = () => source.close();The key infrastructure requirement for SSE in production is a reverse proxy configured to not buffer responses. In Nginx, this means proxy_buffering off and appropriate X-Accel-Buffering: no headers. Without this, tokens are buffered and delivered all at once, defeating the purpose of streaming.
Pattern 3: Validate Outputs, Not Just Inputs
Input validation is standard practice. Output validation — asserting that the LLM's response conforms to a schema before using it — is less common but equally important.
A prompt that works correctly 95% of the time is not good enough for production. The 5% of malformed responses need to be caught, logged, and handled — either by retrying with a modified prompt, returning a degraded response, or surfacing a clear error. Without output validation, these cases silently corrupt downstream state.
Waiflo's ws_output_schema makes this validation automatic. The runtime injects the schema into the prompt (increasing compliance significantly) and validates the response before it is passed to the next step.
Pattern 4: Separate Prompt Logic from Application Logic
Embedding prompts inside application code creates an implicit coupling between two things that change at very different rates. Application code changes with feature releases. Prompt logic changes with model behavior, business requirements, and evaluation results — often more frequently, and often by different people.
When prompt logic lives in a separate file (a .waiflo.json workflow), it can be updated, reviewed, and deployed independently. A product manager can iterate on prompt language without touching application code. A data scientist can improve output schemas without a full release cycle.
Pattern 5: Build for Observability from Day One
AI features have failure modes that are different from traditional software bugs. The system does not crash — it returns a response that is subtly wrong in a way that is hard to detect automatically. Catching these failures requires deliberate investment in observability:
- Log every LLM call with its inputs, outputs, latency, and token cost. Not just errors — everything.
- Track output schema violations as a metric. A rising violation rate is an early warning that model behavior has shifted.
- Run evaluations in CI against a fixed set of test cases whenever a prompt changes. Regressions should be caught before they reach production.
- Version your prompts so you can correlate changes in output quality with specific prompt edits.
None of these are novel. They are the same observability practices that apply to any distributed system, adapted for the specific failure modes of LLM-based components. The teams that build reliable AI features treat them as software engineering problems with a probabilistic component — not as magic that either works or doesn't.