An AI development project isn't complete when version one works. Data shifts, APIs evolve, and tools can quietly degrade. Factoring in maintenance from the design phase avoids single-person dependencies, reveals operating costs, and ensures long-term reliability.
An artificial intelligence development project is not finished when its first version is up and running. Data changes, APIs evolve, and users introduce situations that were absent from initial testing. A solution can continue to respond while becoming less reliable, slower, or excessively costly.
For an SME or a scale-up, planning for maintenance right from the design stage avoids relying on a single individual and discovering operating expenses only after launch. The goal is not to anticipate everything, but to be able to detect an issue, understand its root cause, and fix it without rebuilding the entire solution. This is prepared across the architecture, tests, budget, and the service provider’s commitments.
AI maintenance goes well beyond fixing bugs. It covers software components, data used, models, and connections to your business tools. This is about maintaining the AI solution itself, not predicting industrial equipment breakdowns.
Three complementary forms of maintenance exist. Corrective maintenance addresses malfunctions. Preventive maintenance mitigates risks, particularly through security updates and regular health checks. Adaptive maintenance supports changes in vendors, document formats, or internal workflows.
One specific feature of AI requires extra attention: the absence of a technical crash does not prove service quality. An assistant can reply promptly with outdated information. A classification system can run without any software errors while misrouting an increasing number of inquiries to the wrong department.
The NIST AI Risk Management Framework integrates risk measurement and management throughout the entire lifecycle. For businesses, this means controls must outlive initial delivery.
The first decision is to describe what "functioning properly" actually means. High uptime is not enough if outputs are unusable. Conversely, an outstanding answer delivered too late may create zero value.
In AI development, maintenance criteria must stem from business usage. For a tool drafting customer support responses, you can distinguish the following dimensions:
Dimension | Potential Metric | Decision to Anticipate |
|---|---|---|
Quality | Share of suggestions accepted without major edits | Re-evaluate instructions, sources, or the model |
Latency | Response time, including long-tail queries | Adjust resources or cap certain processes |
Availability | Share of requests successfully handled technically | Trigger fallback workflow |
Cost | Expenditure per validated output | Revisit routing or handled volume |
Security | Results of access and privacy checks | Restrict affected feature and remediate |
Thresholds must match your business operations, not a generic promise of "accuracy." An error in a reviewed draft does not carry the same consequences as an automated action in billing software.
Specify expected service hours as well. A support response within four business hours is not a guaranteed resolution within four hours. These trade-offs extend the scoping of an AI development project, with one extra question: how will these commitments be verified post-launch?
Business rules, prompts sent to the model, connectors, and the UI should not form a monolithic block that is impossible to modify piece by piece. If an instruction needs to evolve, you must be able to test it without unnecessarily redeploying the entire application.
Version your code, prompts, configurations, and test datasets. For an external model, record the exact model identifier and parameters used. This traceability helps compare two versions, even though it does not guarantee strictly identical outputs on every run.
Maintainable AI development also links an output back to its sources. For document-augmented generation, known as RAG, this requires tracking retrieved documents and their version. Without this information, distinguishing a flawed document from flawed reasoning becomes difficult.
A provider can alter its API, deprecate a model, or change pricing. An internal connector can also break following an update to your CRM. Map out these dependencies along with their rate limits and access mechanisms.
Graceful degradation should be explicit: handoff to an operator, queueing, or disabling a specific feature. Silently substituting one model for another is no guarantee of continuity, as its outputs may diverge.
Reversibility does not mean switching providers will be free. It means knowing which components to adapt, which tests to rerun, and what data to retrieve. Designing modular abstraction interfaces reduces effort without eliminating the need for validation.
Tests must reflect situations actually encountered by the business: routine queries, ambiguous phrasing, incomplete documents, and edge cases that should be rejected. A dataset made up solely of straightforward examples provides a misleading view of quality.
For each case, define the expected outcome or acceptance criteria. In invoice extraction, specific fields can be verified precisely. For summaries, you should instead evaluate fidelity to source material, omissions, and the presence of unsubstantiated claims.
The golden reference set should be versioned and enriched with real-world incidents, rather than replaced with every update. Keeping a stable baseline makes regression detection possible. If custom fine-tuning is planned, keep training data strictly separate from evaluation data.
This preparation makes AI development less dependent on a successful one-off demo: the release decision relies on repeatable validations.
Traditional tests remain essential for verifying access permissions, calculations, data formats, and inter-application communication. AI evaluations complement them; they do not replace them.
For generative responses, word-for-word matching is often unsuitable. Instead, check meaningful properties: are dollar amounts accurate, sources reachable, and constraints respected? An auxiliary model can help evaluate outputs, but its judgments must be audited against human samples and cannot serve as the sole validation for sensitive cases.
Include prompt injection and abuse tests whenever the tool ingests external content. A document containing malicious instructions must not bypass business rules or expose restricted information. For actions with real-world consequences, implement human-in-the-loop approvals and restricted permissions.
Observability means having the necessary information to understand what is happening in production. It must be designed during AI development, because useful traces are difficult to reconstruct after an incident has occurred.
For each transaction, a unique identifier can correlate status, duration, model version, and any errors. Add operational business metrics: major manual corrections, drop-offs, and requests taken over by staff. An increase in technically successful responses can mask a drop in their real-world utility.
Do not log everything by default. Prompts and responses can contain personal or proprietary data. Define necessary log data, authorized access, and retention windows. The CNIL resources on artificial intelligence provide valuable guidance on integrating data protection principles by design.
For knowledge-based systems, also plan for source deletion or updates: copies, search indexes, and caches must be handled in compliance with applicable rules. A file deleted from a shared drive must not remain queryable indefinitely through a lingering vector index.
Finally, every alert must trigger a defined action. A dashboard that nobody looks at is not a maintenance strategy.
Adjusting a prompt can alter service behavior as drastically as a code change. Updates therefore warrant a structured validation pipeline, even when they seem minor.
Start by replaying tests in a staging environment. If results are acceptable, release the new version to a limited scope with risk-appropriate monitoring. Compare quality, latency, and costs before rolling it out more widely. For high-stakes workflows, a comprehensive prior sign-off may remain mandatory.
AI development must also support rolling back to a known stable version. Ensure the previous configuration remains compatible with existing data, indexes, and dependencies. A rollback button is ineffective if the earlier model is no longer hosted or supported.
Lastly, distinguish service restoration from consequence remediation. Rolling back to an older version does not retract an email already sent or reverse an accounting ledger entry already posted. Automated operations require their own audit trails, idempotency safeguards, and, whenever possible, compensation routines.
The recurring budget is not limited to model API tokens. It can encompass hosting, storage, vector search, observability tooling, human review, and maintenance interventions. The exact breakdown depends on the chosen architecture.
A practical metric is the cost per validated result. It divides selected operating costs by the volume of genuinely usable results delivered over the same timeframe. This prevents confusing volume generated with actual value produced.
As a purely illustrative example: a solution costs €1,200 over a month, including human oversight and maintenance, excluding initial capital expenditure. It delivers 3,000 validated outputs. Its cost stands at €0.40 per validated result. This figure does not prove profitability on its own: the value delivered must still be weighed against the prior manual process.
From the AI development phase onwards, request volume projections and traffic spike scenarios. Also set out consumption caps, billing alerts, and what happens when quotas are exceeded. A hard ceiling that abruptly shuts down a critical feature must be paired with a contingency fallback.
Not all maintenance tasks follow the same schedule. Security patches require an established policy. Dependency updates must be tracked. Answer quality, meanwhile, should be reassessed according to operational criticality and observed performance signals.
A drop in performance does not automatically warrant retraining. It can stem from outdated reference files, an expired credential, shifting business terminology, or a degraded search retriever. Diagnosis must precede the remediation choice.
Data drift refers to changes in input distributions compared to those used during design or evaluation. It represents a signal to investigate, not automatic proof that the service has failed.
Finally, budget the time needed for these reviews. For smaller organizations, a few targeted, consistently executed checks are worth far more than an overly ambitious framework lacking ownership and funding.
Maintainability is also verified at delivery. A solution that is operational but impossible for others to take over remains fragile. AI development must produce artifacts usable by someone other than the original author.
Require at least the following assets:
Access and ownership: code repository, service accounts, IP rights over deliverables, and administration protocols.
Operational documentation: architecture blueprints, dependencies, configuration files, and restart or disaster recovery procedures.
Quality baselines: reference evaluation sets, acceptance criteria, and recent benchmark results.
Incident runbooks: triage steps, feature kill switches, fallback paths, and rollback guides.
Support framework: scope covered, operating hours, SLA response times, and change-request workflows.
Exit terms: data export formats, knowledge transfer sessions, and decommissioning of obsolete credentials.
Documentation must be put to the test. Can an authorized engineer restore the application or rotate an API secret purely by following the runbook? If they must ask for verbal guidance at each step, the handover is not complete.
Also draw a clear line between fixing a defect and adding a feature: this distinction prevents many contractual disputes. To then allocate business and technical responsibilities, our article on the team needed to maintain an AI breaks down the roles to cover, without assuming every single one requires a dedicated new hire.
Should a maintenance agreement be planned from day one? At a minimum, you must define who intervenes, over what scope, and with what commitments. A standalone contract is not always required, but responsibilities, credentials, and budget must be clarified before going live.
Does an AI powered by a third-party API need maintenance? Yes. The provider manages their platform, not your connectors, source files, access controls, or the business relevance of generated answers. Relying on an API shifts part of the responsibility without eliminating maintenance for your solution.
Does the model need regular retraining? No, not systematically. Many solutions rely on pre-trained foundation models without custom fine-tuning. Maintenance then focuses on prompt engineering, reference data, integrations, or model selection. Retraining should address a diagnosed issue and be thoroughly validated prior to rollout.
Can we start with a lightweight setup? Yes. For a low-risk initial AI project, a focused golden dataset, selective logging, cost monitoring, and a basic fallback procedure form a proportionate foundation. Monitoring can then scale as usage and operational stakes expand.
Before signing off on a proposal, ask how degradations will be detected, how updates will be verified, and how the solution can be maintained independently. The answers should translate into concrete deliverables and clear accountability, not just good intentions.
Impulse Lab supports companies through AI opportunity audits, custom solution development, business tool integrations, and adoption training. To embed maintainability into your scoping phase from day one, present your project to Impulse Lab detailing your use cases, constraints, and operational capabilities.
Our team of experts will respond promptly to understand your needs and recommend the best solution.
Got questions? We've got answers.
Leonard
Co-founder
