How to Evaluate an AI Implementation Partner: 12 Questions to Ask

The Twelve Questions Replace the Slide Deck

Standard partner evaluation processes for AI implementation work measure capabilities, case studies, and pricing. The processes catch the obvious mistakes and miss the expensive ones. The expensive mistakes show up four to six months into an engagement, after the contract is signed and the pitch team has moved on.

ISG's 2024 services market analysis tracks the renewal pattern: roughly 47 percent of AI engagements renew at lower scope after year one (ISG, "2024 Cloud Services Market Trends"). The contraction is not because partners are bad. The contraction is because the evaluation missed the questions that surface real fit.

Twelve diagnostic questions, asked before contract, separate partners who deliver from partners who pitch well and stall in execution. Each question is uncomfortable. Each one produces information the standard RFP does not.

The Twelve Questions

The questions cluster into four areas: team, delivery practices, commercial alignment, and exit posture. Each area has three questions.

Team questions. Q1: Who specifically is on my account, named with title and tenure? Q2: What is your senior-to-junior ratio on engagements like mine? Q3: Can I meet the actual delivery lead, not just the pitch team, before signing?

Delivery practices questions. Q4: Walk me through a project where you failed and what changed afterward. Q5: How do you handle scope changes, and what is the typical scope-change ratio across your AI engagements? Q6: What does your incident response look like when something breaks in production at 2am?

Commercial alignment questions. Q7: How is your team compensated, and how does that compensation align with my project outcomes? Q8: What pricing structure do you recommend for my specific workload, and why? Q9: What is your standard exit clause structure if the engagement is not working?

Exit posture questions. Q10: What knowledge transfer happens at the end of the engagement, and what artifacts do I keep? Q11: What is your typical engagement length, and what happens at end of contract? Q12: Have you had clients move to internal capability rather than renewing with you, and how did you handle it?

A partner that answers all twelve credibly is rare and worth shortlisting. A partner that answers six or seven credibly is a candidate. A partner that bristles at the questions is signaling something useful.

What Each Area Surfaces

The team questions surface the gap between the pitch team and the delivery team. Pitch teams typically include partners and senior architects who participate in two or three kickoff meetings and then transition off. Delivery teams are the people who actually do the work. The gap between these two populations is the leading cause of engagement disappointment.

The delivery practices questions surface operational maturity. Partners that have run real production AI work have failed at things, learned, and adjusted. Partners that have not have not. The failure question is the most diagnostic because it requires honesty about something most pitch teams want to obscure.

The commercial alignment questions surface incentive alignment. Time-and-materials engagements incentivize hour expansion. Fixed-price engagements incentivize scope minimization. Outcome-based engagements incentivize partnership. Knowing the partner's compensation structure tells you what behavior to expect.

The exit posture questions surface long-term intent. Partners who build engagements that customers cannot leave have different intent than partners who build engagements that leave customers self-sufficient. Both are legitimate businesses. They produce different engagement outcomes.

The Questions Beyond the Twelve

Two additional questions, when warranted, add useful signal.

If the partner has technical depth in a specific category that matters to you, ask: walk me through your last three production AI architectures in this category, with specifics. The answers reveal whether the depth is real or marketed.

If regulatory exposure is meaningful, ask: how have you supported clients through audits, examinations, or regulatory inquiries related to AI? The answers reveal whether the partner has the institutional muscle for the regulatory side or whether they have been lucky enough not to need it.

Both questions are situational rather than universal. They belong in the evaluation when the situation justifies them.

The Reference Discipline

Reference calls usually disappoint. The pattern is: the partner provides three or four favorable references, the references give politely positive feedback, the evaluating team concludes the partner is acceptable, the engagement underdelivers.

The reference discipline that produces useful signal has three properties.

Talk to references whose engagements match yours in size, industry, and timeframe. Not the largest customers. Not the most successful. Customers whose engagement profile is similar to yours.

Talk to practitioners, not just executives. Executives describe strategic value. Practitioners describe operational reality. Both are useful; the second is more diagnostic.

Ask about expansion, contraction, or stability of the engagement over time. Partners that expand engagement scope at reference customers are delivering. Partners that retain stable scope are acceptable. Partners that contract scope at reference customers are signaling something the executive description may not capture.

The reference discipline matters because it is the only data point the partner does not directly control.

What This Looks Like in Practice

A four to eight week evaluation process for an AI implementation engagement above $500K annual value typically includes: a written RFP response from each candidate, in-person presentation with the actual delivery team for shortlisted candidates, the twelve questions answered in writing and in conversation, reference calls following the three properties above, and a working session with the delivery team on a real engagement scenario.

The total time investment is meaningful and pays back in engagement quality. Teams that compress the evaluation below four weeks consistently produce worse selection outcomes.

What Logiciel Does Here

Logiciel works with enterprises selecting AI implementation partners, including the case where prior selections have not delivered. The work is typically structured around the twelve-question framework adapted to the specific engagement profile.

The Evaluate AI Implementation Partner framework covers the eight-item structural filter that complements the diagnostic questions. The AI Implementation Partner Checklist framework covers the engineering quality bar that good partners meet.

A 30-minute working session is enough to walk through how to apply the twelve questions to your specific situation.

Frequently Asked Questions