How to Choose a Partner for AI and Automation Work in South Africa: 12 Questions

Choosing a partner for automation and technology work is difficult for a simple reason: the market can make a prototype look impressive long before it can make a production workflow reliable.

A business therefore needs to evaluate more than model knowledge or presentation quality, and should establish whether AI is justified at all.

The partner should be able to connect:

business value → workflow design → data → architecture → governance → evaluation → adoption → operating performance

If one of those links is missing, the project can still produce a compelling demo while failing to produce a useful capability.

Start by defining what you are actually buying

“AI consulting” can mean very different things, and some of those services may be better framed as workflow, data or technology work:

• executive strategy;
• readiness assessment;
• use-case discovery;
• workflow automation;
• agent development;
• data engineering;
• model integration;
• custom product development;
• governance design;
• training and adoption;
• managed AI operations.

Do not issue a vague brief and then compare suppliers as if they offered the same thing.

Define the decision or outcome you need first.

For example:

We need to identify and validate the three highest-value AI automation opportunities across customer service and finance, with a prioritised roadmap and one production-ready pilot recommendation.

That is much easier to evaluate than “help us with AI transformation”, and it gives a supplier a fair test of whether AI is actually required.

Question 1: How do you identify valuable use cases?

A strong answer should begin with business processes, customer problems, operational constraints and measurable outcomes.

Be cautious if the supplier begins with a catalogue of technologies.

Look for a method that considers:

• business value;
• workflow frequency and volume;
• time or cost currently consumed;
• error rates;
• customer impact;
• data availability;
• integration feasibility;
• risk;
• organisational readiness.

The best AI opportunity is not necessarily the most sophisticated one.

Question 2: How do you decide whether AI is required at all?

This question is surprisingly revealing.

Some workflow problems are better solved with:

• clearer process design;
• ordinary automation;
• better search;
• data quality improvements;
• software integration;
• business rules;
• training.

A partner whose revenue depends on putting AI into every problem may have difficulty recommending a simpler solution.

Technology independence matters.

Question 3: Who will actually do the work?

Ask for the named delivery roles, not only the senior people in the pitch.

Depending on scope, you may need capability across:

• product or business analysis;
• workflow design;
• AI engineering;
• software engineering;
• data engineering;
• UX;
• security;
• governance and risk;
• change and adoption.

One person may cover several roles, especially in a specialist consultancy. The issue is whether the capability exists, not whether the org chart is large.

Question 4: How will you evaluate the system?

“Users liked the demo” is not an evaluation plan.

Ask how the partner will create representative test cases and define acceptable performance.

Useful evaluation dimensions can include:

• task success;
• factual accuracy;
• correct classification;
• extraction quality;
• policy adherence;
• correct tool use;
• correct escalation;
• latency;
• operating cost;
• human override rate.

For generative systems, the evaluation set should include difficult and ambiguous examples, not only happy-path prompts.

NIST's AI RMF and Generative AI Profile provide a useful reference point because they treat measurement and management of risk as lifecycle activities.

Question 5: What happens when the AI is uncertain or wrong?

A production partner should be able to discuss failure without becoming visibly offended by the concept.

Ask about:

• confidence or uncertainty handling;
• human escalation;
• fallbacks;
• tool-call failures;
• duplicate actions;
• malformed inputs;
• unavailable systems;
• unsafe or disallowed requests;
• rollback.

Reliable systems are designed around failure modes, not the assumption that the model behaves perfectly.

Question 6: How will data be protected?

The answer should cover more than “the vendor is secure”.

Ask:

• which data the solution requires;
• how data reaches the model;
• whether prompts or outputs are retained;
• what is logged;
• where data is processed;
• which people and systems can access it;
• how secrets and credentials are managed;
• how environments are separated;
• what happens when staff leave or access changes.

The details depend on the use case, but the supplier should be comfortable reasoning through them.

Question 7: How do you choose models and vendors?

Avoid suppliers whose architecture is secretly “whatever model we happen to resell”.

A credible decision can consider:

• task quality;
• latency;
• cost;
• context requirements;
• tool use;
• structured output reliability;
• data handling;
• regional availability;
• enterprise controls;
• vendor stability;
• portability.

The answer should also acknowledge that model quality and pricing change. Architecture should expect change rather than treating today's model leaderboard as a constitution.

Question 8: What should be deterministic versus AI-driven?

Strong AI systems often combine normal software with probabilistic models.

For example:

• permissions should be deterministic;
• transaction limits should be deterministic;
• required approval steps should be deterministic;
• classification may use AI;
• drafting may use AI;
• exception reasoning may use AI;
• final high-risk approval may remain human.

Ask the partner to explain that boundary.

Question 9: How will the system be monitored after launch?

Production AI needs operating telemetry.

Expect a view of:

• task completion;
• failures;
• escalations;
• human overrides;
• latency;
• cost;
• user feedback;
• model or prompt version;
• business outcomes.

If the proposal ends at deployment, the partner is ignoring the part where real users arrive and creatively discover all the assumptions nobody tested.

Question 10: What capability remains with us?

The organisation should not become completely dependent on a supplier to understand its own workflow.

Ask what will be documented and transferred:

• architecture;
• data flows;
• prompts and system instructions where relevant;
• evaluation cases;
• operating procedures;
• configuration;
• runbooks;
• access controls;
• model/vendor dependencies.

Ownership should be contractually clear as well.

Question 11: How do you prove business value?

Before launch, agree on a baseline.

Possible outcomes include:

• reduced handling time;
• lower cost per case;
• faster response;
• lower error rate;
• higher conversion;
• better employee throughput;
• reduced backlog;
• improved customer satisfaction;
• higher compliance with process standards.

Then compare the AI-enabled workflow with the baseline.

A model benchmark is not a business case.

Question 12: When would you recommend stopping?

A good partner should be willing to stop a use case when:

• value is too small;
• data is not fit for purpose;
• integration cost overwhelms the benefit;
• risk cannot be controlled sensibly;
• users do not adopt the workflow;
• ordinary automation performs better;
• operating cost is unattractive.

A supplier that cannot describe a stop condition is selling delivery volume rather than decision quality.

Red flags in an AI proposal

Be cautious when you see:

• guaranteed percentage savings before discovery;
• no baseline measurement;
• no named evaluation method;
• no discussion of human oversight;
• vague data handling;
• a proprietary “AI platform” that cannot be clearly explained;
• model choice treated as the main strategy;
• no integration analysis;
• no operating support model;
• a pilot designed only around curated examples;
• no exit or portability plan;
• no clear ownership of generated IP, code or configuration.

Score the partner across six capabilities

A simple procurement matrix can score 1 to 5 across:

Weight the categories according to the engagement.

A strategic assessment may weight commercial framing more heavily. A production agent may place more weight on engineering, evaluation and governance.

Proof should match the kind of work

Ask for evidence relevant to your scope.

For a production build, a thought-leadership article is not equivalent to shipped software.

For an executive strategy engagement, a coding portfolio is not proof of organisational transformation capability.

Useful evidence can include:

• architecture examples;
• anonymised workflow diagrams;
• evaluation methodology;
• production case studies;
• references;
• demonstration of observability and controls;
• sample deliverables;
• clear explanation of trade-offs from previous work.

Contract for outcomes and boundaries

The agreement should make clear:

• scope;
• assumptions;
• dependencies;
• deliverables;
• client responsibilities;
• data handling;
• intellectual property;
• third-party service costs;
• acceptance criteria;
• change control;
• support after launch;
• termination and handover.

AI does not repeal procurement discipline.

A sensible appointment sequence

For a material automation or AI programme, consider separating decision stages.

Stage 1: assessment

Identify opportunities, readiness, risks and priorities.

Stage 2: proof

Validate the most important assumptions with controlled evidence.

Stage 3: production pilot

Integrate one use case into the real workflow with monitoring and governance.

Stage 4: scale

Expand only after evidence supports it.

This gives the organisation several opportunities to stop, adjust or change suppliers before sunk cost becomes strategy.

Further reading

• NIST AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework
• NIST Generative AI Profile: https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence

Abrasim's Automation & Technology engagements begin with opportunity and readiness because the highest-value decision is often not which AI to buy. It is which workflow deserves investment, whether a simpler intervention is better, what evidence would justify scaling, and what controls make the chosen approach safe enough to operate.