I sell AI engineering services from Ahmedabad, so read this with that bias in mind; it is also the checklist I would want on my desk if I were hiring. The relevant facts about my own practice: 17+ shipped systems, 60+ merged PRs across 24 external repos (OWASP, Microsoft AI-Engineering-Coach), 4 hackathon finals, and fixed-scope delivery with written scope, fixed INR price, code in your repo, and 40/60 payments. Every case study on this site carries a limitations section and a how-it-is-tested section. This essay is the buyer-side version of that honesty: how to screen an AI engineer without getting fooled by polish.

The wrong screen

Portfolio screenshots prove taste, not delivery. Dashboards with gradients and a plausible chart prove someone can arrange divs. Model names dropped in chat ("we use the latest frontier model") prove access to marketing, not judgment about when a smaller model wins. Certifications prove attendance; nobody has ever been saved by a badge. None of these show that the candidate can take a messy business problem, ship a system that survives contact with real users and real data, and then tell you the truth about what it cannot do.

The deeper problem with the wrong screen is that it selects for confidence. The candidates who perform best in a vibe interview are often the worst to hire, because an AI system fails quietly and a confident talker is never quiet. You are not hiring for enthusiasm; you are hiring for the engineer who will tell you your data is not ready before you pay to find out.

The right screen: five questions

Ask these five. Then listen for shape, not for story.

  1. Walk me through a production incident you caused and what you changed afterwards. Good answers are brief, concrete and owned: the system, the failure, the blast radius, the fix, the guardrail added afterwards. Red flags: a hero story where they single-handedly saved the company, or "I have never caused an incident in production", which means they have and did not notice or did not say.
  2. Show me the eval set for a system you shipped, and what fails below the floor. Good answers: they can describe or show the eval set, the pass threshold, the failure modes below the floor, and what they did about them. Red flags: no evals at all, or evals that only cover the happy path, or "we tested it manually and it worked well".
  3. What did your last system get wrong in production? Good answers: a specific, owned miss (a failure class, a drift, a bad assumption about inputs or users) and the change it forced. Red flags: blame the model, blame the data, blame the client, blame the requirements; never an owned miss.
  4. How do you verify an agent's work without trusting its summary? Good answers: re-derive verification from artifacts, re-run tests, re-parse outputs, diff structure, bound the retries, keep a human review gate. Red flags: "the agent said it passed". If they cannot explain verification independent of the agent's own report, they will ship you plausible output and call it done.
  5. What would you refuse to build for me, and why? Good answers: a real refusal with a reason (not measurable, not viable with the data available, not honest to ship, not worth the cost). Red flags: "nothing, whatever you need, I can build it", which is how you get a six-month boondoggle.

Good answers are brief, concrete and owned. Red flags are hero stories, missing evals, blame-the-model deflection, and an inability to name a single limitation of their own work. Notice that none of these questions can be answered by reading a landing page; they can only be answered by having done the work.

The artifact test

Before any contract, ask the finalist for a written one-page approach to your actual problem: what they would build, in what order, what they would measure, what they would not do yet, and what could make the whole thing fail. It costs them an hour; it reveals everything. You learn how they think under no pressure, whether they ask the right questions back, whether they understand the problem or merely pattern-match it to a previous project, and whether they can write plainly, which is the entire medium of every scope doc, status update and handoff note you will ever receive from them.

The artifact test also filters for the candidate who will tell you the truth in week one instead of week six. Vague artifacts produce vague projects. Sharp artifacts with explicit exclusions are the closest thing to a free sample of collaboration you will ever get.

Paid trial beats interview theater

A fixed-scope paid pilot is the highest-signal evaluation available, and it beats any interview format. The bands in my own practice: ₹15,000-35,000 for a workflow automation pilot, ₹25,000 and above for a product slice. Real work, real repo, real review, typically one to two weeks. You see commit quality, communication cadence, estimation honesty, and whether they finish what they scope. An interview shows you how someone talks; a pilot shows you how someone works, and only one of those ships your product.

Structure the pilot like a real engagement in miniature: written scope, definition of done, one revision round, fixed price. If they resist a paid pilot on principle, that is data. If they ask for it, that is data too.

Structure and incentives

Whatever the arrangement, the mechanics should look like this. A fixed quote in writing. 40% to start, 60% on delivery. Two revision rounds included. A definition of done written before any work begins. You own the code and the cloud account from day one, not after final payment. A handoff doc at the end covering architecture, env vars, failure modes, and how to operate the thing without them.

For reference, the commercial terms I work under: fixed-scope delivery from ₹25,000 for a 2-4 week engagement, ₹1,500 per hour for advisory, retainers of ₹30,000-75,000 per month for ongoing work, 40/60 payments, a free 15-minute intro call, and a fixed quote in writing. These terms exist because they force clarity on both sides. You cannot half-specify your needs and blame the delivery gap on the vendor, and the vendor cannot gold-plate the scope and bill for it. Friction at the contract stage is the cheapest place to find out whether both parties are serious.

Red flags in contracts and promises

  • Open-ended hourly with no cap and no definition of done. The incentive is inverted: the longer it takes, the more they earn. You need a cap, a scope, and a finish line.
  • "The AI will handle it" with no description of how the AI's output is verified, evaluated or reviewed by a human. Unverified AI output is a liability dressed as leverage.
  • No handoff doc, no test or eval story, no access for you to your own repo and cloud account. If you cannot operate the system without the vendor, you did not buy a system; you rented a dependency on a person.
  • Reluctance to put scope, price and timeline in writing. Verbal agreements dissolve exactly when they matter. Anything not written down will be renegotiated under stress, usually against you.
  • Promised outcomes without named assumptions. "Accuracy will be 95 percent" or "we will automate all of it", with no stated data assumptions, no eval set, no floor below which they will tell you to stop. Guaranteed outcomes without visible assumptions are marketing, not engineering.

Putting it together

The shape of a good hire process is short. Screen out the wrong signals. Ask the five questions. Read the artifact. Run a paid pilot. Sign a written scope with 40/60 and a definition of done. Then hold them to the same standard every case study on this site is held to: a limitations section and a how-it-is-tested section, written by the person who shipped the work. The honest ones will not blink at that request; the rest will negotiate, deflect, or disappear.

Even if you don't hire me, use this checklist; if you do, get in touch. And ask any candidate you shortlist to produce a limitations section and a how-it-is-tested section for their last project. The quality of that one-page answer will tell you more than any interview you have ever run.