In short
Evaluate an engineering vendor by verifying claims rather than reading proposals. Write a one page brief first, because vague briefs produce quotes that differ by five times and mean nothing. Then interrogate the two most relevant case studies, call references with questions that expose the worst month, and interview the engineers who would actually do the work. Compare itemized scope and exclusion lists, never bottom lines, because a cheap quote is usually an incomplete quote. Argue four contract clauses: intellectual property on payment, repositories in your accounts from day one, a defined exit, and termination for convenience. Then run a paid pilot of two to four weeks, because the pilot is where the decision is genuinely made and the proposal is not.
Choosing an engineering vendor is a supplier decision with the failure modes of a hiring decision. The proposal tells you almost nothing, the real signal lives in how the vendor works rather than what they claim, and the cost of a wrong choice compounds quietly for years through rework, staff turnover and a codebase nobody wants to inherit.
Directory listings of top companies will not resolve it either, because the right vendor depends on your project shape, your budget structure, and how much engineering judgment you already have in house. Two buyers with identical budgets can correctly choose different vendors.
This is the evaluation framework we would want a client to use on us. It covers the brief that makes quotes comparable, how to verify what vendors claim, how to read quotes that refuse to be comparable, the four contract clauses worth arguing over, how to design a pilot that produces evidence, and the first month signals that predict year two. We are a vendor writing this, so read it adversarially. Every test below is one we are prepared to pass, and how we actually work is on our custom software development page.
Key takeaways
- Write the one page brief before contacting anyone. It is both the input that makes quotes comparable and your first filter, because a vendor who interrogates the brief is demonstrating the behavior you are trying to buy.
- Insist on meeting the engineers who would do the work, and name them in the contract. Senior people in the pitch and juniors in the delivery is the most common failure in this industry.
- Compare exclusion lists, not totals. The informative question is what each vendor did not price.
- A rate is not a cost. Total cost is rate times duration times rework, and the rework coefficient is invisible on a rate card.
- Four clauses predict the ending: intellectual property assigned on payment, repositories in your accounts from day one, a documented exit, and termination for convenience with reasonable notice.
- Run a paid pilot sized so you can walk away from it. You are buying evidence about estimation accuracy, communication under pressure and code quality, not a prototype.
- The strongest single predictor is how a vendor behaves when they are wrong. Test it deliberately before you sign, because you will be relying on it for years.
Before you evaluate anyone, define what you are buying
Most bad vendor choices are made before the first call, at the point where the buyer has not yet decided what they are buying. A vendor building your core product for the next five years is a fundamentally different purchase from a team augmenting yours for two quarters, which is different again from a fixed scope build of a well specified internal tool. Each of those purchases has a different right vendor, a different right contract, and a different definition of a good price.
Write one page before contacting anyone. It needs the business outcome you are buying, an honest description of what exists today, what done looks like in twelve months, the budget range, who on your side will make product decisions weekly, and what happens after delivery, meaning who maintains the thing. Vendors quote against whatever you hand them, so a vague brief reliably produces quotes that differ by several times and cannot be compared.
That page is also your first evaluation instrument, and it is the cheapest one you will ever use. Send it out and watch what comes back. A vendor who immediately asks sharp questions about the twelve month definition, or who challenges the budget shape against the outcome, is demonstrating precisely the behavior you are trying to buy. A vendor who returns a polished proposal without questioning anything has told you they will build what you asked for rather than what you need.
Decide honestly, in writing, how much engineering judgment you hold in house. If the answer is not much, weight the evaluation toward vendors who show their reasoning and explain trade offs in business terms, because you will be depending on their judgment and not merely their labor. If you have a strong technical leader already, you can weight toward execution capacity and pay less for advisory depth you will not use.
The one page brief, in six lines
- The business outcome, not the feature listState what changes in the business when this works: a cost that falls, a process that stops needing people, a revenue line that becomes possible. Feature lists invite vendors to price the features and ignore whether they produce the outcome.
- What exists today, honestlyCurrent systems, their age, what integrates with what, and which parts nobody understands anymore. Undisclosed legacy is the most common cause of an estimate collapsing in month three.
- Done in twelve monthsOne paragraph describing the state of the product a year out. This is what separates vendors who plan a trajectory from vendors who plan a delivery.
- The budget rangeA range, not a number, and a real one. Withholding it does not improve your negotiating position, it produces quotes aimed at the wrong scope that you then have to discard.
- Who decides, weeklyName the person with authority to make product decisions in a weekly meeting. Projects without this person stall regardless of vendor quality, and good vendors will ask for the name before they quote.
- Who owns it after deliveryYour team, the vendor on a retainer, or nobody decided yet. This changes the architecture, the documentation standard, and the honest price.
Everything on this list is something a vendor will otherwise assume, and every assumption they make is a difference between their quote and the next one. Writing it down is the single highest leverage hour in the entire process.
Verify claims: portfolios, references, and the engineers themselves
Every vendor claims relevant experience. Your job is not to collect those claims but to distinguish verified fact from marketing, and the distinction is usually reachable in a single well structured conversation. The tests below are cheap, they take a few hours in total, and they are the difference between buying a track record and buying a slide deck.
Start with the case studies, and go narrow rather than broad. Pick the two most relevant projects and ask what the team size and duration were, what the vendor built versus inherited, what they would do differently, and whether the product is still live today. A team that built the thing answers fluently, remembers the specific decisions, and volunteers the failures without being pushed. A reseller of other people's work goes vague at exactly the third question.
Then take references seriously enough to ask uncomfortable questions. The useful ones are not about satisfaction. Ask how the vendor handled the worst month of the project, how accurate the estimates turned out to be, who the actual engineers were and whether they changed, and whether the referee would re engage at the same price. That last question is the most honest satisfaction metric that exists, because it prices the experience rather than describing it.
Insist on meeting the engineers who would work on your project, and run it as a hiring interview rather than a sales call. Ask them to explain a hard decision from a previous project, and listen for whether they describe trade offs or just outcomes. Bait and switch staffing, meaning senior people in the pitch and juniors in the delivery, is the most common structural failure in this industry, and the remedy is simple and rarely refused by honest vendors: name the team in the contract, with a clause requiring notice and equivalent replacement.
- Interrogate two case studies, not ten. Depth beats breadth. Team size, duration, built versus inherited, what they would change, and whether it is still running. Fluency and volunteered failures are the signal.
- Ask references about the worst month. Every project has one. How the vendor behaved during it predicts how they will behave during yours, and referees answer this question far more candidly than a general satisfaction question.
- Interview the engineers, not the account manager. Treat it as hiring. Then name those engineers in the contract with a notice and equivalent replacement clause.
- Read their written artifacts. Ask for a real technical proposal, an architecture note and a weekly status report, redacted as needed. You are buying years of written communication, so look at a sample of it before you commit.
- Use review platforms for pattern detection only. Consistent themes repeating across many reviews are signal, positive or negative. Individual reviews, in either direction, are noise.
Questions that produce evidence, and questions that produce marketing
Do this
- "Who wrote the authentication layer on that project, and can I speak with them?"Names a component and a person. Answerable in one sentence by the team that built it, and awkward for anyone who did not.
- "What was your first estimate, and what was the actual?"Asks for two numbers that either exist or do not. Vendors with estimation discipline know both figures and can explain the gap.
- "What would you cut if my budget dropped by a third?"Tests whether the vendor understands which parts of the plan carry the value, and reveals whether the proposal has padding they already know about.
- "What is the thing you are least sure of in this plan?"A vendor with no uncertainty is either not looking or not telling you. The quality of this answer predicts how risks will be reported later.
Not this
- "Do you have experience in our industry?"The answer is always yes, and it costs the vendor nothing. Industry familiarity matters, but it has to be verified through a specific project rather than asserted.
- "Are you agile?"Universally answered yes, and the word has been drained of content. Ask instead what happens in their process when a sprint commitment is missed.
- "How many developers do you have?"Headcount is not capacity and certainly not quality. The relevant number is who is available for your project, by name, and when.
- "Can you do it cheaper?"Asked before scope is settled, this invites the vendor to quietly remove things you had assumed were included, which is how incomplete quotes are born.
The pattern is that specific, falsifiable questions about a named project cannot be answered from a script, while general questions about capability invite the answer the vendor has already rehearsed.
Vendor vocabulary, translated
- Discovery phase
- A paid period of analysis before the main estimate. Legitimate and usually valuable, but confirm in advance what artifacts you receive and whether they are yours to take to another vendor if you do not proceed.
- Ballpark estimate
- A figure given before scope exists. Useful for filtering out an order of magnitude mismatch, and useless for planning. Never treat a ballpark as a commitment, in either direction.
- Change request
- The mechanism by which a fixed price contract handles anything the specification did not cover. Ask to see the process and the pricing basis before signing, not after.
- Blended rate
- One hourly figure averaged across seniority levels. It hides the actual mix, so ask for the composition, because a blended rate can rise in real terms while looking flat if the mix shifts junior.
- Bench
- Engineers available and not currently assigned. A vendor with a genuine bench can start quickly. A vendor who promises a start date with no bench is planning to hire against your project, which is a schedule risk you should be told about.
- Handover
- The documented transfer of a system to you or another vendor. If it is not defined in the contract with concrete deliverables, it is a promise rather than an obligation.
Not all of these are deceptive. Most are legitimate industry terms that carry a specific commercial meaning a buyer should be aware of before nodding along in a meeting.
Compare quotes on scope and exclusions, not on the bottom line
Quotes for the same brief routinely differ by multiples, and the bottom line is the least informative number on the page. The differences almost never reflect efficiency. They reflect what each vendor decided to price. One included quality assurance, deployment engineering, project management and a contingency. Another priced feature development alone and will invoice everything else as changes, at a moment when you have no leverage left because the project is half built.
So force comparability rather than hoping for it. Send every finalist the same one page brief. Require itemization by area: design, engineering split by component, quality assurance, project management, infrastructure setup, and project contingency stated as a line rather than hidden in the estimates. Then require an explicit exclusion list and a statement of the assumptions the price depends on. Compare the exclusion lists first, because that is where the real difference between two proposals lives.
Ask every vendor the same closing question: what would make this estimate wrong? The quality of that answer measures the honesty of the estimate more reliably than any reference call. A vendor who names two or three specific risks, with the mechanism by which each would move the number, is estimating. A vendor who says the estimate is firm has either padded it heavily or has not thought about it.
On rates, regional differences are real and legitimate. Engineering talent in Vietnam, to take the case we know from the inside, offers a genuine cost to seniority advantage, and that is a large part of why offshore engagement models exist at all, ours included through offshore development. But a rate is not a cost. Total cost is rate times duration times rework, and a team that costs thirty percent less per hour while running fifty percent slower with double the defect rate is not cheaper by any measure that reaches your budget. Reference checks and pilots, not rate cards, are what tell you the rework coefficient.
The same brief, three quotes, and what the gap actually is
| Line item | Quote A (lowest) | Quote B (middle) | Quote C (highest) |
|---|---|---|---|
| Feature engineering | Priced | Priced | Priced |
| Quality assurance | Excluded, developer testing only | Priced, part time tester | Priced, dedicated tester with a written plan |
| Deployment and environments | Excluded, assumes you have this | Basic setup priced | Full pipeline, staging and rollback priced |
| Project management | Excluded, engineer coordinates | Priced at ten percent | Priced at fifteen percent with a named manager |
| Contingency | None stated | Ten percent, stated | Twenty percent, stated with the trigger conditions |
| Documentation and handover | Not mentioned | Mentioned, not itemized | Itemized deliverables listed in the contract |
| Stated assumptions | None | Three | Nine, including two that contradict your brief |
| What the difference really is | A partial scope at a full price | A complete scope with thin margins for error | A complete scope that has been thought about |
An illustrative comparison built to show the mechanism rather than to report real proposals. The three columns describe the same requested scope. Read down the rows and the apparent price difference resolves into a difference in what was priced.
What to require in every quote so the quotes can be compared
Itemization
- Engineering
- Split by component or workstream, with the seniority mix stated for each rather than blended into one figure.
- Non engineering
- Design, quality assurance, project management, infrastructure and any third party license costs, each as its own line.
- Contingency
- A stated line with a stated percentage, plus the conditions under which it would be consumed.
Boundaries
- Exclusions
- An explicit list of what is not in the price. This is the most important section of the response.
- Assumptions
- Every condition the price depends on, including things they expect from your team and your existing systems.
- Estimate basis
- How the numbers were produced: comparison to a previous project, decomposition into tasks, or judgment. All three are acceptable. Not knowing is not.
Delivery terms
- Named team
- The people who would do the work, with their availability dates and the replacement clause if someone leaves.
- Cadence
- What you receive weekly and monthly, and what a demo consists of.
- Change process
- How a scope change is priced and approved, with a worked example rather than a policy statement.
Send this as the required response format with the brief. Vendors who cannot or will not produce it in this shape have told you something useful about how they will report progress later.
Pick the engagement model for your uncertainty, not your comfort
The three standard engagement models allocate risk differently, and the right one is determined by how well specified your project genuinely is rather than by which one feels safest. Buyers under budget pressure gravitate toward fixed price because it looks like certainty, and that instinct is correct only in the narrow case where the specification is genuinely stable.
Fixed price suits genuinely fixed scope: small, well understood builds where you are confident in the specification and unlikely to change your mind. Its mechanism is that the vendor absorbs estimation risk and prices that risk into the number, which is a fair trade. Its failure mode is that every ambiguity becomes a change order negotiated from the weakest position you will ever hold, which is midway through a build you have already funded.
Time and materials suits evolving products, and it is the honest default for anything exploratory. It puts estimation risk on you, which is only acceptable with real transparency in exchange: you see the board, the commits and the hours, and you can stop at any point. A vendor offering time and materials without that visibility is offering you the risk without the control, which is the worst structure available.
A dedicated team, meaning named engineers working as an extension of your organization over quarters, suits long running products where continuity and accumulated context are the actual value being purchased. The team learns your domain, and that knowledge is the thing you are buying in year two. This is the model behind our dedicated development team engagements, and it is the wrong model for a three month bounded project, where you would be paying for context that never gets used.
One structural note that applies to all three. Whichever model you choose, insist that work happens in your repositories, your issue tracker and your cloud accounts from the first commit rather than in a vendor environment you receive at the end. This single arrangement removes most of the leverage a departing vendor would otherwise hold, and it costs nothing to set up on day one.
Which model fits which situation
| Fixed price | Time and materials | Dedicated team | |
|---|---|---|---|
| Specification is stable and detailedYou know precisely what you want and will not change it | Yes | Yes | No |
| Product direction is still formingYou expect to learn from users and change course | No | Yes | Yes |
| Multi year core productDomain context compounds and continuity is the value | No | Partial | Yes |
| Hard external deadlineA regulatory date or a launch event that cannot move | Partial | Partial | Yes |
| You have no technical leader in houseYou will be relying on the vendor for judgment | No | Partial | Yes |
| Budget must be exact and approved in advanceProcurement will not fund a range | Yes | No | Partial |
| Small bounded internal toolA few months, well understood, low integration risk | Yes | Yes | No |
Read across the row that matches your actual situation rather than the one you wish applied. Partial means the model can be made to work with specific structural additions, noted in the row.
Choosing the model in one question
How confident are you that the requirements will still be right in three months?
-
Very confident, and the specification is written
Fixed price, with a change process agreed in writing before signing
You can genuinely transfer estimation risk to the vendor because there is a stable target to estimate against. Insist on seeing a worked example of how a change would be priced, because you will need it at least once.
-
Not confident, and the project is a few months long
Time and materials, with weekly demos and full visibility of the board and the hours
Fixed price against an unstable specification converts every learning into a commercial negotiation. Time and materials prices the learning honestly, provided you have the transparency to steer it week by week.
-
Not confident, and this is a core product for years
Dedicated team, named in the contract, working in your accounts
The value you are buying is accumulated context, which only exists if the same people stay. Price continuity deliberately rather than re buying domain knowledge every project.
The determining variable is not project size or budget. It is how likely the requirements are to change while the work is in progress, because that is what each model prices differently.
The four clauses that decide how this ends
Most of a software development contract is boilerplate that neither party will ever read again. Four clauses are not, and they determine what happens on the worst day of the relationship rather than the best one. Argue these and concede elsewhere; a vendor who negotiates reasonably on all four is telling you something important about their confidence in their own performance.
First, intellectual property assignment on payment. You should own the code, the designs and the deployment configuration, and ownership should transfer as you pay rather than on final acceptance of the whole engagement. The distinction matters: assignment on final acceptance means that if the relationship ends in month eight of twelve, ownership of eight months of paid work is contestable exactly when you least want a legal argument.
Second, repositories and infrastructure in your accounts from day one. This is not a legal clause so much as an operational one, and it is the single most effective protection available to a buyer. If the source code lives in your version control organization and the cloud resources live in your accounts from the first commit, then the handover risk that dominates vendor relationships largely evaporates. Vendors who resist this specific arrangement are usually resisting the loss of leverage, which tells you what they were relying on.
Third, a defined exit. Specify what handover consists of as concrete deliverables: architecture documentation, runbooks for operating the system, credential transfer, and a stated number of hours of transition support at an agreed rate. Without concrete deliverables, handover is a promise, and a promise from an organization that has just lost your account is worth what you would expect.
Fourth, termination for convenience with reasonable notice, typically thirty days on a long engagement. This clause is the one buyers most often skip and the one that most changes behavior, because it keeps both sides performing on merit rather than on lock in. A vendor who cannot accept it is asking you to commit to a relationship you cannot leave, on the strength of a proposal.
The four clauses, what to ask for, and what a refusal means
| Clause | What to ask for | What a refusal usually means |
|---|---|---|
| Intellectual property | Assignment of all work product to you as invoices are paid, not on final acceptance | The vendor intends ownership to be a bargaining chip if the relationship ends early |
| Repositories and cloud | Your version control organization and your cloud accounts from the first commit, vendor gets access | The handover has not been thought about, or leverage is being preserved deliberately |
| Exit and handover | Named artifacts plus a stated block of transition hours at an agreed rate | Handover is expected to be a renegotiation rather than a delivery |
| Termination for convenience | Either side may end with thirty days notice, work in progress paid for | The commercial model depends on you being unable to leave |
None of these requests are unusual and none of them are aggressive. They describe the arrangement a confident vendor already prefers, because it removes the suspicion that their commercial position depends on lock in rather than performance.
Design a paid pilot that produces evidence, not a demo
A paid pilot of two to four weeks is the cheapest evaluation instrument that exists, and it is the only one that observes the vendor doing the actual job rather than describing it. Be openly suspicious of any vendor who resists one, and be equally suspicious of your own instinct to skip it because the proposal was persuasive. The proposal is a writing sample. The pilot is a work sample.
The design of the pilot decides whether it produces evidence. A pilot scoped as an impressive demo produces an impressive demo, which is the least useful artifact available, because a demo can be assembled by a strong designer and a weak engineering team in the same fortnight. Scope the pilot instead as a thin vertical slice of the real system: one genuine user flow, wired end to end, touching the real integration that worries you most, deployed to an environment you control, with tests.
Say explicitly at the start that you will judge the pilot on how it was built and how it was communicated, not on how much of it there is. Then require an estimate before the work begins and compare it to the actual at the end, because estimation accuracy on a two week task is a genuine predictor of estimation accuracy on a twelve month program, and it is the only measurement of it you can obtain before committing.
Then introduce a deliberate change in the middle. Halfway through, alter a requirement in a small but real way, as would inevitably happen in a live project. How the vendor responds, whether they quantify the impact and offer options or simply absorb it silently and slip, is the most informative twenty minutes of the entire evaluation. Absorbing it silently is not a good sign, because it means schedule impact will not be reported later either.
Size the pilot so that walking away from it is genuinely comfortable. If the pilot is large enough that abandoning it would hurt, it has stopped being an evaluation and become a commitment, and the vendor will be able to feel that shift as clearly as you can.
A four week pilot that measures the right things
-
Scope one real vertical sliceBefore start
Choose a single genuine user flow that touches the integration you are most worried about, and require it end to end: interface, business logic, persistence, and the external system. Refuse a scope made of screens with no backend, because that is the part of the work that is never the risk.
-
Require a written estimate and a planBefore start
Ask for the estimate broken down by task with the assumptions stated, plus who will do the work. Record it. This is half of the measurement, and it costs the vendor a few hours.
-
Set up access in your accountsWeek 1
Your repository, your cloud project, your issue tracker. Doing this during the pilot proves it is workable and rehearses the arrangement you want for the real engagement.
-
Attend the weekly demo, and read the status noteWeekly
You are evaluating whether the demo shows working software rather than slides, and whether the written status matches what you observed. A gap between those two is the most reliable early warning there is.
-
Introduce one real change midwayWeek 2 or 3
Alter a requirement in a small but genuine way. Then watch whether the impact is quantified and options are offered, whether it is absorbed silently, or whether it is refused as out of scope on a pilot.
-
Review the code with someone technicalWeek 4
If you have no in house technical reviewer, pay an independent engineer for four hours. Tests, structure, error handling, readability. Four hours of independent review before a multi year commitment is the best value spend in this entire process.
-
Compare estimate to actual, and ask what they learnedClose
The gap matters less than the explanation. A vendor who can tell you precisely why the estimate moved has an estimation process, and that process is what you are actually buying.
Adjust the duration to the project, but keep the sequence. The instrumentation, meaning the estimate recorded up front and the mid pilot change, is what converts a trial into a measurement.
Pilot scoping that reveals capability, and pilot scoping that hides it
Do this
- One flow, end to end, deployedInterface through to database through to the external integration, running in an environment you control. This exercises every layer where competence actually varies.
- The integration you fear mostThe legacy system, the payment provider, the hospital records interface. If it is going to be the problem, discover that for the price of a pilot rather than in month five.
- Tests and a deployment pipeline includedAsk for these explicitly and watch whether they were already part of the plan. Whether a vendor writes tests when nobody is contractually forcing them to is a durable indicator.
- A written architecture noteTwo pages on how they would build the whole system, produced from what they learned in the pilot. This is the single most revealing artifact you will receive.
Not this
- A clickable prototypeBeautiful, quick, and completely silent on engineering ability. It answers a design question you probably were not asking.
- Ten screens with mocked dataBreadth chosen deliberately to avoid depth. No integration, no persistence, no evidence about the part that will be hard.
- A rewrite of something they already builtYou will see their existing solution rather than their problem solving. Give them your problem, not one they have already solved.
- Unpaid workIt attracts vendors with idle capacity, invites minimum effort, and gives you no right to the output. It also tells you what you think their time is worth, which shapes the relationship from the start.
Every item in the left column forces the vendor to demonstrate something that cannot be faked in a fortnight. Every item in the right column can be satisfied by a designer working alone.
The first month signals that predict year two
After enough projects the predictors of delivery quality turn out to be observable within the first month, and almost none of them appear on a rate card. Communication rhythm is the first: a fixed weekly demo of working software, a written status against a visible plan, and bad news arriving early and unprompted rather than in answer to a direct question. The unprompted part is the whole signal.
Engineering hygiene is the second, and it is visible even to a non technical buyer if you know where to look. Code review as routine rather than as an exception, automated tests running on every change, deployments that are boring and frequent rather than events, and the vendor working in your repositories and tools rather than in a private environment you will receive at the end. You do not need to read the code to observe all four.
Above all, watch how the vendor handles being wrong. The first missed estimate, the first defect that reaches production, the first design decision you push back on. Vendors who surface the problem themselves, quantify the impact and arrive with options are the vendors who will still be performing in year two. Vendors who go quiet under pressure will go quiet at the worst possible moment, which by definition is the moment you most need them not to.
A final filter that costs nothing: ask the vendor to argue against their own proposal. Where might it fail, what are they least sure of, and what would they cut if the budget fell by a third. Confident vendors engage with this happily and often improve their own proposal in the process. Fragile ones deflect, reassure, or treat the question as an objection to be handled. The vendor you want treats your skepticism as a professional courtesy, because the entire relationship you are buying is one long exercise in earned trust, and they know it.
What to expect, and what to check, in the first ninety days
-
Week one to twoSetup and first slice
Access in your accounts, environments running, the first thin slice of real functionality merged and deployed, and the plan written down where you can see it.
Done when Something real is deployed to an environment you own, and you can name what is being worked on this week without asking.
-
Week three to sixRhythm establishes
Weekly demos of working software, written status notes that match the demos, defects tracked visibly, and the first estimate variance discussed openly rather than discovered by you.
Done when You have received at least one piece of bad news that you did not ask for, and it arrived with options attached.
-
Week seven to twelvePredictability or drift
Velocity becomes measurable, the backlog reflects what you learned rather than only the original plan, and the deployment path is routine enough to be uneventful.
Done when You can forecast the next month with reasonable confidence, and the vendor produces the same forecast independently.
-
The ninety day reviewDecision point
Compare the original plan to reality, review the code independently if you have not yet, and revisit the engagement model in light of what you now know about the work.
Done when A deliberate decision to continue, adjust the model, or exercise the termination clause while it is still cheap to do so.
This is the review cadence we would want a client to hold us to. Each phase has an exit condition that is observable rather than a matter of impression, which is what makes it possible to act early.
The signals, in the order they predict outcomes
- Bad news arrives unpromptedThe strongest single signal available. A vendor who tells you about a slip before you notice it has a working internal process and has decided to be honest with you about it. Both are rare and both are load bearing.
- The weekly demo shows working softwareNot slides, not a design file, not a screen recording of a mock. Software running in an environment, exercised in front of you, including the parts that are not finished.
- They work in your repositories and toolsObservable in one click, and it eliminates most of the handover risk in the relationship. Resistance to this is informative out of all proportion to how minor it sounds.
- Estimates are revised with explanationsRevision is normal and expected. The signal is whether each revision comes with a mechanism, and whether the mechanism turns out to be right the next time.
- The named engineers are the actual engineersCheck in the first two weeks by attending a technical discussion. Substitution that happens quietly in month one will happen again in month nine.
- Defects are visible and triagedA vendor with no visible defect list is not tracking defects, which means the quality picture you have is the one they chose to give you.
- The polish of the proposal deckIncluded here to be explicit that it belongs at the bottom. Sales polish and delivery quality are produced by different people and correlate weakly at best.
Ordered by predictive weight in our experience of both delivering and inheriting projects. The first item outweighs everything below it, and the last item on any similar list is always the sales material.
Frequently asked questions
How long should choosing an engineering vendor take?
From brief to signed contract, four to eight weeks is realistic for a substantial engagement, with the paid pilot accounting for most of that. Compressing it below three weeks means skipping either the engineer interviews or the pilot, which are the two stages that carry the most predictive weight. Buyers under time pressure often reverse this and cut the pilot to save two weeks, then spend six months discovering what the pilot would have told them. If the deadline genuinely cannot accommodate a pilot, at minimum run the engineer interviews and pay for an independent review of a code sample from a recent project.
Is it reasonable to ask several vendors to do a paid pilot?
Yes, for a significant long term engagement, and it is more common than buyers assume. Run two finalists on the same pilot scope, pay both, and tell each that another vendor is running the same scope. The comparison is far more informative than either pilot alone because it controls for the difficulty of the task. For smaller engagements the cost is usually not justified, in which case pick one finalist and keep the second warm rather than dismissing them, since a pilot that goes badly is a real outcome you need a path out of.
How do we compare quotes when the vendors are in different countries?
Compare on total cost of the outcome rather than on rate. Build a like for like view by requiring the same itemization from everyone, then add the costs that vary by arrangement rather than by rate: your own management overhead, time zone friction on decision making, travel if any, and the expected rework. A materially lower rate can be a genuine advantage, since regional differences in engineering cost are real, or it can be seniority substitution that arrives as defects later. The reference calls and the pilot are how you tell those two apart, and no amount of quote analysis will do it for you.
What if we do not have anyone technical to evaluate the engineering?
Buy the capability for a few hours rather than skipping it. An independent senior engineer, engaged for half a day to review a code sample or the output of a pilot, is inexpensive relative to the decision and will tell you things no reference call can. Weight the rest of your evaluation toward vendors who explain trade offs in business terms and show their reasoning in writing, because you will be depending on their judgment more heavily than a buyer with in house engineering leadership. Also insist harder than usual on the four contract clauses, since the arrangements that protect you matter more when you cannot assess the work directly.
Are fixed price contracts a bad idea?
No, they are a good idea in a narrow set of circumstances and a bad idea outside it. Fixed price works when the scope is genuinely fixed, meaning you have a written specification you are confident in and a project small enough that the specification will still be right when the work finishes. It works badly for exploratory product development, because every ambiguity becomes a change order negotiated when you have already funded half the build. If procurement requires a fixed number, one workable pattern is a fixed price discovery phase producing a specification, followed by a fixed price build against that specification, which puts the uncertainty in the cheap phase.
Should we care where the vendor is located?
Care about the practical consequences rather than the location itself. The consequences that matter are working hour overlap for decisions, which determines how many days a question costs; the legal jurisdiction of the contract and whether enforcement is realistic; where your data may be stored and processed, which may be constrained by regulation in your sector; and the depth of the local engineering market, which determines whether the vendor can replace a departing engineer quickly. A vendor eight time zones away with four hours of daily overlap and a strong local market is often a better arrangement than a nearby vendor with a thin bench.
What are the strongest early warning signs to walk away?
Four in particular. Refusal to let you meet the engineers who would do the work, which is nearly always concealing a staffing substitution. Refusal to work in your repositories, which is preserving leverage. An estimate presented as firm with no stated assumptions and no answer to what would make it wrong, which means it has not been estimated. And any resistance to a termination for convenience clause, which is an explicit request that you commit to a relationship you cannot exit. Any one of these justifies removing a vendor from consideration on its own, and they are all detectable before you have spent anything.
And if you want a vendor prepared to be evaluated on exactly these terms, AgileTech is an engineering partner in Vietnam that will show you the named team, the itemized estimate and the pilot scope before asking for your trust.