Adding AI and analytics to a product: from ChatGPT APIs to embedded dashboards

Somewhere between the demo and the roadmap, most product teams have now had the same meeting: the one where someone shows a language model summarizing a document or answering a question, and the room agrees the product needs this, without agreeing on what "this" is. The meeting has a sibling, held for a decade, about dashboards: the business wants to see its data, a tool gets bought, and a year later the reports are exported weekly and read never. Both meetings fail the same way, by starting from a capability instead of a workflow.

This guide starts from the workflow. It explains what a language model API actually provides and what it decidedly does not, why retrieval over your own data, not model choice, is the main quality lever, and why the data foundation underneath, the pipelines and the governed platform, decides the ceiling of both the AI features and the analytics. It then sorts the candidate first features by how reliably they repay their cost in production, and closes with the unglamorous work that separates shipped AI from faded prototypes: evaluation, guardrails, and cost budgets.

It is written from delivery experience rather than enthusiasm. Our engineers build AI features and the data platforms under them for products whose teams have usually already run the demo meeting, and the pattern across those projects is consistent: the AI feature is the visible tenth of the work, the data and evaluation underneath are the rest, and the projects that budget that way are the ones still running a year later.

What a language model API actually provides

The models behind products like ChatGPT are available to any product team through an API: send text in, get text out, pay per unit of text processed. What the API provides is completion, a startlingly capable engine for continuing, transforming, summarizing and answering in natural language, trained on broad public text. What it does not provide is any knowledge of your business. It has never seen your product catalog, your policies, your tickets or your prices, and when asked about them it will answer anyway, fluently, from general patterns, which is precisely the danger. The fluency is constant; the grounding is your job.

This division of labor is the single most useful thing to understand before scoping an AI feature, because it relocates the work. Teams that believe the model is the product spend their time comparing providers and burning weeks on prompt phrasing. Teams that understand the model is an engine spend their time on what surrounds it: the retrieval that feeds it the right facts at the right moment, the interface that sets honest expectations, the evaluation that catches drift. The second group ships things that survive contact with real users; the first group ships a chat box that is impressive for a week and wrong just often enough to be abandoned.

The rental model also has architectural consequences worth taking seriously on day one. Per-request pricing means every feature needs a cost model, not just a latency budget. Provider dependence means the model API belongs behind an interface you own, the same insulation rule that governs payments in a commerce build, so a price change or a better model is a configuration change rather than a rewrite. And the data you send is leaving your infrastructure, which makes the provider agreement, what is retained, what is trained on, where it is processed, a compliance document, not paperwork, especially anywhere near personal or regulated data.

What about training your own model? For most product teams the honest answer is: much later, possibly never. Fine-tuning earns its cost in narrow, high-volume cases with thousands of quality examples, teaching consistent format, tone or domain shorthand, and it does not solve the knowledge problem, because facts change faster than anyone retrains. The pattern that solves the knowledge problem is retrieval, feeding the model your current facts at question time, and it is the subject of the next section because it, not model selection, is where AI feature quality is actually decided.

The vocabulary the meeting needs

Language model (LLM)

A model trained on broad text to continue and transform language. The engine behind ChatGPT-class products, rentable per request through an API.

Prompt

Everything sent to the model for one request: instructions, context, retrieved facts and the user’s question. Engineering the prompt is engineering the feature.

Token

The billing and length unit of model APIs, roughly a short word. Costs and context limits are counted in tokens, which is why long documents need retrieval rather than pasting.

Retrieval (RAG)

Fetching the most relevant pieces of your own data and placing them in the prompt so the model answers from your facts. The main quality lever in product AI.

Hallucination

A fluent, confident answer that is factually wrong. Contained by grounding, citations and honest interface design, never fully eliminated.

Evaluation (evals)

A maintained set of real questions with known good answers, run against every prompt or model change. The regression test suite of AI features.

Retrieval: where answer quality is decided

Retrieval-augmented generation is a heavy name for a simple contract: before the model answers, the system finds the handful of passages in your own data most relevant to the question and places them in the prompt, with instructions to answer from them and to say so when they do not contain the answer. The model stops being an oracle and becomes a reader, and the difference shows immediately in production: answers cite real policies instead of plausible ones, admit gaps instead of papering over them, and update the moment the underlying document does, because the knowledge lives in your data rather than in the model’s training.

The engineering weight sits in the retrieval, not the generation, and it is ordinary engineering wearing a new vocabulary. The source documents must be split into chunks that preserve meaning, indexed by semantic similarity so that a question phrased one way finds a policy phrased another, and refreshed on a schedule so the index never argues with the source of truth. Every one of those choices, chunk size, index freshness, how many passages to retrieve, moves answer quality more than switching model providers does, which is why teams that A/B their retrieval outperform teams that A/B their models.

Retrieval also carries the permission model, and this is the part that turns security reviews grim when skipped. The index typically spans documents with different audiences, contracts, HR policies, customer records, and the retrieval layer must filter by what the asking user is allowed to see before anything reaches the prompt, because the model has no concept of authorization and will happily summarize a salary table into a chat window. Access control belongs in the retrieval query itself, enforced by the same identity the rest of the product uses, not in a prompt instruction asking the model to be discreet.

The pattern generalizes past chat, which is where most of its production value hides. The same grounded pipeline drafts replies in a support tool with the relevant policy already cited, summarizes the customer’s history when a ticket opens, turns a natural-language question into a database query for the analytics feature in the next section, and pre-fills a form from an uploaded document. In each case the interface is not a chat box but a button inside an existing workflow, and features shaped that way get adopted faster because nobody has to learn to prompt.

The data foundation both capabilities stand on

The AI features above and the analytics features below share a dependency the demos never mention: they can only reach the data you have actually collected, connected and cleaned. Most organizations discover, at the start of either project, that their data lives in silos, the product database, the CRM, the support desk, spreadsheets with one owner each, with the same customer named three ways and no agreed definition of basics like "active user." No model and no dashboard fixes that; the fix is a data platform, and it is the unglamorous majority of both projects.

The platform pattern is settled enough to describe plainly. Pipelines pull from each source system on a schedule or a stream and land the data in one central store, a warehouse for structured, query-shaped work, a lake when raw and unstructured material needs to live alongside it. Transformations then turn the landed raw material into governed, documented models: the customer table, the revenue definition, the order lifecycle, each with an owner and a tested definition. This governed layer, not the raw sources, is what dashboards query and what retrieval indexes, which is how both features end up agreeing with each other and with the finance report.

Quality checks belong inside the pipeline, not downstream of the complaints. Row counts that suddenly halve, duplicate keys, null rates that spike, definitions that drift between systems: each is cheap to detect at load time and expensive to discover in a board meeting or, newly worse, in an AI answer delivered with perfect fluency. The AI era raises the stakes on data quality in exactly this way: a broken chart looks broken, while a model grounded in bad data produces confident, well-written wrongness that nobody squints at.

The practical advice for a product team is to scope the foundation honestly but not maximally. The first AI or analytics feature does not need every source connected; it needs the two or three systems that feed that feature, piped, cleaned and governed properly, with the platform designed so the next source is an addition rather than a redesign. Teams that build the foundation feature-by-feature ship value while the platform grows; teams that pause everything for a two-year data program tend to arrive with a beautiful platform and no surviving sponsor.

Foundation questions to settle before the feature

• Which systems hold the facts this feature needsName the two or three actual sources, their owners and their update cadence. The feature can reach nothing that is not piped.
• What is the agreed definition of the core entitiesCustomer, active, revenue, order: one written definition each, with an owner. Every unreconciled definition becomes a wrong answer later.
• Where does the governed copy liveOne central store that dashboards query and retrieval indexes, so the chatbot and the chart cannot disagree with each other.
• How fresh must the data beDaily batch is dramatically cheaper than real-time streams. Let the workflow, not ambition, set the freshness requirement.
• Who may see whatAccess rules enforced in the platform and the retrieval query, by the product’s own identity system. The model cannot be trusted with discretion.
• What personal data enters promptsAnything sent to a rented model crosses an infrastructure boundary. Minimize, anonymize where possible, and read the provider retention terms as a compliance document.

Analytics in the product: from Power BI to embedded views

The analytics conversation usually starts with a tool, and Power BI is the tool it starts with most often: connect the sources, build the dashboards, share them with the team. For internal decision-making this works, and the honest advice is mostly about discipline rather than technology: build the dashboards on the governed data layer rather than on direct connections to raw systems, or every department will soon have its own version of revenue; give each dashboard a named owner and a decision it serves, or the workspace becomes a museum of charts nobody reads; and treat the semantic layer, the shared definitions of measures and dimensions, as the actual product, because the visuals are the easy part.

The distinct and more product-shaped question is embedded analytics: putting the charts inside your own application, for your own users, rather than inside a BI tool for your own staff. A logistics platform showing each customer their delivery performance, a marketplace showing sellers their conversion funnel, a clinic system showing utilization, in each case the analytics is a product feature with a login, not a report with a distribution list. The embedding route can still be a BI tool rendered inside the product, which buys speed at the cost of per-viewer licensing and limited control, or purpose-built charts against the governed layer, which cost more engineering and remove both constraints. The deciding variables are usually seat economics at your user count and how deeply the charts must match the product’s interface and permissions.

What makes embedded analytics succeed is the same thing that makes the AI features succeed: placement inside the workflow. A chart used daily is a chart that answers a question someone asks daily, at the moment they ask it, on the screen where they act on the answer. The weekly emailed export fails not because the numbers are wrong but because it arrives divorced from any decision. This is also where the two halves of this article meet: the natural-language question box on top of the governed data layer, ask in words, get the chart and the query that produced it, is retrieval-grounded generation applied to analytics, and it is among the highest-adoption AI features a data product can ship, precisely because it removes the query language barrier without removing the verifiable query.

One warning applies to both routes: analytics inherits every sin of the data underneath, visibly. The dashboard that disagrees with the finance spreadsheet destroys trust in both; the customer-facing chart that shows a different total than the invoice generates a support ticket with a screenshot attached. This is why the governed layer from the previous section is not optional plumbing but the analytics feature itself: the chart is just the last mile of a definition that must be right all the way down.

Analytics that gets used

Do this

• Build every chart on the governed layerOne set of definitions under the BI tool, the embedded views and the AI answers, so no two surfaces can disagree about revenue.
• Attach each dashboard to a decisionName the question it answers, who asks it and when. A chart without a recurring decision is decoration with a refresh schedule.
• Put the chart where the action isInside the screen where the user acts on the answer. Embedded and in-context beats exported and attached, every time.

Not this

• Connect dashboards straight to raw sourcesDirect connections bypass the definitions, and every department invents its own metrics. The disagreement surfaces in the worst meeting possible.
• Ship a dashboard for every requestDashboard sprawl buries the three views that matter under thirty that do not. Retire as deliberately as you create.
• Let the AI answer numbers without the queryA natural-language analytics feature must show the query and data behind each answer, or a hallucinated number will eventually reach a decision.

Choosing the first AI feature that earns its keep

With the foundation understood, the roadmap question becomes concrete: of everything a language model could do in the product, what should be first? The demo-driven answer is usually the most impressive thing, an autonomous agent, a fully conversational interface, and it is usually wrong, because impressiveness and production reliability run in opposite directions. The evidence-driven answer starts from workflow pain the team can already see: where do users read a lot to answer a simple question, write the same thing repeatedly, or leave the product to ask a person something the data already knows.

Three feature shapes repay their cost with high reliability. Retrieval-grounded answers, over documentation, policies or the customer’s own records, work because the failure mode is contained: a wrong answer cites its source and is caught at the citation. Drafting assistance, replies, summaries, descriptions, reports, works because the human edits and signs, so the model contributes speed while the person keeps accountability; adoption is fast because the feature accelerates an existing task rather than demanding a new habit. And embedded analytics with a natural-language layer works because it widens access to data that already exists, with the verifiable query as its citation. All three share one property: the model assists a human decision rather than replacing it.

Two feature shapes deserve deliberate deferral. Fine-tuned custom models solve narrower problems than teams expect, demand training data most organizations have not curated, and freeze knowledge that retrieval would keep current; they earn consideration only after a grounded feature is live and its logged usage has become training data. Autonomous agents, model-driven flows that take actions rather than drafting them, demo brilliantly and fail expensively, because each additional autonomous step multiplies error rates and the failure handling dwarfs the happy path. The disciplined path to both is through the reliable three: ship assistance, log everything, measure where the human accepts the suggestion unchanged, and let that evidence, not the demo circuit, argue for autonomy one approved step at a time.

Costs, guardrails, and evaluation: the unglamorous majority

Per-request pricing is a genuine architectural input, not a finance detail. A feature that calls a large model on every keystroke, sends bloated prompts, or retries without limits can cost more per active user than the subscription that user pays, and the bill arrives after the habit has formed. The engineering responses are standard once named: a per-feature cost budget agreed before launch, measured per request and per user; caching for the answers that repeat, which in support and documentation features is many of them; routing easy requests to smaller, cheaper models and reserving the large one for the hard cases; and prompt discipline, because every redundant token in a high-volume prompt is a recurring charge. Teams that instrument cost per request from the first week treat it like latency, a number with an owner; teams that do not, meet it as an incident.

Guardrails are the second budget line, and they are mostly product design rather than exotic safety engineering. Constrain the surface: a feature that answers questions about the user’s own orders needs refusals for everything else, stated politely and tested deliberately. Filter inputs, because users will paste secrets and attackers will paste instructions designed to override yours, and the system must survive both. Shape outputs: citations on factual answers, confidence-appropriate wording, and an always-visible path to a human for the cases the feature should not own. And log every request and response, both because incidents need forensics and because the logs are the raw material of the improvement loop.

Evaluation is the discipline that keeps the feature good after launch, and it is the one most teams skip because nothing forces it on day one. An AI feature’s behavior shifts when the provider updates the model, when a prompt is edited, when the retrieval index changes shape, and none of those shifts fails a unit test. The working practice is an evaluation set: a few hundred real questions with known good answers, drawn from logs and refreshed as the product moves, run automatically against every change, with accuracy, refusal correctness and citation rates tracked over time. It is the regression suite of the AI era, and its existence is the most reliable single signal that a team is running AI in production rather than visiting it.

The closing arithmetic is the one this article opened with, and it decides project success more than any model choice. The visible AI feature, the prompt, the chat surface, the integration, is roughly a tenth of the work. The data pipelines and governance beneath it, the retrieval and its permission model, the evaluation harness, the cost instrumentation and the guardrails are the rest. Budgets shaped like the demo fail in the first quarter of production; budgets shaped like this paragraph ship features that are still answering correctly, at known cost, a year later. Our own technology trends review makes the wider argument, but this is the load-bearing version: the teams that win with AI are the ones that respected the boring nine tenths.

A grounded AI feature, from evidence to production

1. 1

   Pick the workflow, not the technologyWeek zero

   Find where users read much to answer little, or write the same thing repeatedly. The feature is whatever removes that, with a model inside it.

2. 2

   Pipe and govern the two or three sources it needsFoundation

   Not the whole company’s data; this feature’s data, cleaned, defined and permissioned in one governed layer.

3. 3

   Build retrieval before generationThe quality lever

   Chunk, index and permission-filter the sources, and test that the right passages surface for real questions before any prompt work.

4. 4

   Ship narrow, with citations and a human pathFirst release

   A contained scope, sources shown on every answer, refusals for everything else, and an obvious way to reach a person.

5. 5

   Stand up the evaluation set and cost meterBefore scale

   Real questions with known answers, run on every change; cost per request on a dashboard with an owner and a budget.

6. 6

   Let the logs choose what is nextThe loop

   Accepted drafts, failed retrievals and refused questions are the roadmap. Expand where the evidence points, one workflow at a time.

The team and the first quarter

The staffing shape follows the arithmetic of the previous section. The scarce skill is not prompt writing, it is data engineering: the pipelines, the governed models, the retrieval index and its permission filter are ordinary backend and data work, and they are most of the schedule. A product engineer owns the feature surface and its unhappy paths; someone, often the same engineer at first, owns the evaluation set and the cost dashboard as first-class artifacts. What the team does not need at the start is a research scientist, because nothing in the reliable first features involves training a model; the rented engine is the point.

A realistic first quarter looks unheroic and ends with something alive. Weeks one and two: choose the workflow from evidence, name the sources, write the eval questions before writing any prompt, because the questions are cheap while opinions are still loose. Weeks three through six: pipe and govern the sources, build the retrieval with its access filter, and get honest retrieval quality on the eval set before generation exists at all. Weeks seven through ten: the feature surface, thin, cited, with refusals and the human path, shipped to a small cohort with logging on. The remaining weeks: read the logs, fix the retrieval misses, tune the costs, and write the second feature’s evidence case from what the first one revealed.

The build-or-buy question runs through every layer, and the honest map is layered rather than singular: rent the model always; rent the pipeline and warehouse tooling, which is mature and commodity; own the governed definitions, the retrieval configuration, the evaluation set and the prompts, because they encode your business and travel with you across providers. Teams that own that middle layer switch models in an afternoon when a better or cheaper one appears, and teams that let a single vendor own it discover that the switching cost was the price of the shortcut. The same logic our platform selection guide applies to workflow tools applies here: the escape hatch you will need is the one to check before signing.

Frequently asked questions

The experience behind this arithmetic is first-hand. AgileTech is a software development company in Vietnam whose data and AI engineers build the whole stack this guide draws, pipelines to product surface, and who will tell you honestly when the feature you want is a data project first.