Five Decisions Before Building a Healthcare AI Product

A healthcare AI prototype can look finished early. A model reads a referral packet and produces a useful summary, and the team starts asking for build estimates. Those estimates are likely to be wrong until a few product decisions are settled, because they determine whether a clinic will pay for the product, what regulatory exposure it carries, what buyers will ask in diligence, and what it costs to operate.

None of them shows up in a demo, which makes them easy to defer, and each is expensive to reverse once the data model, interface, and contracts are built around it. Here is how I would work through them with a founder starting now.

Consider a hypothetical startup building a tool for specialty clinics. It reads incoming referral packets, drafts a summary for the intake nurse, flags missing documents, and suggests how urgently the patient should be seen.

Decide who the product serves and what it should improve

Start with a plain-language intended-use statement: who uses the output, what decision it informs, in what setting, and in which states. Then ask whether the product improves that work enough for a clinic to buy it. For the referral tool:

• Who uses it, and who approves the purchase? The intake nurse uses it daily; a practice administrator probably signs, and a clinical lead has to trust it. The nurse wants less rework, the clinical lead wants no urgent referral missed, and the administrator wants a cost that makes sense against staffing.
• Where does it fit in intake today? Referrals may arrive by fax, EHR inbox, or portal. A summary in a separate tool, seen after the nurse has opened the packet, adds a step instead of removing one.
• Does it reduce total review time? Measure time per referral including checking and correcting the summary. One the nurse must verify line by line can take longer than reading the packet.
• What pilot result justifies expanding? Agree before the pilot, with the clinical lead and buyer: for example, lower review time with no urgent referral missed, judged by the clinic's own reviewers.

Intended use also defines the product itself. The tool could be a summary and missing-document checklist for intake staff, an urgency suggestion shown to a nurse who makes the scheduling call, or automated messages telling patients how soon to book. The first is an administrative aid, the second enters a clinical decision, and the third talks to patients, which brings its own rules.

California shows why these distinctions matter. Under AB 3030, covered healthcare providers must label AI-generated messages about a patient’s health and explain how to reach a human. Scheduling reminders are excluded, as are messages read and reviewed by a licensed or certified healthcare provider.

For the referral tool, whether urgency guidance goes directly to patients or passes through a nurse’s review changes the compliance questions. Confirm which requirements apply before building that workflow.

The duty lands on the provider, so your product has to make it easy to meet: configurable disclosure text, a review step that cannot be skipped, and a record that it happened. The states you sell into become a scoping decision, because each new state may add requirements.

For this referral workflow, I would start with staff-facing summaries and missing-document checks in a few states, with a human making every clinical decision. The general principle is a first release with a useful, bounded purpose and manageable risk, which is not always staff-facing. The AI-enabled platform I helped shape as fractional CTO for a healthcare nonprofit serves the public directly. Its AI companion was built to help people understand general information and find the next step, not to diagnose, prescribe, or make clinical decisions, with explicit safety boundaries and escalation to human support. The architecture also keeps educational interactions separate from clinical and identifying information.

Draw the AI boundary before designing the interface

Next, define what the model may produce and how a clinician checks it. This is where product design and regulatory status meet.

FDA’s clinical decision support guidance sets four criteria for excluding certain software from medical-device regulation. A key requirement is that clinicians can review the evidence behind a recommendation and use their own judgment.

That becomes harder when a decision is too urgent for meaningful review. Clinical recommendations sent directly to patients also fall outside this particular exclusion.

For the referral tool, showing an urgency rating is therefore a different product decision from summarizing a packet. Have regulatory counsel assess the intended use and applicable FDA oversight before committing to that feature.

A clinician needs enough information to judge whether a recommendation is reliable: what information the tool used, what is missing, and how it was tested. For the referral tool, that means linking facts to the original packet, flagging missing information, and explaining how the urgency feature was evaluated. Asking the model to explain its own rating is not enough. That explanation can also be wrong. The nurse needs evidence they can check and a clear understanding of the tool’s limitations.

FDA’s advisory committee materials on generative AI mental health devices highlight risks such as unsafe advice, missed warning signs, biased responses, and models agreeing with users when they should challenge them.

The materials focus on mental health and are not binding guidance, but they offer useful questions for other healthcare AI features: what unsafe behavior should we test for, which checks should run in code, and when should the system hand off to a person?

Map where patient data goes

A signed business associate agreement (BAA) from the model vendor is only one part of your privacy plan. You still need to assess how your application collects, sends, logs, and stores patient data.

HHS’s guidance on HIPAA and cloud computing explains that organizations must conduct their own risk analysis when using cloud services. For a model API, that means mapping every service that handles patient data and deciding what each can store, for how long, and who can access it.

HHS does not certify products as HIPAA compliant, so vendor claims need supporting evidence. Fisher Phillips recommends requesting vendors’ risk analyses and lists of other providers that handle the data. Be ready to provide the same information to your customers.

For the referral tool, I would answer these early, in a data-flow diagram and vendor list:

• Does the whole packet go to the model, or only the sections the summary needs?
• Are prompts and responses logged, where, and who can read them? Debug logs and evaluation datasets quietly become second copies of patient records.
• Which other services touch the data, such as OCR, error tracking, or analytics?
• Who holds the encryption keys, and how is production access granted?

Decide how you will show it works for each customer

Accuracy on a developer's own data can say little about a specific clinic. An external validation of a widely deployed sepsis prediction model at Michigan Medicine found it performed substantially worse than its developer reported. At a score threshold of 6, it flagged 18% of hospitalizations, 12% of flagged hospitalizations involved sepsis (the hospitalization-level positive predictive value), and it missed 67% of sepsis cases. That is one model at one health system with 2018–2019 data, but the lesson carries: alert burden and missed cases matter as much as headline accuracy.

So decide how each customer will see evidence on their own data. Past referrals with the urgency the clinic's nurses assigned are a useful comparator, but agreement alone is not the test: historical calls can be wrong, and a tool can match most of them while missing the cases that matter. I would measure:

• disagreements, each reviewed by a qualified clinical reviewer to judge which call was right
• urgent referrals rated as routine, the most serious failure
• summary statements the packet does not support
• how often the tool failed to flag a missing document
• total review time per referral, including corrections

These become the pilot criteria from the first decision. Decide who labels the data, who signs off, and what result stops a rollout.

Clinics will also want to know how you have tested for bias. Under federal nondiscrimination rules, covered healthcare organizations must identify decision-support tools that use characteristics such as race, age, or disability and take steps to reduce discrimination risks.

For the referral tool, be ready to explain what information drives the urgency suggestion and how you checked whether it works reliably across different patient groups.

Decide who owns the product after launch

From its first day with real patients, the product needs a clinical owner who decides what counts as an acceptable error, someone watching for drift, and a route for urgent cases. Escalation is a staffing decision as well as a feature, and it belongs in the budget.

Model and prompt updates can change how the product behaves. Before releasing an update, rerun your evaluation tests, document what changed, and tell customers about changes that affect how they use the tool.

For regulated AI devices, FDA can authorize a change control plan that allows specified updates without a separate submission for each, provided the plan’s testing and acceptance requirements are met. I would apply the same discipline to any healthcare AI product: define how changes will be tested and approved before they reach users.

Turn the decisions into a build plan

Before asking for estimates, I would put the five answers into a short decision record:

1. Intended use and value: user, buyer, workflow fit, target states, and the pilot result that justifies expanding.
2. AI boundary: what the model may produce, how supporting evidence is shown, and which checks run in code.
3. Data flow: what PHI reaches which service, what is logged, and the subprocessor list.
4. Customer validation: evaluation set, clinical review of disagreements, metrics, and sign-off owner.
5. Operations: clinical owner, monitoring, escalation, and the change process for models and prompts.

Have regulatory counsel review the intended use and AI boundary before interface design. Then estimate, including the review screen, audit trail, evaluation tooling, and operating roles.

For the referral tool, the staff-facing summary and missing-document checks ship first. The urgency suggestion waits until it has been evaluated on one clinic's past referrals and the review screen shows its supporting evidence.

If you are working through these decisions for a healthcare product, a Product and Launch Blueprint turns them into a decision brief and phased plan your clinical, technical, and commercial teams can share.