Data Report · RedactSure Research
Chris Sowa · Founder, RedactSure
Published
Last updated
Who Is Responsible When an AI Agent Takes Patient Records? A Healthcare AI Builder on BAAs, Biometrics and Cost
From HAL: Who Signed Off on This?, with Rishov Chatterjee, AI engineer at Curology and former Director of Data Science and AI at Integra Connect
Episode: Who Is Responsible When an AI Agent Takes Patient Records? Rishov Chatterjee. The episode is on Spotify and YouTube. Read the full transcript.
In brief
-
When a vendor's AI agent takes patient records, the hospital that owns the data carries the blame, and the business associate agreement (BAA) with that vendor decides who answers to whom afterward.
-
Frontier models made AI easier to integrate and harder to contain: every API call carries patient context to a third party.
-
Names are not the only identifier. Voice and images can identify a patient, so give the model only the part of the record the task needs.
-
Agents carry hidden costs in every tool call, and cheaper models move the bill to hardware and hosting rather than removing it.
-
Providers contesting denials need evidence at scale, and smaller organizations will need local models and partners to afford it.
About the episode
Rishov Chatterjee has built AI on both sides of healthcare. At Integra Connect, a value-based oncology care platform, he was Director of Data Science and AI and a principal applied scientist, working with practices, hospital systems and insurers. Before that he was a research data scientist at City of Hope. Today he is an AI engineer and data scientist at Curology, the personalized skincare company, where his research includes privacy for patient images.
His answers come back to one question: what does each request to a model carry? The conversation runs from frontier models and de-identification to breach liability, biometric identifiers, the hidden cost of agents and the economics of contesting denials.
Highlights from the conversation
What changed when patient data met frontier models?
Chris Sowa: You have watched AI in healthcare move from early predictive models to today's agents. What is different in how people need to think about patient data and what they sign off on?
Rishov Chatterjee: How it changes in today's context, I would say, is now we're making less decisions from single models. […] These are easier to integrate because they're essentially API calls. But the problem is that you still have to provide a lot of the sensitive context as an input to these models. So a lot of times there's concerns about how much of that data gets retained. Because now all of a sudden you might be sharing patient level information with third party providers.
Earlier predictive models were trained inside the organization and kept patient data at home. A frontier model reached through an API receives whatever context each request carries. For anyone who signs off on an AI workflow, the question moves from whether the model is accurate to what each call sends out of the building, and what the vendor keeps.
How do teams make patient data safe before a model sees it?
Chris Sowa: How long does the due diligence take to make sure the data is safe and patients' PHI is protected?
Rishov Chatterjee: Whenever we think about PHI, there's really two types of standards we think about. One of them is what we call expert determination, where you have someone that really has tons of years of experience with understanding what makes information identifiable. […] The other perspective would be to use something called the safe harbor method. […] The idea is that you would use an agent or another model to essentially identify from what you have extracted, does it fall under those […] attributes? If it does, then that's going to be your signal that you have to fix it. You have to clean it up, scrub it. And once that's done, then you can use the remaining information for modeling.
These are the two de-identification methods in the HIPAA Privacy Rule. Under Safe Harbor, HHS lists 18 types of identifiers to remove, from names and dates to biometric identifiers and full-face photographs. The check has to run before the record reaches the model, not after, or the scrubbing protects nothing.
If an AI agent took 10,000 patient records, who is responsible?
Chris Sowa: If an AI agent took 10,000 patient records out of the organization, who in your organization would be responsible for that?
Rishov Chatterjee: Assuming that the organization that I work for is a business associate of a hospital or an insurance company, then usually what ends up happening is the risk is given to the personnel that owns the data. So in this case, if the data is EHR related information, the EHR is probably coming from the hospital, which is doing a collaboration with the business associate. […] And the blame will actually end up going to the hospital because it's their information that the EHR was originating from.
The federal rule sets the order. A business associate, the outside vendor in HIPAA terms, must notify the covered entity no later than 60 days after discovering a breach, and the covered entity notifies affected patients within the same 60 day limit, and the media when a breach affects more than 500 residents of a state (HHS, Breach Notification Rule). The vendor route is not an edge case. Of the 220.0 million people affected by large healthcare breaches in 2025 and 2026, 123.9 million, or 56.3%, were exposed in breaches that came through a vendor, a business associate under HIPAA (How Many Healthcare Breach Victims Were Exposed Through a Vendor?). When an AI vendor is the business associate, the hospital answers for what its agent could reach.
Can a vendor sell the patient data it was given?
Chris Sowa: A health system leader on the show argued that because the provider is liable for the data, it owns it, and anyone who uses it needs the provider's permission. What do you think?
Rishov Chatterjee: It really depends on a couple of factors. Because even though the data is originally coming from the healthcare provider, the most important aspect is what did we actually do with that data? […] Conceptually, the data would have to change and it would have to be completely deidentified in order for that to become something that is very different from what we originally started with. But technically, if you are selling it and you have a contract as a business associate, you technically are bound by law to be able to chat with the healthcare organization to get permission to sell.
In terms of agent based liability, […] technically it would be in violation of the contracts, the BAA agreement that's created between the business associate and the organization.
Chris Sowa: I always say BA agreements are critical and we need to have them. But if you have to actually use them, something was bad. Something broke in the process.
The two guests reach the same place from opposite sides. Jim Sinkoff of Sun River Health argues from ownership, and Rishov argues from the contract. Either way, the business associate agreement governs what happens after data leaves. It does not limit what an agent could read before it left. That limit has to be set in the workflow itself.
Why are names not the only identifier?
Chris Sowa: A data engineer in Europe told me that swapping out names in call center recordings did not cut it, because the voice itself is a biometric. How do you handle contextual data like that?
Rishov Chatterjee: What's interesting about GDPR is that they look at data overall in a much more comprehensive way, rather than just saying that we have protected health identifiers. […] Data is becoming more and more multimodal as we start to see progression with agent development. And so as you mentioned, the voice […] is absolutely an identifiable signal.
When you have images of a patient, let's say you have a selfie and you're trying to look at their acne, you can still determine who the person is. […] If it's on the left cheek, then why not just create a segment where you're only focusing on the cheek? So that way you can't determine who it belongs to anymore. […] With voice, it's a little bit different. […] You can potentially go into transcription. And when you go into transcription, then you delete the audio recording.
The same point holds under HIPAA: the Safe Harbor list includes biometric identifiers such as voice prints, and full-face photographs (HHS, de-identification guidance). His design rule is the useful part. Give the model the segment of the record the task needs, a cheek rather than a face, a transcript rather than a recording, and the identifying signal never reaches it.
How do you build privacy into a consumer health app?
Chris Sowa: Tell us about what you are building at Curology and how you are using AI.
Rishov Chatterjee: It's really important that we don't store the personal information of any of these patients. […] One of the things that I'm doing with the agents is to say, can we get the same correct predictions at lower resolution, where it's almost impossible to identify the patient? So low resolution is one angle. And then the other angle is the transformation that was done to make it lower resolution. Can we make it difficult for that to be reversed? Because then you can have a hijacker come in and actually get the original resolution, and that would be a problem.
Two controls, each one testable: reduce what the model receives, and make the reduction hard to reverse. The second applies well beyond images. A protection an attacker can undo only delays the exposure.
What does an agent really cost once the whole organization uses it?
Chris Sowa: Engineers now tell me they have three or four agents writing code while they supervise. What are you seeing?
Rishov Chatterjee: One thing that everyone also neglects to mention is the hidden costs of doing this. There's a lot of expenditure that goes into every tool that gets called and every service that has a response that interacts with the agent. So all of these expenses, even though it might seem insignificant for one person doing it, when the whole organization starts doing it, it exponentially increases.
And this has led to significant budget issues for a lot of organizations, where they […] have to consider controlling budgets to top tier models to only certain dollar amounts as cap limits per month or even on a quarterly basis.
Cheaper models are not an automatic fix. Later in the episode he notes that running open models across an organization means paying for hardware, hosting and scaling, which can come close to the cost of frontier APIs. That makes model choice a budget decision made workflow by workflow. It is also a reason to keep data controls independent of the model. If what the AI is allowed to see is decided outside the model, a team can change models for cost without reopening the privacy review.
Who can afford to contest a denial?
Chris Sowa: Insurers appear to be winning the AI contest on denials so far. What happens to the providers and nonprofits that cannot afford the AI to compete?
Rishov Chatterjee: The capabilities become less for the nonprofits for sure, because they have less access to being able to see the documentation that's coming from the insurance company, analyzing that and being able to draft evidence to overturn those denials to begin with. I've seen this firsthand working at Integra, where I actually had to build a solution like this for doing appeal generation for a revenue cycle management group focused on emergency medical care. […] The insurance company felt that […] the facility that it was traveled to apparently was not the nearest appropriate facility. […] But then they would never actually give you the justification for that. They would never show you on a map, hey, here are all the facilities and here's why that's the nearest one.
A lot of times the best way to solve it is just provide the evidence yourself. […] It does cost money because you have to use APIs for figuring out that geographical structure. […] When you're doing this for hundreds of thousands of denials, that's really when it becomes very expensive. […] You would probably have to rely on local models […] and then from there you have to seek partnerships with companies that do have access to a lot more infrastructure.
An appeal is an evidence package, and every part of it, the record, the map and the draft, touches patient data. Providers that turn to cheaper models or partners to keep up will send that data to more places. The controls on what each model and partner receives need to grow with the volume.
What leaders who sign for AI can take from this conversation
-
For every model call, write down what patient context it carries and what the vendor retains.
-
Settle in the BAA who notifies whom, and on what timeline, before an AI vendor touches records.
-
Treat voice and images as identifiers, and send the model only the segment the task needs.
-
Make every reduction of data hard to reverse.
-
Budget agents by workflow, and keep data controls independent of the model so a cost change does not reopen the privacy review.
Where RedactSure fits
Rishov's answers come back to one question: what does the model receive? RedactSure answers it on the screen. Patient identifiers become tokens before any model reads the page, so an agent can complete a coding, billing or appeal task without the real values. A named person approves each consequential action, and a record of what the AI saw and did is kept for the organization. Because the controls sit outside the model, a team can pick a lower cost model without changing what it is allowed to see. RedactSure does not anonymize audio, so voice data needs its own step first. The work gets done. The data stays hidden from the AI.
Frequently asked questions
Who is responsible when a vendor's AI agent takes patient records?
In Rishov Chatterjee's view, the hospital that owns the data carries the blame, and the vendor, as the business associate, has to tell it what happened. Under the HIPAA Breach Notification Rule, a business associate must notify the covered entity no later than 60 days after discovering a breach, and the covered entity then notifies the affected patients.
Does HIPAA de-identification cover voice recordings and photographs?
Yes. The Safe Harbor method lists 18 types of identifiers, including biometric identifiers such as voice prints, and full-face photographs. Rishov's approach is to send the model only the segment a task needs, such as a cropped image or a transcript in place of a recording.