AI Systems Used in Benefits Eligibility Determination by Government Agencies

Government benefits agencies now run AI across five distinct jobs: pulling in data, checking income and household details, flagging possible fraud, applying eligibility rules, and routing cases to the right queue. That sounds like one system doing "AI eligibility." It's actually five different jobs bolted together, and the law treats each one differently. Whether a system helps a caseworker or replaces one matters enormously, both for the people applying and for the agencies that get sued when things go wrong.

Start with the one job AI cannot do, by law: make the final call. Under federal law governing benefits determinations and the Mathews v. Eldridge balancing test (424 U.S. 319), an approval or denial is a legal determination, not a data output. Mathews doesn't require a hearing before every single decision. But it does require that whatever process exists be fair given the stakes, and courts have consistently read that as meaning a human, not a model, owns the decision.

SNAP shows why this distinction actually holds up. The thresholds aren't something a model infers from patterns in old case files. Gross income at 130% of the federal poverty line, net income at 100%, asset limits of $3,000 (or $4,500 for elderly and disabled households), these numbers sit in statute, in a federal law governing food assistance. § 2014. A caseworker or a computer applies them the same way: correctly, or not. There's no judgment call for a model to learn its way into.

So what does AI legitimately do? Pull wage data from a federal benefits agency through a verification system. Match household composition against vital records. Run OCR on submitted documents so a caseworker doesn't retype every field by hand. Flag cases that look likely to be ineligible so they get routed for faster review. All of that is assembly work, gathering and organizing facts a human still has to weigh.

What AI struggles with is murkier territory: self-reported hardship claims, income for someone working off the books, anyone without a clean paper trail in the usual databases. Fraud prediction sits in the same danger zone. Pushing the threshold too loose lets fraud slip through. Pushing it too tight gets legitimate applicants flagged, creating exactly the kind of due process and disparate impact exposure agencies are trying to avoid.

The scale involved makes precision non-negotiable. SNAP serves 42 million Americans every month. A single state agency can process 50,000 to 200,000 eligibility transactions monthly. Compare that to the caseworker doing it by hand: 4 to 6 hours per application, calling employers, chasing paperwork, reconciling numbers across databases that don't talk to each other. The pressure to automate is obvious. So is the risk: automate the fact-gathering, fine. Automate the judgment, and a small error rate turns into thousands of wrongful denials before anyone notices.

Translating eligibility rules into code

States have been building integrated eligibility and enrollment systems, IEE for short, so someone can apply for SNAP, Medicaid, TANF, and child care through one portal instead of four separate ones. Code for America reported in 2025 that 35 states offer an integrated application covering at least three programs, real progress on the user experience side. That's real progress on the user experience side.

Underneath it, nobody designing these systems to work together produces effects that are less visible and much harder to address, and that mismatch is what makes integration so difficult. Federal benefits policy was written program by program, over decades, by different committees, for different purposes. Nobody designed SNAP rules and Medicaid rules to plug into the same system. States now have to build integration on top of federal requirements that were never meant to integrate.

One approach getting real attention is "Rules as Code": turning statute and regulation directly into executable software, so the code is provably the same as the law rather than a separate interpretation of it. Georgetown's Beeck Center, through its Digital Benefits Network, has been researching this since 2021. In March 2025, the DBN published findings from an experiment testing whether large language models could translate SNAP and Medicaid policy into code across seven states. The point wasn't to prove AI can replace policy analysts. It was to test whether AI can speed up the first draft of that translation.

Policy changes constantly. The 130% federal poverty line threshold resets every October (for 2025, that's $1,967 a month for a single adult). Legislatures amend statutes. Courts issue rulings that shift how a rule applies. Every one of those changes has to flow into the IEE system, and the DBN's interviews with those seven states found wildly different update speeds. One state pushes releases on a frequent cycle through a vendor contract. Another aims for quarterly major updates with monthly minor patches. Others tie changes to multi-year roadmaps that move much slower.

None of the seven states run pure agile or pure waterfall. It's a hybrid, usually built around vendor teams, and that hybrid nature is itself a source of lag. A policy that reads clearly on paper can turn ambiguous the moment someone has to encode it as a rule with defined inputs and outputs. A caseworker facing that ambiguity pauses and asks a supervisor. A coded ambiguity doesn't pause. It runs the same way, silently, on every case that hits it, at whatever volume the state processes.

Building these systems from scratch takes time few people outside the field appreciate: a minimum of 18 months from picking a vendor to going live, with 24 to 36 months more typical. Add 6 to 9 months for USDA-FNS regulatory approval. Add another 4 to 6 months of caseworker training after launch. A rule change that takes Congress a single vote to pass can take a state agency the better part of two years to fully implement in its systems.

The legal architecture that is supposed to keep humans in control

Every adverse decision, an assisted-by-one-model denial included, is legally an "adverse action" under a federal statute governing benefits determinations. § 1320a-8 and the relevant state welfare codes. That label triggers four specific obligations: written notice explaining the reason, that explanation in plain language (not policy jargon), the right to appeal to an actual person, and a record trail that state welfare auditors and the federal Office of Inspector General can review.

Systems also need to show, in that audit trail, exactly where a human caseworker either confirmed or overrode each flag generated by one model. That's a compliance requirement, not a nice-to-have logging feature, and agencies that fail it risk losing federal match funding, historically 50% of SNAP administrative costs under 7 CFR 277.4, though the One Big Beautiful Bill Act is currently... It's a compliance requirement, and agencies that fail it risk losing federal match funding, historically 50% of SNAP administrative costs, though the One Big Beautiful Bill Act is cutting that match to 25% starting in fiscal year 2027.

The Administrative Procedure Act adds another layer: a caseworker has to document their own reasoning and retain real authority to override the model's recommendation. Rubber-stamping the system's output does not meet that bar; courts have treated it as a liability, not a defense. It's not meaningful review, and courts have treated it as a liability, not a defense.

What does meaningful review actually look like? A reviewer needs access to the original evidence, the specific version of the rule that got applied, the system's generated summary, some indication of the model's confidence or uncertainty, and, most importantly, the genuine ability to change the outcome. A caseworker who only sees the AI's recommendation and clicks "approve" out of habit isn't exercising oversight. They're providing legal cover for a decision the system already made.

Auditors increasingly look for what's called a "decision envelope": the scheme and jurisdiction involved, the applicant's identity and how it was verified, the facts that got confirmed and where they came from, which documents were received and their verification status, and the exact policy and rule versions applied. Notice what's missing from that list: a transcript of the interaction. The envelope documents the decision's inputs and authority.

Federal AI policy shifted hard in 2025. President Trump's Executive Order 14179, signed in January, directed agencies to review and roll back Biden-era AI actions (which EO 14148, also signed January 20, 2025, had already separately revoked) and told agencies to prioritize innovation over restriction. The America's AI Action Plan laid out more than 90 actions meant to speed deployment across government. OpenAI and Anthropic both landed government-wide access deals priced at $1 per agency per year, a figure low enough to function as a deliberate adoption incentive.

The binding guidance right now sits in a federal budget office directive, issued in April 2025, which requires impact assessments and human oversight practices for federal AI systems. It replaced prior Biden-era AI guidance and, as of this writing, remains the current standard.

But policy on paper and policy in practice aren't the same thing. Research on algorithmic accountability has raised a pointed concern: human oversight requirements can end up legitimizing a flawed system rather than fixing it. A caseworker's signature on a form creates the appearance of accountability. It doesn't guarantee the caseworker actually had the time, training, or authority to catch what the algorithm got wrong. That gap, between oversight on paper and oversight in practice, is where a lot of the real damage happens.

How Robodebt and MiDAS scaled into mass harm

Australia's Robodebt program ran from 2016 to 2020. Centrelink, the country's welfare agency, used income averaging to detect and pursue supposed overpayments: a method that was cheap to automate but legally shaky, because averaged annual income doesn't reliably reflect what someone actually earned week to week.

The damage was staggering. Roughly $2 billion in debt notices went out to 700,000 current or former welfare recipients. The Commonwealth unlawfully raised A$1.73 billion in debts against 433,000 people. Before the system was ruled unlawful, $751 million had already been wrongly clawed back from 381,000 people. Australia's Royal Commission, reporting in 2023, led to a government settlement of $2.4 billion, according to the paper "Beyond Explanation: Evidentiary Rights for Algorithmic Accountability.""

Michigan ran its own version with MiDAS, a system built to catch unemployment fraud. It carried an 85% error rate. People faced severe financial hardship over debts the state manufactured through a broken algorithm. Michigan's unemployment benefits agency kept defending the system even after early warnings piled up, and it took litigation to force reform. Bauserman v. Michigan UIA ended in 2024 with a $20 million settlement; the same "Beyond Explanation" research reports this.

Why did both systems run so far past the point of obvious harm? Automation removed the friction a human reviewer naturally provides. A caseworker manually signing off on thousands of nearly identical wrongful rejections would, at some point, start asking questions, or simply refuse to keep going. A system doesn't get uncomfortable. A system just keeps running the same logic against the next case without noticing that a pattern feels wrong. It just keeps running the same logic against the next case, and the next, at whatever speed the hardware allows.

That's not a side effect of these two systems. Researchers on algorithmic accountability describe this as the defining feature of algorithmic administrative error: a systemic, unintended deviation from the legally correct outcome, rooted in data problems, design choices, or the system's own built-in limits, and one that scales fast while staying nearly invisible until someone goes looking. Robodebt and MiDAS didn't fail because of some rare glitch. They failed in a way that's structurally built into how automated decision systems behave when nothing forces a pause.

What actually stopped them wasn't an internal audit or a quiet policy fix. It was litigation, brought from outside the agency, years after the harm had already piled up. This pattern recurs repeatedly in this space, revealing that accountability currently rests with outside actors rather than agencies themselves.

Tennessee's TennCare ruling and the federal pattern of algorithmic eligibility failures

On August 26, 2024, US District Court Judge Waverly Crenshaw Jr. ruled that Tennessee's automated Medicaid eligibility system, called TEDS and built by Deloitte along with other contractors, violated the Medicaid Act, the Due Process Clause of the Fourteenth Amendment, and the Americans with Disabilities Act.

The court's findings were specific and damning: TEDS failed to load relevant data correctly, assigned beneficiaries to the wrong households, and produced incorrect eligibility determinations as a result. The class action covered more than 100,000 people whose benefits had been cut off, plus over 100,000 qualified individuals with disabilities who were affected by the system's failures.

One finding stands out beyond the technical failures. The court found that TennCare had intentionally left information out of legally required notices, information people needed in order to appeal. That's not just an algorithm making mistakes. That's a process actively working against the remedy the law is supposed to guarantee. An error-prone system paired with a notice process that hides the path to appeal creates two failures that compound each other in a way that's much harder to fix after the fact.

The plaintiffs were represented by the National Health Law Program, the Tennessee Justice Center, the National Center for Law and Economic Justice, and the pro bono firm Selendy Gay PLLC, a combination that offers a working model for how nonprofit and legal advocacy groups can team up against a state agency's automated systems.

Tennessee isn't an outlier. The 2026 paper "Beyond Explanation: Evidentiary Rights for Algorithmic Accountability" documents a broader pattern. In one state, a named plaintiff. v. Armstrong (2016) challenged the algorithm used to set disability benefit budgets; the formula was struck down, disclosure of the budget tool's documents was ordered on request, and the state had to build a new system. In Arkansas, Ledgerwood v. Arkansas (2019) targeted the ARChoices home care algorithm and forced system changes. Oregon discontinued an algorithmic child welfare tool in 2022 after congressional scrutiny raised concerns about racial bias.

Research published on arXiv looked at algorithmic systems across government more broadly and found a consistent pattern: worse outcomes, human biases baked into the design, and racial and socioeconomic disparities getting wider, not narrower.

Lining up Robodebt, MiDAS, TennCare, Idaho, Arkansas, and Oregon side by side reveals one thread that runs through all of them. The fix never came from inside the agency. It came from litigation, from congressional scrutiny, from outside advocacy groups doing the work of catching what internal governance didn't. And in every case, years passed between deployment and correction. That gap is where the harm accumulates.

The legislative response: state laws rushing to fill a federal vacuum

No federal law governing AI in benefits determination has reached a floor vote. The 119th Congress has introduced the AI Act of 2025, the Algorithmic Accountability Act, and a handful of sector-specific bills. None have moved. Federal policy, for now, is whatever the executive branch decides through executive order. It can shift with the next administration.

That instability is visible in the details. EO 14179, from January 2025, rolled back the prior administration's AI executive order and told agencies to prioritize speed and innovation. Then, in December 2025, a new executive order created an AI Litigation Task Force with one job: challenge state AI laws that conflict with federal policy. That's a direct signal that the federal government expects friction with the states, and is preparing to fight it in court.

States aren't waiting around. Fewer than 200 AI-related bills got introduced across all state legislatures in 2023. That jumped to over 600 in 2024, with nearly 100 becoming law. In 2025, every single state introduced at least one AI bill, for a total of 1,208, and 145 were enacted; MultiState tracked these figures. The pace hasn't slowed. By March 2026, with the legislative session still underway, 1,561 AI-related bills had already been introduced across 45 states.

Colorado's path shows how unsettled this area still is. The state passed a first-of-its-kind law in May 2024 targeting "high-risk" AI systems, requiring reasonable care against algorithmic discrimination and mandating documentation for government services, among other applications. That law never actually took effect. Governor Jared Polis signed a narrower replacement, the Colorado Automated Decision-Making Technology in Consequential Decisions Act, on May 14, 2026, repealing the original before its effective date and setting the new law to take hold January 1, 2027.

Texas took a different route. The Texas Responsible Artificial Intelligence Governance Act passed in June 2025 and became effective January 1, 2026, and it applies to both the companies building AI systems and the agencies deploying them within the state.

So the picture right now: a federal government actively trying to slow down state regulation through litigation, and a growing patchwork of state laws that don't agree with each other on what "meaningful oversight" even means. Whichever side wins that fight determines whether algorithmic accountability in benefits programs ends up with real teeth or stays mostly symbolic. For someone applying for benefits today, that means the protections available depend heavily on which state they live in. The right to know an algorithm played a role in a denial, and the right to a real human appeal, might be firmly established in one state and nearly nonexistent in the one next door.

What citizens can do when an algorithm affects their benefits

Start with notice. Any adverse action tied to an AI-assisted system has to come with a written explanation, in plain language, that ties the denial to specific evidence and specific rules. A denial letter that just says "ineligible" without connecting that outcome to a fact and a rule is a compliance failure worth challenging.

Next, the right to a human appeal. Federal law requires a caseworker to review a case before any adverse action becomes final, and a caseworker who just defers to whatever the system recommended hasn't satisfied that requirement. Document every call, every letter, every interaction with the agency. That paper trail becomes the evidence if the case ever moves to appeal or litigation.

It's also possible to request the decision record itself. A properly governed system should be able to produce the exact policy version that got applied, which data sources were checked, and the specific threshold that triggered the outcome. That request can go through the administrative appeals process, and if the agency won't produce it voluntarily, discovery in litigation can compel it.

Groups like the National Health Law Program, the Tennessee Justice Center, and the National Center for Law and Economic Justice have already built the playbook for challenging these systems, TennCare being the clearest recent example. The AI Incident Database keeps a public record of documented algorithmic failures across sectors, and advocates draw on it as precedent when building a new case.

The harder problem is what happens between one person's bad experience and an actual law that fixes the underlying system. Someone who spots a pattern, a denial that keeps happening to people in similar situations, can document it and bring it to an advocacy group. But turning that individual experience into legislative change is a long process, and it's often opaque to the person who started it. Civic tools that help people turn a documented problem into a structured proposal, one that can be shared and co-signed by others who've hit the same wall, and routed to the elected officials who actually oversee these agencies, start to close that gap between one person's harm and a systemic fix.

None of this is a side issue in how AI gets used in government. It's the main event. These systems now shape whether someone gets Medicaid, food assistance, or unemployment benefits, decisions that affect whether people can pay rent or feed their kids this month. Understanding what these systems actually do, where the law draws its lines, and what happens when an agency crosses one is the main event. It's the difference between a benefits system that answers to the people it serves and one that just runs.