Natural Language Processing in Public Comment Analysis for Federal Rulemaking

The Administrative Procedure Act of 1946 requires federal agencies to consider every "relevant matter presented" during a rulemaking and respond to it in the final rule. That sounds manageable until you see what agencies actually get: not thousands of comments, but millions, on dockets covering everything from broadband policy to national forest management. Natural language processing tools now do the sorting work that made this legal standard nearly impossible to meet by hand. Here's how that technology actually works, where it's been tested, and where it still falls short.

What the volume of public comments actually looks like at scale

The Federal Register publishes tens of thousands of documents a year, and the comments attached to them pile up fast. The Forest Service once got more than 1 million public comments on a proposed rule tied to the Roadless Rule. The FCC's net neutrality docket drew more than 22 million. Those are the extreme cases, but dockets running into the tens of thousands happen often enough that nobody at these agencies treats them as unusual anymore.

The law says agencies have to consider all of it. No team of reviewers, however dedicated, can read a million comments one at a time and give each one the attention the APA implies. A white paper from FI Consulting stated that without analytical tools, reviewers end up sampling documents, and that sampling happens haphazardly, without consistency, without controls, and without any audit trail showing how a comment got handled.

The legal requirement is total. The human bandwidth to meet it is not. That gap, not any single piece of software, is the actual problem NLP got brought in to solve.

It's also a routing problem, and a nastier one than it sounds. A single proposed rule might touch air quality standards in one section and worker safety in another. Getting each comment in front of the right subject-matter expert inside the agency adds a whole extra layer of work on top of just reading everything.

How spam and bot campaigns distort what agencies actually hear

Bots post comments on federal dockets. Not occasionally: routinely, usually as slight variations on a single script run through a text generator. This is documented behavior, not a hypothetical risk agencies are bracing for.

MITRE Corporation's Sanith Wijesinghe has a useful phrase for it: "deep fake comments." Many of these submissions read close enough to human writing that a reviewer can't tell the difference just by reading them. Mark Febrizio at George Washington University's Regulatory Studies Center has engaged with a related worry: as generative AI gets better at sounding human, whether comments it writes will slip past spam filters built to catch older, cruder bot patterns.

Washington has noticed. The Biden administration's 2023 executive order on modernizing regulatory review directed OIRA to look at guidance and tools for mass comments, computer-generated comments, and comments falsely attributed to people who never wrote them. The House passed a bill in May 2024, introduced two months earlier, that targets AI-generated comments specifically and would require some form of human verification.

Here's where "block the bots" stops being a full answer. Scholars of the rulemaking process have argued that mass comment campaigns aren't an abuse of the rulemaking process. They don't fully democratize it either, most likely, but they don't grind agency operations to a halt the way outright fraud would. Oliver Sherouse, an SBA regulatory economist, raises a different worry: mass campaigns, even legitimate ones, can bury comments from small businesses that don't have the resources to compete for attention in a flooded docket.

That's the real tension running under all of this. Any spam filter strict enough to catch bots also risks catching genuine citizens, especially the ones with the fewest resources to speak up twice if their comment gets flagged the first time. A filter tuned for precision on one side loses recall on the other. Agencies haven't found a setting that avoids both failure modes at once, and it's worth asking whether one exists.

What NLP tools actually do when applied to a comment docket

William Eggers, writing in Route Fifty, describes NLP as a form of AI that spots patterns in language, works out meaning, and infers things the text doesn't say. Applied to a comment docket, that turns into a handful of concrete jobs.

• Topic identification. The model works out which part of a proposed rule a comment is actually addressing, without a programmer having to define every possible topic ahead of time.
• Grouping and clustering. It finds comments that share a theme, including near-duplicates and exact copies, which matters enormously in mass-campaign dockets.
• Spam detection. It flags likely bot or template comments so reviewers spend their limited time on comments that actually need a person's judgment.
• Sentiment analysis. It gauges whether the overall tone toward a proposal runs positive, negative, or neutral.
• Routing and summarization. It sends comment clusters to the right internal expert and builds summaries that keep the actual reasoning in a comment intact, not just a topic label stuck on top.

Before any of that analysis happens, the text goes through preprocessing: normalization, tokenization, lemmatization, stop-word removal, and translation for comments not submitted in English. A November 2025 study on ResearchGate walked through methods ranging from older probabilistic approaches like LDA topic modeling up through transformer-based models like BERT. Its sentiment breakdown found 42.8% of comments landed as positive, 28.7% as neutral, and 28.5% as negative, with negative comments associated with themes including service delivery, infrastructure, and perceptions of the policy process.

None of this replaces a reviewer's judgment. What it gives that reviewer is a structured, traceable path through the docket, so time gets spent interpreting the comments that need interpreting, not sorting the ones that don't.

The CDO Council pilot: what a federal-wide NLP tool actually produced

The Chief Data Officer Council ran a pilot built around this exact problem, publishing it as a proof-of-concept in 2021 through resources.data.gov. The approach used neural network models trained on huge volumes of text, models that arrive already carrying a working knowledge of English grammar and a fair amount of general world knowledge, on top of whatever docket-specific training got added later.

The design choice that mattered most: the models were built to generalize. The goal wasn't a tool tuned to one agency's dockets. It was something any federal agency could pick up and run, regardless of subject matter or the kind of comments coming in. In practice, the pilot could group comments by general content, surface the topics and themes running through a docket, and flag duplicates and near-duplicates for review.

GSA's Ken Ambrose hosted the source code publicly on GitHub, an open-source move clearly meant to invite other agencies to build on the work instead of starting from scratch. The framing behind the pilot was blunt: save time, save money, get more actual value out of every comment reviewed. Not replace the legal obligation to respond, but make meeting it possible at this scale.

What the pilot didn't settle: how to deploy something like this where privacy rules restrict what data can leave an agency's own systems, and how to do argument-level summarization, not just topic grouping, at scale. Those questions got picked up later.

How LLM-based pipelines push the capability further, and where they run into trouble

A 2026 study in Information Processing & Management, from Dong, Zhu, Jensen, and colleagues, built a unified pipeline meant to do three things at once: measure how much of a docket is spam, detect the templates spam comments are built from, and generate summaries that preserve the actual argument inside a substantive comment.

The researchers tested it against three agency datasets, drawn from a telecommunications regulator, a natural resources agency, and a labor rights division, picked specifically because they represent very different comment volumes and comment types. On the spam-quantification task, the pipeline hit F1 scores of 0.987, 0.993, and 0.995 across the three datasets, reliably working out what share of each docket was spam versus genuine submission.

The harder task is summarizing arguments in a way that keeps the reasoning intact. There, few-shot prompting (giving the model a small number of worked examples to learn from) beat zero-shot prompting by a wide enough margin to stand out on its own. That gap matters: it means the model needs examples of what a good argument-summary looks like, not just an instruction to write one.

Then there's the privacy problem, and it's not a minor caveat. Several agencies interviewed for the study treat data privacy as a top-line concern, which rules out sending comment text to cloud-based models like GPT-4. So the researchers tested locally deployable alternatives instead: Llama 3.1 8B and Mistral 7B, both far smaller than GPT-4 and both able to run on an agency's own hardware. Llama performed comparably to GPT-4 on the pipeline's tasks. For agencies that can't send data outside their own walls, that's a real result, not a footnote.

One problem the study surfaces and doesn't fully solve: spam-biased topic distribution. Flood a docket with bot comments, and the topic model built from that docket ends up reflecting the bots' talking points rather than what actual members of the public are saying. Argument-extraction errors are the other open issue. LLMs can mis-attribute a claim or drop it entirely, and given that the legal standard is responding to substantive points, an error like that carries real substantive consequences. It's the exact failure the APA was written to prevent, just committed by a machine instead of an overworked reviewer.

Why agencies have been slow to adopt these tools even when they work

The tools work, well enough to be worth using. Adoption still lags, and the reasons for that lag aren't really about the technology. The bottleneck here isn't accuracy scores.

Some agencies have data science teams that could run a pipeline like the one above tomorrow. Many don't, and a generalizable toolset still needs somebody on staff who knows how to run it. Beyond staffing, there's a policy gap: agencies haven't settled internal rules for how an NLP-flagged duplicate or a clustered summary should count toward the APA's "consideration" standard. Does a summary of 10,000 near-identical comments satisfy the requirement to address them? Nobody has a clean answer written down anywhere.

Legal caution plays a role too. Agencies worry about data provenance, about commenter privacy, about whether a court could challenge a rule if a comment got sorted wrong or excluded by an automated system. And the privacy restrictions that pushed the 2026 study toward Llama and Mistral apply agency-wide, not just to one research team: cloud-based tools stay off the table for agencies handling sensitive policy areas.

Platforms like Regulations.gov and the Federal Register provide access points for comment data, but the pipeline connecting those platforms to an agency's own analytical tools isn't standardized yet. The pattern here looks familiar to anyone who's watched government IT before: pilots succeed, and then generalization stalls. The CDO Council put its code on GitHub for anyone to use. Code being available and code being adopted turn out to be two very different things. What's holding this back is mainly non-technical, tangled up in organizational habit, legal risk-aversion, and plain resource constraints. Better models alone won't untangle it.

What genuinely responsive rulemaking would require beyond better spam filters

Filtering out bots and grouping comments by topic gets an agency partway there. The legal bar sits higher: agencies have to respond to unique, fact-based comments, which means understanding the argument a comment makes, not just knowing what bucket it falls into.

The problem starts before any analysis happens. A citizen who writes an unstructured paragraph with no reference to a specific rule provision is much harder for any system, human or automated, to parse and route than a comment that names the exact section it's responding to. Most people submitting comments get no guidance at all on how to write something that survives triage and actually lands on a decision-maker's desk.

Treating mass comments as noise has a real cost too, and agencies should resist the reflex. Some of those campaigns represent genuine, if coordinated, public sentiment. The harder technical challenge is telling organized advocacy apart from manufactured volume, rather than lumping both under the same spam label. Sentiment analysis has a similar limit: it tells an agency how people feel, not what they know. A cluster of negative sentiment isn't automatically the kind of new data or policy argument the APA is asking agencies to weigh.

A civic gap causes most of this, and no spam filter was ever going to fix it. The comment process is open to anyone in theory. In practice, it's dominated by industry groups, law firms, and advocacy organizations that have the resources to write comments agencies are inclined to take seriously. The fix agencies keep reaching for is better filtering on the input side, when the shortage is actually on the output side, in ordinary citizens' ability to write a comment that reads like the ones industry lawyers file. What technology can actually do here is help ordinary citizens write comments that are specific, tied to an exact provision, and backed by something more than opinion, the same logic that animates broader efforts to make technical policy participation more accessible. A rulemaking comment and a legislative proposal aren't the same instrument, but they share the same access problem: without some kind of structured help, an ordinary citizen's comment tends not to reach the decision-maker in a form the decision-maker can actually act on.