New York City’s AI business chatbot is a cautionary example. Promoted as a helpful resource for small-business owners, the system at times provided unethical and potentially illegal guidance to users. The problem wasn’t just a technical error; it was a governance failure. There were no clear guardrails, no review process to catch harmful outputs and no defined accountability when the system went wrong.
Contrast this with cities that have taken a more deliberate approach. Midland, Texas, introduced an AI-assisted chatbot alongside a digital 311 platform to help route resident questions and service requests. The tools were limited to low-risk, clearly defined functions, with staff oversight and clear handoffs to humans when needed. Residents could still reach a person, and city staff retained authority to review and correct information. The difference between Midland’s success and New York’s failure lies not in the sophistication of the technology but in the maturity of its governance.
So how can a local government determine whether an AI tool is right for the community?
Remember that New York City chatbot? Here’s what went wrong: The system told business owners they could take a portion of workers’ tips, fire employees for being pregnant and engage in price-fixing with competitors — all illegal under New York law. When the problems emerged, the city couldn’t quickly disable the system. There was no review process to catch harmful outputs before they reached the public. Business owners following this AI-generated advice could face lawsuits, fines and labor violations.
Can city officials explain how the chatbot produced those answers? Can they show which data it relied on? Who reviewed the guidance before it went live? Who’s helping the business owners now?
The stakes get higher. In the Netherlands, a tax authority algorithm flagged thousands of families as child-care benefit fraudsters. Families were ordered to repay tens of thousands of euros. Many went bankrupt. Homes were lost. Children were taken into foster care. When questioned, officials couldn’t explain how the algorithm made its decisions. Years later, the government discovered the system was wrong — these families weren’t fraudsters. As the scandal unfolded, the entire Dutch cabinet resigned. The harm to families was irreversible.
So how can a local government deploy this emerging technology without ending up doing more harm than good? Before expanding AI use in your city or county, bring the following exercise to your next vendor meeting or budget discussion. If you see red flags, you’re not ready. If you see mostly yellow, proceed carefully with safeguards. Green means you’ve done the groundwork.
RED FLAGS — Stop, Not Ready
• No one can name who’s responsible when the AI fails.
• Staff can’t override or turn off the system.
• The vendor can’t — or won’t — explain how it works in plain language.
• There’s no testing plan before going live.
• You can’t explain it to residents or tell them how to challenge it.
YELLOW FLAGS — Proceed With Caution
• Only technical staff understand how it works.
• The budget covers only purchase, not oversight.
• Success metrics are vague.
• Front-line staff weren’t consulted.
• There’s no opt-out for residents.
GREEN FLAGS — Conditions Favorable
• Technical, legal, operational and resident voices are all represented.
• Decisions are traceable and reviewable.
• Humans approve outputs before they reach the public.
• The system has defined boundaries.
• You’re starting small.
Government has navigated moments like this before. Progressive-era reforms reshaped public administration by aligning efficiency with ethics. Today, AI presents a similar inflection point when it comes to accountability.
Here’s the simplest test of all: Could you explain your AI system to a non-technical councilmember in five minutes — what it does, how it’s supervised and what happens when it fails? If not, you probably don’t understand it well enough to deploy it.
Before your next AI pitch meeting, ask yourself: Do we have more green lights than red flags?
If not, the most innovative thing you can do is wait until you’re truly ready.
Katryna Peart is an independent AI consultant specializing in large language model fine-tuning, bias detection and stress testing for enterprise deployments. She has published research on municipal AI governance with the International City/County Management Association.
Governing's opinion columns reflect the views of their authors and not necessarily those of Governing's editors or management.
