NYC MyCity chatbot tells businesses to break the law

What happened

The Markup, with THE CITY, found that New York City's official MyCity business chatbot routinely told owners they could break the law, including taking workers' tips, refusing cash, and discriminating against tenants paying with housing vouchers. The city left the bot running for months after the findings.

Root cause

An LLM deployed as an authoritative government advice tool produced confident but unlawful guidance, with no reliable grounding in actual regulations or guardrails against illegal answers.

Fix / outcome

The city added disclaimers and defended the bot before eventually retiring it. The case underscored public-sector liability for AI-generated advice, alongside Air Canada.

Sources