A New York City AI chatbot designed to help businesses and landlords circumvent government regulations was found to be providing users with inaccurate and potentially illegal information.
markup report In an effort to harness the power of artificial intelligence to improve government services, New York City Mayor Eric Adams (D) announced the introduction of AI-powered chatbots in October 2023. Powered by Microsoft’s Azure AI service, the chatbot provides New Yorkers with reliable information about starting and operating a business in the city from over 2,000 NYC Business web pages. However, five months after its launch, The Markup’s investigation revealed that the chatbot was offering advice that could lead to companies breaking the law.
The chatbot has been found to provide inaccurate information on a wide range of topics, including housing policy, worker rights, and rules for entrepreneurs. When asked if landlords were required to accept tenants with Section 8 housing permits, the chatbot incorrectly stated that they were not required to accept these tenants. This advice violates New York City law, which makes it illegal for landlords to discriminate based on source of income, except in small buildings occupied by the landlord or her family.
Rosalind Black, director of Citywide Housing at Legal Services New York City, a legal aid nonprofit, tested the chatbot herself and found more misinformation. The bot incorrectly stated that it was legal for landlords to lock tenants out and that there was no limit to the amount of rent they could charge residential tenants. Black emphasized that these inaccuracies are related to the city’s basic housing policy and that chatbots should be removed if they cannot provide accurate and responsible information.
The chatbot lack of knowledge also extends to consumer and worker protections. It rejected a 2020 law requiring businesses to accept cash, incorrectly stated that restaurant owners can receive tips from employees, and incorrectly claimed there were no regulations regarding notifying staff of schedule changes. The bot’s inaccuracies remain even when questions are asked in other languages.
The city acknowledges that the chatbot is an experimental program and may occasionally generate inaccurate or biased content, but users have little way to know whether the information they receive is false. Pop-up notifications encourage visitors to report inaccuracies through the feedback form, but acting on false information can have serious consequences for business owners.
read more Markup here.
Lucas Nolan is a reporter for Breitbart News covering free speech and online censorship issues.
