Concerns Rise Over AI Use in Education

Every day, millions of students rely on AI chatbots for assistance with schoolwork and emotional support, often without a clear understanding of where their interactions and personal information may end up.

As AI becomes an integral part of the educational landscape, concerns regarding data usage and the responsibility of schools to protect student information are growing among educators, parents, and students alike.

At the start of the school year, the two largest school districts in the U.S., New York City and Los Angeles, announced a pause on student use of AI technologies for a year while they explore effective ways to integrate such tools into classrooms.

This pause comes amidst increasing backlash against AI and screen time, prompting Microsoft to announce its commitment to implementing protective measures and privacy standards for its AI tools in schools, developed in collaboration with the American Federation of Teachers (AFT), the nation’s second-largest teachers union.

Although the safety measures outlined in the agreement are significant, Josh Golin, the executive director of the online safety nonprofit Fairplay, cautioned that their effectiveness hinges on all AI companies, particularly major tech firms, consenting to similar guidelines. He also raised questions about the appropriateness of AI products in classrooms and the appropriate age for their introduction.

“I have concerns that a prominent framework like this could lead schools to mistakenly conclude, ‘If they’re managing data privacy and safety well, we should adopt this tool,’” Golin remarked.

The AFT expressed optimism that the binding agreement, which includes third-party compliance audits, could establish a new industry benchmark. OpenAI and Anthropic, two other leading AI firms, indicated they are also negotiating privacy and safety agreements with the AFT. However, Google has yet to clarify whether it will adopt similar protections.

Agreement Limits Student Data Usage for AI Training

The agreement with Microsoft specifies that it will refrain from using data from students or educators to train AI systems, with a narrow exception to protect student safety. It also restricts the use of collected data for selling, advertising, or product development purposes.

Furthermore, the agreement disallows AI tools designed to foster emotional dependence or prolonged engagement beyond what is necessary for educational tasks. It incorporates audit and transparency measures requiring the company to communicate in straightforward terms to families regarding how its tools function.

Brad Smith, Microsoft’s vice chair and president, asserted that the standards set a high benchmark for child privacy and AI safety, committing to extend these agreements to every school district they partner with starting November 1.

AFT President Randi Weingarten termed the agreement as a pioneering step towards establishing industry norms in the absence of federal and state legislation on AI in academic environments.

“We want parents to maintain control over their children’s education. We want educators to dictate that, not educational technology. This agreement reflects that aim,” Weingarten emphasized.

She made her remarks at a joint press conference with Smith at the United Federation of Teachers headquarters in New York City, home to the AFT’s National Academy for AI Instruction for educators, which was initiated last year with significant financial backing from Microsoft, OpenAI, and Anthropic.

Alongside their yearlong suspension of AI use, the educational systems in New York City and Los Angeles plan to conduct thorough audits of their educational technology contracts and AI tools, emphasizing privacy, transparency, and accountability. In New York City, the ban aligns with all elementary and middle school students, while in Los Angeles, it extends through high school.

Teachers Union Reaches Out to Google and AI Companies

Weingarten reported that she proactively contacted Google to inform them about the new agreement before it was made public, hoping to encourage the company to endorse the initiative.

The tech giant has yet to respond to inquiries about whether it will implement analogous protections.

“Will Google embrace this framework? That’s a significant question,” said Golin, underscoring the widespread usage of various AI-driven educational tools from smaller firms within schools.

Google faced criticism after it decided to enable its Gemini chatbot for many K-12 students utilizing Google Classroom, a widely used learning management system. The company claims that Google Classroom serves around 150 million educators and students globally.

Critics argued that Google’s actions intensify the competition among tech giants to expand their influence in educational settings and capture the next generation of AI users. This new integration only affects schools that permit student access to the Gemini application or Notebook, marking a notable shift as before, it was mainly available for students aged 18 and older.

Weingarten mentioned that dialogues are ongoing with OpenAI and Anthropic as well, both of which publicly commended the newly established agreement.

OpenAI, the creator of ChatGPT, labeled the agreement as a significant achievement for AI safety and privacy in academic institutions. The firm has been collaborating with the AFT and anticipates finalizing its own agreement soon.

Anthropic also expressed support, stating, “We commend efforts aimed at responsible AI implementation in education,” while collaborating with AFT to establish a “gold standard for safety and privacy.”

New Standards Shift Privacy Responsibility to Companies

Experts in data privacy suggest that these new standards could influence how schools negotiate with AI providers, regardless of whether they are affiliated with Microsoft.

“I am hopeful that this alleviates some pressure on individual school districts while dealing with large tech firms,” said Elizabeth Laird from the Center for Democracy and Technology. Previously, school districts’ ability to secure privacy protections was often linked to their financial resources. “This standard signifies progress toward eliminating that disparity across districts,” she noted.

Laird praised the agreement for exceeding current legal requirements for vendors collaborating with schools. For instance, Microsoft has pledged not to utilize sensitive data often exploited by companies, such as identifiers based on gender, race, age, or location, which could still expose an individual’s identity even without their name.

Yet, Laird cautioned that until other companies commit to similar standards, the agreement illustrates the challenges of establishing AI regulations without legislative backing and also emphasizes that the responsibility of safeguarding student privacy lies primarily with tech companies rather than on schools, students, or parents.

Smith from Microsoft acknowledged that tech companies have been slow to recognize these responsibilities.

“There are crucial discussions and debates occurring nationwide about when, how, and for what purposes AI should be utilized in schools,” Smith remarked. “It is our duty, even our obligation, to provide AI in a manner that instills trust among teachers, students, and their families.”