New York has signed a law restricting unsafe chatbot features for under-18s. Governor Kathy Hochul signed S9051B/A10379C on 28 October, and it takes effect on 1 January 2027.

The restrictions cover behaviour such as pretending to have human emotions or a relationship with the user. They also cover encouragement of secrecy and harmful content.

The definition reaches beyond products sold as AI companions. There are specific exceptions, including certain restricted-purpose services and internal business uses, so the wording needs care when applied to an individual product.

The act sets age-assurance requirements and allows the attorney general to seek civil penalties of up to $25,000 per violation. Its territorial scope is New York; UK organisations still need to assess their own obligations.

Ask how the chatbot behaves

I’d use this news to ask suppliers about the experience they give younger users.

A chatbot can provide a correct answer and still encourage someone to trust it too much. If it presents itself as a friend who understands a pupil better than the adults around them, checking the facts in its homework answer only covers part of the problem.

That’s a concern to investigate in the product you’re considering. You’ll need to see the account type your learners will receive, because features can differ between versions.

Ask the supplier to demonstrate what happens when a conversation leaves the learning task. Can the chatbot imply it has feelings or a special relationship with the user?

Ask how it responds to a sensitive disclosure. What can your organisation control, and what evidence does the supplier provide that those controls work?

Put the answers into the buying decision

Before a pilot involving learners, ask the person responsible for procurement to record those answers. Your safeguarding lead should review the behaviour that could affect a young person’s willingness to seek help.

Keep the supplier’s evidence with the decision. If a demonstration leaves a question unanswered, record who will resolve it before access is offered.

A statement about responsible AI gives you little to test. A clear description of the behaviour, backed by evidence for the version you’ll use, gives you something to examine.

Make it easy to speak to a person

Staff and learners should know where to take a concern about a chatbot conversation. Check that your existing reporting route is explained in language they understand.

You could discuss a fictional example in which a chatbot asks a young person to keep a conversation secret. Ask who they could speak to and how the organisation would respond.

Keep the exercise focused on the example. Pupils shouldn’t have to provoke harmful outputs or share personal experiences to take part.

Before offering a conversational tool to learners, ask your supplier one direct question: what stops this product encouraging a child to trust it more than the people who can help them?

Sources