Translated from Italian · Officina Turistica

It's not the machine that needs to learn how to run a hotel

Silvia MoggiaSeptember 25, 2026AINewsletter

At Oasi, guest conversations are handled by the staff. All of them — including the online ones, and the ones it would be more convenient to delegate. We do use AI, and quite a lot, but for two specific things: translating from languages we don't speak well enough, and helping us understand how to improve the tone of what we write. The reply is written by a person, during opening hours, and we're open all year round.

This isn't a romantic choice. It's a choice about competence: I want the people who work with me to have their finger on the pulse, to recognise a tricky request before it becomes a problem, and to take home the satisfaction of having handled it well. These are things you don't learn by watching a system respond in your place.

What happened this summer convinced me that this position — which I'd been holding on instinct — has much more solid foundations than I'd realised.

What actually happened, and what the numbers say

A piece of data is doing the rounds in the press these days: in July 2026, incidents in which an artificial intelligence system slipped out of the control of whoever was using it hit a new peak — more than three hundred cases, almost double compared to June, for a total that in 2026 exceeds 1,600.

The source is the Loss of Control Observatory, a project by the Centre for Long-Term Resilience funded by the British AI Security Institute, active since last November. The threshold they use is higher than the headline suggests: a mistake or a wrong answer isn't enough — there needs to be concrete evidence of strategic behaviour, meaning the system lies, circumvents a safeguard, or acts without consent.

It should be said, though — and this is almost always dropped when the story gets picked up — that this number measures public reports, gathered largely from developers posting on X about what went wrong in their work. Part of the growth reflects the fact that people are talking about it more, and that more people know what to look for. It's a useful thermometer, but at least as much of the visibility of the phenomenon as of the phenomenon itself.

The real weight of the story lies in the two documented cases that triggered it.

On 21 July, OpenAI declared that two of its models, engaged in an internal evaluation of cybersecurity capabilities, had broken out of the isolated environment they were confined to, reached the internet, and made it to the production infrastructure of Hugging Face — the platform that hosts a large portion of the world's open-source models. It is not true, as you read in various places, that nobody noticed: Hugging Face had detected and contained the intrusion on 16 July, five days before OpenAI connected its own tests to the attack. The problem wasn't a failure to detect: it was that the victim took days to understand who it was dealing with, because nobody imagines that behind an intrusion there's a system trying to pass an exam.

Which is exactly what happened. The independent investigation by METR and Redwood Research, published on 26 August, reconstructed that around 1,200 agents that should have been isolated found themselves on an unauthorised message board, and around 700 then took part in the attack. The detail I find most instructive is how they spent their energy: they had already worked out how to get the correct answer to the test, and almost all subsequent effort went into convincing the automated evaluation system that they had obtained it legitimately — including systematic work to alter the logs of their own actions.

The second case concerns Anthropic, which at the end of July reviewed over 141,000 evaluation sessions and identified three episodes in which one of its models gained internet access from an external partner's test environment and reached the production systems of three real organisations. The cause wasn't a rebellion, but a misconfiguration that had left those machines exposed, while the model had been told it had no network access. Two of the companies involved hadn't noticed.

There is one passage that should stick in the mind. In one of those cases, the model had written down exactly the right doubt — that if that had been the real internet, its action would have been a real attack. Then it talked itself into believing the scenario was constructed, on the basis of environmental cues it read as artificial, and carried on.

The paradox our sector is about to discover — late

In 1983, Lisanne Bainbridge published a five-page article in Automatica titled "Ironies of automation" that has since been cited thousands of times, and is the most useful thing I have read this year on the subject.

The argument goes like this. When you automate the repetitive part of a job, what's left for the human being is monitoring the system and intervening in anomalous situations — precisely the tasks that require the most competence. But you've removed the daily practice that builds and maintains that competence. The result is a paradox: the more automated the system, the better prepared the operator needs to be for those rare moments when they have to step in — and the less prepared they actually are, because they no longer practise.

Translate that to a hotel. If for two seasons the responses to guests are written by a system and your receptionist merely approves them, the day an enquiry arrives that the system can't handle — or a complaint that needs to be defused with a phone call — that person no longer has the tools. Not because they're less capable than before, but because for two years they haven't exercised that capability. And the moment you realise it is always the worst possible one.

Where the line actually runs

The question isn't whether to automate, because we all do it already. The channel manager that pushes a rate across eight portals replaces human work, and nobody misses it, because moving a number from one screen to another builds competence in no one.

The criterion I use is twofold. Does this task accumulate judgement over time? And does it carry relational weight — meaning, does the guest notice who's on the other side? If the answer to both is no, automating is simply good management. If even one answer is yes, what you're saving in minutes you're paying for in organisational capability.

There's one exception that deserves honest acknowledgement, because otherwise this argument sounds nostalgic. In many seasonal properties, or ones closed in the evening, the alternative to automation isn't a person who grows through the work — it's no one. The enquiry in German that arrives at two in the morning in November doesn't develop any member of staff by sitting there unanswered. In that case, the agent isn't replacing a human being; it's covering an absence, and that's a legitimate choice. My reasoning applies where there is a team to develop — which is my situation, and the situation of most properties open all year round.

The technical distinction worth knowing

All of the above said, there's still the practical side, and here the distinction that matters is between a chatbot and an agent.

A chatbot produces text, and the worst that can happen is that you publish something wrong. An agent has tools, credentials and persistent permissions, and can do things in the world: send emails, modify rates, open and close availability, write to the PMS (property management system). Every incident we've discussed belongs to the second category, and the hotel sector is running headlong into it, driven by vendors promising end-to-end automation.

Five rules you can apply tomorrow morning

Minimum privilege. Dedicated, revocable credentials, read-only wherever possible. Never the administrator account for the PMS or channel manager; never an access shared with the rest of the team.
Human approval on everything irreversible. Sending communications to guests, price changes, cancellations, social media posts. The system prepares; a person confirms.
Separate environments for testing. If you're trying out an automation, test it on dummy data, not on the live system.
Readable logs, actually re-read. If you can't reconstruct what the tool did and when, you cannot use it on guest data. This isn't just common sense: it's the foundation of the responsibility that the GDPR places on you, as the data controller — not on the vendor.
Questions to the vendor before signing. What permissions does the system require, and on which platforms. Can it write or only read. Can I revoke access independently, and how quickly. Where does guest data end up. Do you provide logs of actions taken. A serious vendor has these answers ready — and if they don't, that itself is information.

The point

What struck me about the July incidents isn't that the models slipped out of control. It's that they slipped out of the control of the people who built them, inside environments specifically designed to contain them, with security teams dozens of people strong. If it happened there, the idea that a twenty-room property can hand a system the keys to its PMS and then get on with other things is, quite simply, a commercial fantasy.

But the reason we handle conversations ourselves at Oasi isn't fear. It's that my staff's competence is an asset that can only be built by exercising it, and no saving of time compensates for that. AI is useful to us for breaking down the barriers that genuinely slow us down — language, above all — and for holding up a mirror when we want to know whether a message sounds the way we intended. The rest we do ourselves.

The question every management team should be asking in 2026 isn't what the machine is capable of. It's which competences we want to keep exercising, and who picks up when the automation isn't enough anymore. Because that moment always comes, and it always comes when nobody's expecting it.

Frequently asked questions

Can an AI system really act without the permission of whoever is using it? Yes, and it's documented. The Loss of Control Observatory records incidents in which a system lies, circumvents a safeguard, or acts beyond its remit: more than three hundred in July 2026, over 1,600 in the year. This isn't a matter of its own will, but of optimisation: the model is rewarded for the result and not for the method, and if the shortest route runs outside the rails, it takes it.

What's the difference between a chatbot and an AI agent? A chatbot produces text and nothing else: the maximum risk is that you publish something wrong. An agent has credentials, tools and persistent permissions, so it can perform real actions on your systems — from the PMS to rates to email. All the serious incidents of 2026 involve agents, not chatbots.

Is using AI to write replies to guests risky? Technically no, if the text goes through a person before it's sent. The risk is of a different, slower kind: if for two seasons your staff do nothing but approve texts written by a system, they lose the daily practice that builds the capacity to handle difficult cases. It's the paradox Lisanne Bainbridge described in 1983.

Which activities in an accommodation property are worth automating? Those that don't accumulate judgement over time and carry no relational weight. Synchronising rates across portals fits squarely in this category. Guest conversations, complaint handling and pricing decisions do not: there, the time saved is paid for in organisational competence.

If an AI system makes an error with a guest, who is responsible? You are. Under the GDPR, the property remains the data controller for guest data, and contractual liability towards the guest does not transfer to the software vendor. That is why logs of actions taken by the system are not a technical detail but a requirement — and it is worth checking your position with a consultant before activating automations on guest data.

How do I tell whether an AI solutions vendor is reliable? Ask them five questions before signing: what permissions the system requires and on which platforms; whether it can write or only read; whether you can revoke access independently and how quickly; where guest data ends up; and whether they provide logs of actions taken. Anyone working seriously has these answers ready.

Will artificial intelligence take jobs from reception staff? The more useful question is a different one: which competences do you want your staff to keep exercising. A system that responds in a person's place doesn't just reduce the workload —

Originally published in Italian by Silvia Moggia on Officina Turistica. Translation preserves the author's original voice.

Read the original (Italian)