While early AI copilots waited for human instructions, the next generation of autonomous AI agents is designed to work independently.
They can access databases, execute tasks — such as drafting and sending emails, processing invoices, and organising files — and make decisions in real time.
Because these agents can operate at high speeds, a single mistake can affect business operations in seconds. To safely integrate these agents into an organisation, business leaders need a robust plan for managing data, security, and governance.
How to do this was the key discussion during our most recent Sifted Talks, in partnership with intelligent content management platform Box. Our panellists included:
- Mick McNeil, founder and CEO at agentic AI developer Deliverance AI
- Eléonore Hébrard, solutions engineer at Box
- Leo Rosales, cofounder and CTO at fintech and AI payments company Ralio
- Hanah-Marie Darley, cofounder and chief AI officer at AI cybersecurity company Geordie AI
The evolution of AI agents
The main shift from AI copilots to autonomous agents lies in the latter’s ability to make decisions without human prompting.
McNeil compared the process of businesses safely deploying these agents to onboarding a human employee. He explained that trust and autonomy must be earned over time rather than granted immediately.
“Think about day one when you bring in a new salesperson. You give them the scope of the role. You give them access to different systems, and then a manager continues to watch that salesperson do tasks over time,” he said.
“At no point on day one with AI agents should you give full autonomy across business management systems,” he added.
Agents should not bypass privacy or unauthorised access.
While human employees make mistakes, an AI agent's capacity and speed make the potential for damage much larger, said Hébrard.
“When AI shifts from generating text to taking autonomous actions and they have access to databases, the potential blasts of an error escalate very significantly,” she said.
But humans should always remain accountable, even when agents act fully autonomously, added McNeil.
“Humans are the ones who provide the agents the capability and the right to execute things in a certain way.”
Data, context and safeguards
Before an agent can be trusted to execute a task, the business data and context it relies on must be organised, Hébrard said.
“You don't have an AI problem, you have a content problem,” she said. “Agents can hallucinate. They sometimes fail when fed disorganised data and when they find duplicates or outdated files.”
But just feeding correct and clean data to an AI agent isn’t enough. Businesses must build a framework to give agents strict boundaries and continuous oversight, said McNeil.
At no point on day one with AI agents should you give full autonomy across business management systems.
“When you give an agent knowledge, you also have to give it context,” he said. “Just as we define the scope of a role for a human, we must instruct the agent on how to think and act when it looks at that data.”
Organisations using third-party AI models should also approach deployment with scepticism around the model's reliability, said Rosales.
“We [at Ralio] follow zero trust principles. We assume the model is compromised and we build around it,” he said.
“We define what it can and cannot do and we make sure we block the agent when it's trying to do something that’s not in the mandate,” he added.
Accountability and reimagining ‘human-in-the-loop’
Accountability must be established before an agent can even access a database. The success or failure of deploying agents securely depends on bringing in Governance, Compliance, and Risk (GCR) teams before pilots begin, said McNeil.
“That function is either going to be the enabler or the disabler of agents being scaled with success,” he said.
To ensure these teams can actually track and control their agents, there needs to be strict access boundaries, said Hébrard.
“Agents should never have broad access and permissions must be task-specific, time-bounded, and context-aware,” she said.
Beyond basic permissions, controls must be enforced at the content level to prevent agents from surfacing restricted information to the wrong users.
“Agents should not bypass privacy or unauthorised access. We’ve heard some stories where people were being exposed to content [by agents] they should not have been exposed to,” Hébrard added.
There's still a deep requirement to understand the context of a business, how processes are delivered in a human world, and translate that to an agent.
While keeping humans involved is essential, poorly implemented human oversight can actually reduce security, said Darley.
“‘Human-in-the-loop’ has become an overused term in the industry because it means allowing a person to make a decision in an execution chain,” she said.
“The challenge with a lot of teams is that you don't specify who that person is or what information or context they need to have in order to make those decisions,” she added.
Deployment strategies
When it comes to building and deploying these systems, companies often focus too heavily on the technology rather than the business process itself, said McNeil.
“There's still a deep requirement to understand the context of a business, how processes are delivered in a human world, and translate that to an agent,” he said.
Without a detailed deployment strategy that outlines how AI can be used safely, the use of unsanctioned tools increases.
“If you don't give employees something that’s working, they’ll find a way to make it work. The best way to mitigate that risk is to give your employees a security tool to give them the space to run with AI,” said Hébrard.
However, Darley argued that a potentially bigger threat to businesses is the "silent failure" of deployed AI agents.
These are “agents that are operating but organisations don't have an understanding of how they're behaving in real time. Something could happen without raising any alarm bells,” she said.
Humans are the ones who provide the agents the capability and the right to execute things in a certain way.
Successfully and safely deploying these agents means accepting that the margin for error is small, said Hébrard.
“[Organisations should be] enforcing dynamic security control at the content layer,” she said.
“[Organisations should] log every prompt and reasoning step and be able to explain 100% of what happens during audits.
