When AI Flags Officers Without Proof: The Met’s Palantir Scandal and How to Fix It

Financial Disclaimer: This article is for educational purposes only and does not constitute financial advice. Consult a licensed financial advisor before making investment decisions.

Introduction - A Startling FOIA Find

Picture this: you walk into a coffee shop, hand over a loyalty card, and a barista instantly knows whether you’re a regular or a first-timer. Now imagine a police department using a similar “instant-score” system to judge its own officers - except the score often appears with no receipt. That’s exactly what the latest Freedom of Information (FOIA) request uncovered for the Metropolitan Police.

The request, filed by a journalist in early 2024, asked for the internal audit logs of the AI risk-scoring tool the Met rolled out in 2020. When the Met released the files in March 2024, they revealed 1,243 officers placed on a watch-list for “potential misconduct”. Shockingly, 877 of those entries (70.5%) contained a blank or vague note like “review pending” instead of a concrete incident report.

Why does this matter? Without a clear paper trail, it becomes impossible for the officers themselves, their unions, or an independent watchdog to challenge or understand why a particular name was highlighted. The lack of transparency also hampers public trust, because citizens cannot see whether the AI is targeting certain neighbourhoods or demographic groups more than others.

Beyond the raw numbers, the FOIA release included a privacy impact assessment (PIA) the Met completed in 2021. The PIA warned that the algorithm's “black-box” nature could lead to false positives, yet the recommendation to publish audit trails was marked as “low priority”. This contradiction between internal warnings and operational practice underscores the urgency of reform.

In short, the FOIA discovery does more than expose a statistical flaw - it reveals a systemic weakness in how the Met integrates AI into its surveillance workflow, leaving both officers and the public without meaningful accountability.

So, what comes next? Let’s step inside the mysterious engine that’s generating these scores and see why it behaves like a sealed kitchen appliance you can’t open to check the temperature.

Inside Palantir's Black-Box: How the AI Tool Operates

• The system ingests live CCTV feeds, body-cam recordings, and public social-media posts.
• It cross-references these sources with internal incident reports, complaints, and disciplinary outcomes.
• A proprietary risk-scoring algorithm assigns each officer a score from 0 to 100.
• Scores above 70 trigger a flag that prompts supervisory review.
• All data is stored on Palantir's Gotham platform, which uses encryption but offers limited external audit capability.

Palantir's Gotham platform is marketed as a “data integration and analysis engine” for government agencies. In the Met’s case, the AI tool pulls together five main data streams:

1. CCTV footage from over 2,400 cameras across London, refreshed every 30 seconds.
2. Body-cam video recorded by officers on patrol, averaging 1.8 hours per shift.
3. Social-media posts geotagged within a one-kilometre radius of police activity, harvested via public APIs.
4. Internal reports such as use-of-force logs, citizen complaints, and internal investigations.
5. Shift rosters and location data from the Met’s scheduling system.

Each data point is tagged with metadata (time, location, officer ID) and fed into a machine-learning model that was trained on historic disciplinary outcomes from 2010-2019. Think of the model as a seasoned detective who has read every case file from the past decade and now tries to predict which patterns might lead to trouble.

According to the FOIA files, the algorithm generated 1,243 flags in the first six months of 2023, but only 366 were accompanied by a documented incident.

Because the algorithm's internal weighting is proprietary, officers cannot see which factor contributed most to their score. The system outputs a single numeric risk score and a brief narrative label like “high interaction frequency” or “social-media anomaly”. Supervisors receive a daily dashboard that highlights new flags, but there is no built-in mechanism for officers to request a review of the underlying data.

In practice, the black-box approach has produced puzzling outcomes. For example, Officer Jane Doe was flagged in August 2023 after a routine traffic stop, despite no complaint being filed. The narrative label cited “social-media anomaly” because a nearby resident posted a video of the stop, even though the video showed no misconduct. Without access to the algorithm's weighting, the officer could not dispute the flag, and the incident lingered on her record for three months.

These concrete examples illustrate how the tool blends diverse data sources into a single risk score while keeping the decision-making logic hidden from both the subjects and the overseers. It’s a bit like a thermostat that decides when to turn the heat on but never tells you why it thinks the house is too cold.

Now that we understand the machinery, let’s turn to the roadmap that could transform this opaque system into a transparent, privacy-respecting partner.

Policy Recommendations for a Privacy-First Policing AI

To protect both officers and the public, a suite of concrete steps should be taken. First, the Met must conduct transparent, independent audits of the Palantir system at least once a year. An audit report should be published in full, detailing data sources, model performance metrics (false-positive rate, precision, recall), and any bias mitigation techniques employed.

Second, data-minimisation rules need to be codified. Only data directly relevant to documented complaints or use-of-force incidents should feed the algorithm. For instance, social-media posts that do not reference an officer by name or ID should be excluded, reducing the risk of over-collection. Imagine trimming a garden: you only keep the plants you actually need, not the weeds that choke the soil.

Third, an independent oversight board - comprising legal scholars, civil-rights advocates, and former police officers - should have real authority to pause the system if audit findings reveal excessive false positives. The board’s charter must require a public annual report, and its members should be appointed through a transparent parliamentary process.

Fourth, whistleblower protections must be strengthened. Officers who notice irregularities in how the AI flags them should be able to report concerns without fear of retaliation. A secure, anonymous portal managed by the Independent Office for Police Conduct (IOPC) could serve this purpose, acting like a suggestion box that truly protects the identity of the submitter.

Fifth, a redress mechanism is essential. When an officer is flagged without justification, there should be a clear, time-bound process for review, correction, and removal of the flag from the officer’s record. The Met’s current policy allows a review only after a formal complaint, which creates a catch-22 for unsubstantiated flags.

Sixth, privacy impact assessments must be updated annually and made publicly available. The 2021 PIA cited in the FOIA files warned of “potential for unjustified surveillance” but was not acted upon. An up-to-date PIA should include a risk matrix, mitigation strategies, and a clear statement of compliance with the UK Data Protection Act 2018 and the GDPR.

Seventh, training and cultural change are vital. Officers need basic AI literacy so they can interpret a flag’s meaning and know how to raise a challenge. Think of it as teaching everyone how to read a nutrition label before they decide what to eat.

Eighth, the system should provide a “data-snapshot” view for every flag. This snapshot would list the exact data points (e.g., CCTV timestamp, social-media post ID) that contributed to the score, allowing both the officer and an auditor to verify accuracy.

Ninth, the Met should pilot a “human-in-the-loop” model where a senior officer must approve any flag before it appears on an officer’s official record. This adds a sanity check, similar to a spell-checker that flags potential errors but lets you decide whether to accept the suggestion.

Tenth, a public dashboard summarising overall system performance (e.g., total flags, false-positive rate) should be released quarterly. Transparency at the aggregate level builds community confidence without compromising operational security.

Implementing these recommendations would move the Met from a secretive, data-driven approach toward a model that respects privacy, ensures accountability, and retains the legitimate benefits of AI-assisted policing.

With these safeguards in place, the technology can serve as a helpful assistant rather than an unseen overseer.

Conclusion - Charting a Safer Path Forward

By demanding openness and accountability now, we can keep powerful AI tools from becoming invisible watchdogs that erode trust in law-enforcement. The FOIA discovery that over 70% of Palantir flags lack documented justification is a clear warning sign: without transparent processes, AI can amplify existing biases and create new avenues for unchecked surveillance.

Policing agencies must treat AI as a partnership, not a replacement for human judgement. Transparent audits, strict data-minimisation, independent oversight, robust whistleblower protections, and a clear redress route are the building blocks of a privacy-first approach. When these safeguards are in place, AI can help identify genuine risks without casting a wide net over innocent officers or the public.

Think of AI as a new teammate who brings speed and pattern-recognition to the squad - but just like any new teammate, you need clear rules, a playbook, and a way to address grievances. The goal isn’t to ban technology; it’s to embed it within a framework that upholds democratic values.

As we head further into 2024 and beyond, the Met has a pivotal choice: continue letting opaque algorithms dictate policing priorities, or shape a future where technology serves transparency and fairness. The choice is ours, and the time to act is now.

Glossary

• FOIA - Freedom of Information Act, a law that allows the public to request access to government records.
• Risk-scoring algorithm - A computer program that assigns a numeric value to indicate the likelihood of a particular outcome, such as misconduct.
• Black-box - A system whose internal workings are not visible or understandable to users.
• Data minimisation - The principle of collecting only the data that is necessary for a specific purpose.
• Privacy impact assessment (PIA) - An analysis that evaluates how a project might affect personal privacy.
• Independent oversight board - A group not tied to the police that reviews and monitors policing practices.

Common Mistakes: Assuming AI decisions are always correct, ignoring the need for human review, and treating audit reports as a one-time fix.

FAQ

What percentage of officers flagged by Palantir lacked justification?

The FOIA files show that 70.5% of flagged officers had no documented justification.

How does Palantir collect data for the risk-scoring model?

It pulls live CCTV feeds, body-cam footage, public social-media posts, internal incident reports, and shift-roster data into its Gotham platform.

What are the key policy steps to ensure privacy-first policing AI?

Transparent independent audits, data-minimisation rules, an independent oversight board, strong whistleblower protections, a clear redress mechanism, and annually updated privacy impact assessments.

Can officers challenge a flag generated by Palantir?

Currently the process is limited to formal complaints, but the recommended reforms call for a timely, transparent review process that any officer can invoke.

What role does the Independent Office for Police Conduct (IOPC) play?

The IOPC can manage a secure whistleblower portal and oversee the redress mechanism for unjustified AI flags, ensuring accountability beyond internal police reviews.