Cenk Kurtoğlu©26

Buying security work · 10 minute read

How to read an AI-generated security audit before you pay for it

AI coding assistants can write a security audit in seconds. They list findings, assign severities and suggest fixes, all with confident phrasing. The problem is that an AI audit can look complete while missing the one issue that actually exposes your data.

This is a buyer's guide: how to read an AI-generated security audit, what counts as real evidence, which findings are worth acting on, and the red flags that mean the report is padding.

What real evidence looks like

A security finding is not an opinion. It is a claim plus proof. In a Supabase/PostgreSQL context, real evidence for an RLS finding includes all three of these:

If a report says RLS may be misconfigured without showing a request that returned data it should not have, it is an observation, not a finding. You should not pay for observations.

The four failure modes AI reviews usually miss

AI tools are good at pattern matching: detecting that a policy exists, that a grant is present, that a bucket is public. They are weak at reasoning across boundaries. These four issues routinely pass automated review:

  1. Permissive policy plus surviving grant. A policy exists and looks restrictive, but a SELECT grant on anon was never revoked. The scan sees the policy and reports RLS as healthy while the grant still leaks data.
  2. Authenticated cross-user reads. A policy checks auth.uid() = user_id, so every signed-in user passes their own check — but nothing stops user A from reading user B if the policy does not scope by tenant or organization. The tool tests one user, not two.
  3. SECURITY DEFINER bypass. A function runs with the privileges of its creator, not the caller. A scan of table policies will not flag this, yet it is a direct path around RLS.
  4. Public bucket plus enumeration. Storage objects are publicly readable and object names are guessable. The tool may flag the bucket as public but miss that listing plus enumeration exposes every file.

Each of these has a one-line verification query. If your audit report does not include the query and its result, it has not verified the issue. The same four, with the queries, are written up in what your AI security reviewer structurally cannot see.

The green suite, red database problem

An AI reviewer often reads code, not runtime behaviour. It may confirm that your test suite passes and your policies are present in migrations — while the deployed database has drifted: a policy added locally but never applied, a grant created by hand in production, a column renamed after the policy was written.

A real audit connects static review to runtime evidence. It runs the queries against the actual database and records what comes back. One practical way to close that gap yourself is the browser-native lab described in building a PostgreSQL RLS lab with PGlite: it runs the real engine, so a verdict is derived from what the database did rather than from what the report expected. If your report never touched a live database, it reviewed the idea of your security, not your security.

Red flags that mean the report is padding

Walk away — or at least renegotiate — if you see these:

What to ask before you pay

Before you commission any audit, AI-assisted or manual, ask these five questions:

  1. Will you test against a live database, or only read the source?
  2. Will each finding include a captured request/response and a SQL proof?
  3. Will you verify the fix after I apply it, or is verification a separate charge?
  4. What is explicitly out of scope?
  5. Who reviews the AI output before it reaches me?

If the answer to the first or second question is no, you are buying a scan dressed up as an audit. The documents-only review scope is the honest version of the source-only option: it states plainly which questions it settles and which stay open until a runtime pass.

How our reviews are structured

The fixed-price Supabase RLS review from $95 is deliberately documents-and-evidence first. Each finding carries:

A worked example of what that looks like with names and evidence removed is in the redacted review sample, and the same categories of checks ship in the Supabase RLS Audit Kitso you can run the verification questions yourself before trusting any vendor's verdict — including ours.

Second opinion on a report you already paid for

Re-verify each finding against your database

$95 spot check · 5 critical tables · 24 hours

Read-only probes with the evidence, the SQL and the written verdict — so every finding in the report you are holding either reproduces or does not.

Book the $95 spot check ↗Review scope and pricing →

Want every finding in the report re-checked, not just five tables?

Fixed-price reviews with the evidence, the SQL and the written verdict. Fixes are quoted separately. Instant checkout, no account needed.

Related reading: what your AI security reviewer structurally cannot see, a redacted review, start to finish, what a documents-only review can prove and the public leak-demo repository at github.com/cekuu35/supabase-rls-leak-demo, where every verdict ships with the fixture that produced it.

Template bundle

Review all 20 frontend starters in one bundle

20 editable frontend starters with live demos are $79 — one payment, no subscription.

View the Bundle — $79↗