What a moderation pipeline looks like at zero users

Trust and safety for an app accepting user-generated content is not a feature you add later. App Review Guideline 1.2 requires four things to be in place before submission: a report mechanism, a block mechanism, an EULA-at-sign-in, and a 24-hour response SLA. Beyond Apple, there's the legal floor: 18 U.S.C. § 2258A obligates US ESPs to report any CSAM material they encounter, within 24 hours, to NCMEC. Both have to be true on day one. There is no soft-launch version where these are negotiable.

The shape of the pipeline

Every user-generated artefact passes through three gates before it lands in storage. The order matters: cheap checks first, expensive ones last, dangerous ones unconditionally.

1. Authentication. Every POSTrequires a bearer token from Apple Sign-In. App Attest additionally requires the device to be a real, unmodified iOS device. The combination eliminates 95% of automated abuse before any moderation is run.
2. Azure Content Safety. Every image upload to /v1/images/upload is sent to Azure Content Safety before the blob is stored. The four categories — Hate, Sexual, Violence, Self-harm — are scored 0-7. Rejection threshold is severity ≥ 4. Failed uploads return 422 and the image bytes are never persisted. Text payloads (post captions, comments) hit the same endpoint's text-moderation flavour.
3. PhotoDNA CSAM hash check. Microsoft's hash-matching service against the NCMEC database of known CSAM imagery. Runs after Content Safety on every image upload. Match → file with NCMEC, return 422, never surface the bytes to anyone. The PhotoDNA service requires approval and a credential we apply for separately; the code is wired and fail-open until the credential lands.

What happens after a match

On PhotoDNA match the server fires reportToNCMEC, which either POSTs to the CyberTipline API (when the NCMEC ESP credential is set) or logs a structuredcsam_match event to App Insights for manual filing within the 24h window. Either path is documented in the operational runbook and surfaced on the admin dashboard's CSAM queue.

On Content Safety rejection at severity ≥ 4 in the Sexual category — especially when the subject reads as a minor — the same operational procedure applies. App Review treats this gap conservatively: any reasonable possibility of CSAM is filed.

The reports queue + the SLA

The /v1/reports endpoint accepts reports against posts, comments, and accounts. Each report fires a report_submit event into App Insights and appears on the admin dashboard's queue as "open" until an admin reviews it.

The 24-hour SLA is operational, not algorithmic. The daily ops checklist (see OPS_RUNBOOK.md §3) requires a triage pass once per working day. Triage produces one of three outcomes: resolved (content/account taken down), dismissed (no violation), or escalated (legal review). Every action writes to dbo.admin_actions — an append-only audit log keyed by target, indexed for reverse lookup on appeals.

What survives a deletion

When a user deletes their account, three things stay behind for documented retention periods:

• Reports against the account: 12 months, anonymized after 7 days.
• IP addresses: 90 days for fraud detection and ban-evasion review.
• Diagnostic logs: 90 days.

All three retentions are documented in the privacy policy. The account itself, its posts, its history, its avatar, its Apple Sign-In binding — all of that is purged on a 7-day grace cycle.

Why this is in the ledger

Moderation is the part of the system that is least visible to users and most likely to be examined under load. Writing it down here makes two promises clear: the work was done, and there's a paper trail. Both are useful when something goes wrong.