AI Content Moderation

Leverage AI-powered moderation tools to create safer communities with automated content filtering and intelligent threat detection

Create safer online communities with intelligent, automated content moderation. social.plus leverages advanced AI to scan and filter inappropriate content across text, images, and video, ensuring community standards are maintained without constant manual oversight.

Pre-Moderation

Block inappropriate content before it’s published with proactive AI scanning

Post-Moderation

Monitor and review published content with intelligent flagging and automated actions

Overview

social.plus offers two complementary AI moderation approaches:

Pre-Moderation

Proactive Content Filtering

• Content is scanned before publication
• AI generates confidence scores for detected violations
• Content blocked if confidence exceeds configured threshold
• User must modify content to proceed with posting

Post-Moderation

Reactive Content Review

• Content is scanned after publication
• Uses flagConfidence and blockConfidence thresholds
• Automatically flags content for review or removes violations
• Maintains community safety without blocking legitimate content

Getting Started

1. Enable AI Moderation

   Contact our support team to enable AI content moderation for your application.

2. Configure Settings

   Set up confidence levels and moderation categories through the social.plus Console.

3. Test & Monitor

   Test with sample content and monitor moderation effectiveness through analytics.

AI Pre-Moderation

Prevent inappropriate content from reaching your community with proactive AI scanning. Pre-moderation ensures all content meets your standards before publication.

Current Availability: Pre-moderation is currently available for image content, with text and video support coming soon.

Image Content Detection

Our AI pre-moderation scans all uploaded images for inappropriate content across four key categories:

Content Categories

• Nudity: Detection of explicit or inappropriate nudity
• Suggestive Content: Sexually suggestive or provocative imagery
• Violence: Violent or graphic content detection
• Disturbing Content: Content that may be psychologically disturbing

Configuration

1. Enable Image Moderation

   Navigate to Moderation > Image Moderation in your social.plus Console and toggle “Enable image moderation” to ON.

2. Set Confidence Levels

   Configure confidence thresholds for each category based on your community standards.

3. Test Configuration

   Upload test images to verify your confidence settings work as expected.

Understanding Confidence Levels

Important: Confidence levels significantly impact moderation accuracy. Default settings may produce false positives.

Confidence levels represent the AI’s certainty in detecting specific content types:

• Low Confidence (0-30): High sensitivity, may block legitimate content
• Medium Confidence (40-70): Balanced approach for most communities
• High Confidence (80-100): Conservative filtering, may miss some violations

Recommendation: Start with medium confidence levels (40-60) and adjust based on your community’s needs and false positive rates.

Retaining Rejected Uploads

When pre-moderation refuses an image, the upload fails and the file is gone. There is nothing to look at if the uploader disputes the decision, nothing to tune your thresholds against, and no record that the attempt happened at all.

Turning on reserved for review changes that: social.plus keeps the refused original and tells your backend, over a webhook, the moment a rejection happens. Your moderators can then look at what the classifier actually blocked.

Retention is off by default and applies to images, the only content type pre-moderation covers today. The upload response is identical whether it is on or off, so your apps need no changes.

1. Turn on retention

   Send reservedForReview: true to Update file configuration with an admin token.

2. Subscribe to the rejection event

   Add file.didReject to your webhook subscription. It carries the fileId of the retained file — the upload response never does, so this event is the only way to learn it.

3. Review the file

   Fetch the retained image with Download file, using the fileId from the event. Remove it when you are done with Delete file.

What the uploader sees

Nothing changes. A refused image still returns 403 with error code 400314 (UnsafeContent) and no fileId, whether or not retention is on:

{
  "status": "error",
  "code": 400314,
  "message": "Image uploading failed: Nudity content is not permitted"
}

The message names the first category that breached its threshold: Nudity content is not permitted, Suggestive content is not permitted, Violence content is not permitted, or Disturbing content is not permitted.

What is retained

A retained file is an ordinary file document with status set to rejected. It uses the same fileId, the same file endpoints, and the same network access type as any other file your network owns.

Two differences are worth knowing:

• Only the original is kept. No resized variants are generated, so a download returns the original whatever size you ask for.
• The rejection details are not on the file. Reading the file never returns them. They reach you through the file.didReject webhook only.

Retention covers images, matching pre-moderation’s coverage. A file that is deleted is deleted normally, whether or not it was retained.

Rejection and upload events

Three file lifecycle events are available to webhook subscribers:

Event	Fires when
file.didUpload	A user uploads a file successfully
file.didReject	Pre-moderation refuses an image
file.didDelete	A file is deleted

file.didReject is useful on its own. With retention off, you still receive every rejection — with the classifier’s verdict but no file, because no file was kept. A rejected upload raises this event instead of file.didUpload.

Its rejection object lists every category that breached its threshold with the confidence the classifier assigned, echoes your network’s configured thresholds, and carries the message the uploader saw — enough to tell a marginal call from a clear one, and to tune your thresholds from real traffic.

The rejection details are sensitive. They pair your configured thresholds with the classifier’s scores. Treat the file.didReject payload as admin-only data and do not forward it to end users. This is why reading the file never returns it.

AI Post-Moderation

Monitor and moderate published content with intelligent detection and automated response workflows. Post-moderation provides comprehensive scanning across all content types while maintaining user experience.

All AI post-moderation results are surfaced through the Moderation Feed in the social.plus Console, located under Moderation > Moderation feed.

Text Moderation

Detect inappropriate language, hate speech, and harmful text content

Image & Video

Scan visual content for policy violations and harmful imagery

User Profile Moderation

AI moderation for display names, avatars, and descriptions — see dedicated page

Moderation Feed

The Moderation Feed is the central hub for reviewing AI-flagged content. It is organized into two main workflow tabs:

To Review

The To review tab displays all content that requires moderator attention, organized into sub-tabs:

• Posts and comments — Flagged posts and comments from communities and user timelines
• Messages — Flagged messages from channels and direct conversations
• Users — Flagged user profiles — see AI User Profile Moderation

Each flagged item displays:

• The AI moderation label and detected categories (e.g., “AI Mod: Harassment or bullying”)
• PII detection results when applicable (e.g., URLs, person types)
• The number of user reports (e.g., “1 user”, “4 users”)
• The last flagged timestamp
• Available moderation actions

Available actions for posts and messages:

• Delete post / Delete message — Remove the content
• Clear flag — Dismiss the flag and approve the content

Reviewed

The Reviewed tab shows content that has already been moderated, organized into the same three sub-tabs (Posts and comments, Messages, Users).

Each reviewed item shows the action taken and who performed it:

• Flag cleared by [moderator] — Flag was dismissed
• Deleted by [moderator] — Content was removed
• Reviewed by [moderator] — Content was reviewed without a specific action

The Reviewed tab includes an additional Select moderator filter to view actions taken by a specific team member.

Use the filter dropdowns (All feeds / All Channels, Select creator / Select sender) to narrow down the moderation queue by community, channel, or content creator.

Content Coverage

Posts and Comments

• Text, images, videos, clips, files, and livestream content
• Posts across all communities and user timelines
• Comments and reply chains on posts
• Filter by specific community feed or content creator

Messages

• Text, image, video, audio, and file messages
• Messages from group channels, direct messages, and live chat
• Filter by specific channel or message sender

Text Content Detection

The AI text moderation identifies and handles various types of inappropriate text content. Each category is scored on a severity scale from 0 to 7.

Detection Categories

Each name below is a value for the category field. To set thresholds for one, add an entry like this to your configuration:

{ "category": "SelfHarm", "flagConfidence": 2, "blockConfidence": 5, "moderationType": "text" }

• Hate: Hate speech targeting protected groups
• Sexual: Adult content and explicit sexual references
• ViolenceText: Violent threats, graphic descriptions, or dangerous activities
• SelfHarm: Content related to self-harm or suicidal ideation

See API Configuration for the full request, and Understanding Confidence Scores for the default thresholds.

PII Detection

In addition to content policy categories, the AI performs Personally Identifiable Information (PII) detection to flag sensitive data:

• URL — Links and web addresses embedded in content
• PersonType — References to specific person types or identities

Content that passes all AI checks displays “AI Mod: Passed” in the moderation feed. Content with detected violations shows the specific category labels.

Multimedia Content Detection

Comprehensive Scanning: Our AI analyzes both static images and video content frame-by-frame for maximum protection.

Advanced visual content analysis covers extensive categories, each scored on a confidence scale from 0 to 100. Images and videos share the same set of categories.

Every name in the groups below is a value for the category field. To set thresholds for one, add an entry like this to your configuration:

{ "category": "Explicit Nudity", "flagConfidence": 40, "blockConfidence": 80, "moderationType": "media" }

Categories are grouped here for readability only — the grouping has no effect on configuration, and each name is set independently. See API Configuration for a request containing every category.

Adult Content

• Explicit Nudity, Exposed Female Genitalia, Exposed Male Genitalia
• Explicit Sexual Activity, Sexual Activity, Sex Toys
• Non-Explicit Nudity of Intimate parts and Kissing, Implied Nudity, Kissing on the Lips
• Exposed Male Nipple, Barechested Male
• Swimwear or Underwear, Female Swimwear or Underwear, Male Swimwear or Underwear
• Graphic Female Nudity, Graphic Male Nudity, Illustrated Explicit Nudity

Violence & Harmful Content

• Violence, Physical Violence, Weapon Violence, Weapons
• Blood & Gore, Explosions and Blasts
• Self-Harm, Corpses, Emaciated Bodies
• Visually Disturbing, Air Crash

Substance-Related Content

• Alcohol, Alcoholic Beverages, Drinking
• Drugs & Tobacco, Drugs & Tobacco Paraphernalia & Use, Drug Products
• Pills, Smoking

Extremist & Hate Content

• Hate Symbols, Extremist, Nazi Party, White Supremacy
• Rude Gestures, Middle Finger

Other Restricted Content

• Gambling

User Profile Moderation: AI moderation for user profiles (display names, avatars, descriptions) is covered in a dedicated page. See AI User Profile Moderation for setup, admin reset workflows, blocklist configuration, and moderation feed details.

Understanding Confidence Scores

Each category has two thresholds. Content is removed when its score reaches the block threshold, and flagged for review when it reaches the flag threshold.

Confidence Thresholds

Flag Confidence-Media (Default: 40)

• Content scoring at or above this level gets flagged for review
• Lower values = more content flagged (higher sensitivity)
• Recommended range: 30-60 depending on community standards

Block Confidence-Media (Default: 80)

• Content scoring at or above this level gets automatically removed
• Higher values = fewer false positives
• Recommended range: 70-90 for balanced protection

Flag Confidence-Text (Default: 3, or 2 for SelfHarm)

• Content scoring at or above this level gets flagged for review
• Lower values = more content flagged (higher sensitivity)
• Recommended range: 3-4 depending on community standards

Block Confidence-Text (Default: 5)

• Content scoring at or above this level gets automatically removed
• Higher values = fewer false positives
• Recommended range: 4-5 for balanced protection

Score Ranges

Media Moderation (From 0 to 100 Scale)

• 0-39: Content passes moderation (approved)
• 40-79: Content flagged for human review
• 80-100: Content automatically blocked/removed

Text Moderation (From 0 to 7 Scale)

• 0-2: Content passes moderation (approved)
• 3-4: Content flagged for human review
• 5-7: Content automatically blocked/removed

Note: These ranges use default thresholds and can be customized. For text moderation, thresholds should be integers from 0 to 7. Because the classifier produces integer severity levels, a fractional threshold is effectively rounded up to the next integer—for example, 4.5 behaves as 5. There is no finer-grained score between integer levels.

Default Configuration: Media categories start with flagConfidence: 40 and blockConfidence: 80. Text categories start with flagConfidence: 3 and blockConfidence: 5, except SelfHarm, which flags from 2. Monitor your community’s content patterns and adjust these values to optimize for your specific needs.

Configuration Parameters

Parameter Reference

Parameter	Type	Description
enabled	Boolean	Turns AI post-moderation on or off for the network
commentDeletionBehavior	String	delete_whole_comment (default) or delete_attachment_only
confidences	Array	One entry per category, 1 to 100 entries. Sending this field replaces the entire stored list
category	String	Exact name of the moderation category, case-sensitive
flagConfidence	Number	Threshold for flagging content (0-100 for Media and 0-7 for Text)
blockConfidence	Number	Threshold for blocking content (0-100 for Media and 0-7 for Text)
moderationType	String	Type of content: text or media. Images and videos both use media
userModerationExclusionPatterns	Array	Regex patterns matching user IDs that should never be moderated
avatarCustomUrlEnabled	Boolean	Moderates externally hosted avatar URLs. Off by default

Sending confidences replaces the whole list: the API does not merge your entry into the stored configuration. Any category you leave out is removed and stops being moderated. Always send the complete list, as shown in the example below.

API Configuration

Regional Endpoints

Select the appropriate API endpoint for your region to ensure optimal performance:

Region	API Endpoint
Europe	https://apix.eu.amity.co
Singapore	https://apix.sg.amity.co
United States	https://apix.us.amity.co

Both requests use the same path:

GET  /api/v3/network-settings/enhance-moderation
PUT  /api/v3/network-settings/enhance-moderation

The PUT requires an admin access token. The GET can be read with any authenticated user token.

Note: The settings API is designed with eventual consistency, so a subsequent GET may temporarily return the previous configuration. The successful PUT response is the authoritative acknowledgement of the accepted update and can be used for programmatic verification.

Configuration APIs

Retrieve current configuration

curl 'https://apix.eu.amity.co/api/v3/network-settings/enhance-moderation' \
  -H 'Authorization: Bearer <ADMIN_TOKEN>'

The example above sends every available category at its default thresholds, so it is a safe starting point to copy and edit. Change the flagConfidence and blockConfidence values you want to tune, and remove any category you do not want moderated.

The optional fields enabled, commentDeletionBehavior, userModerationExclusionPatterns, and avatarCustomUrlEnabled can be sent in the same request body, and are written only when included.

API Reference

For detailed administration workflows, see the Moderation Overview and analytics export documentation.

Best Practices

Configuration Strategy

• Start Conservative: Begin with moderate confidence levels and adjust based on results
• Monitor Performance: Track false positive and false negative rates
• Community-Specific: Tailor settings to your community’s content standards
• Regular Review: Periodically review and update thresholds as your community evolves

Human Oversight

• Review Queue Management: Ensure consistent review of flagged content
• Moderator Training: Train team on community standards and edge cases
• Appeal Process: Provide clear paths for users to contest moderation decisions
• Transparency: Communicate moderation policies clearly to users

Performance Optimization

• Batch Processing: Handle high-volume content efficiently
• Regional APIs: Use geographically appropriate endpoints
• Webhook Integration: Implement real-time event handling for flagged content
• Monitoring: Set up alerts for unusual moderation patterns