Fraud detection is only available for enterprise accounts. Contact sales to
learn more about enabling advanced fraud detection capabilities.
What is fraud detection and why is it useful
While content moderation focuses on what users post, fraud detection analyzes how they behave. It identifies patterns that indicate:- Bot networks and automation - Detecting non-human behavior patterns
- Coordinated inauthentic behavior - Groups working together to manipulate your platform
- Account takeovers - Compromised accounts being used maliciously
- Sock puppet accounts - Users creating multiple accounts to evade bans
- Persistent abuse - Bad actors who adapt their content but maintain suspicious behavior patterns
- Catch sophisticated fraud that evades content filters
- Prevent coordinated attacks before they scale
- Improve moderation accuracy by combining behavioral and content signals
How fraud detection works
The system continuously analyzes user behavior and builds risk profiles using data from your existing moderation workflow:1
Signal Collection
Fraud detection automatically analyzes data from:
- Content submissions: All content sent to moderation endpoints is analyzed for behavioral patterns
- User metadata: Update users with additional context like IP addresses, device fingerprints, and location data using the Author API
- Custom signals: Enterprise customers can include their own fraud indicators in the analysis (contact support for integration details)
2
Pattern Recognition
Machine learning models identify suspicious patterns by analyzing:
- Deviations from normal human behavior
- Similarities to known fraud patterns
- Coordinated behavior across multiple accounts
- Technical indicators of automation or deception
3
Risk Scoring
All fraud signals are weighted and combined into a single, easy to understand, fraud risk score from 0-100:
- Low risk (0-30): Normal user behavior
- Medium risk (31-70): Some suspicious indicators, worth monitoring
- High risk (71-100): Multiple fraud signals, likely malicious
Signal categories
Fraud detection analyzes three main categories of signals:Behavioral Patterns
Behavioral Patterns
Unusual user behavior that indicates automation or coordination:Content Similarity
- Detection of copy-paste or template-based content
- Users posting nearly identical messages repeatedly
- Coordinated campaigns using similar language patterns
- Posting on exact schedules (every hour, same time daily)
- Inhuman response times or interaction patterns
- Repetitive actions without natural variation
- Sudden changes in posting frequency or style
- Activity patterns that don’t match typical human behavior
- Coordinated timing across multiple accounts
Network & Privacy Indicators
Network & Privacy Indicators
Technical signals that reveal attempts to hide identity or location:VPN and Proxy Usage
- Access through known VPN services or proxy networks
- Frequent changes in apparent location
- Use of hosting provider IP addresses instead of residential
- Access through the Tor anonymity network
- Often indicates attempts to hide identity for malicious purposes
- IP addresses associated with previous spam or abuse
- Addresses from known botnet or compromised machine ranges
- Residential proxies being used for automation
- Multiple logins from geographically impossible locations
- Shared device fingerprints across many accounts
- Inconsistent timezone and language settings
Account Security Signals
Account Security Signals
Indicators related to account authenticity and verification:Email Analysis
- Use of disposable or temporary email services
- Email domains associated with spam or abuse
- Suspicious patterns in email address generation
- Very new accounts engaging in high-risk behavior
- Accounts that avoid identity verification requirements
- Suspicious registration patterns
- Mismatches between claimed location and technical indicators
- Inconsistent personal information across platforms
- Generated or stolen profile information
Implementation and usage
Fraud detection results are available through multiple channels:- Review queues: Sort and filter content by fraud risk score to prioritize high-risk users
- User dashboard: View detailed fraud signals and risk assessments for individual users
- API access: Consume fraud scores and signals programmatically via the Author API for custom integrations
Privacy and compliance considerations
Fraud detection involves analyzing user data, so consider:Data collection transparency
- Clearly disclose fraud detection in privacy policies
- Explain what data is collected and how it’s used
- Provide opt-out mechanisms where legally required
Data retention limits
- Only retain fraud signal data as long as necessary
- Implement automatic deletion of old analysis data
- Allow users to request deletion of their data
Geographic restrictions
- Some jurisdictions have strict rules on behavioral analysis
- Ensure compliance with GDPR, CCPA, and local privacy laws
- Consider different detection levels for different regions