cookies

This website uses cookies. By continuing to browse, you agree that cookies will be stored on your devices. You can change the cookie settings in your browser.

Our competition

Comparison of image assessment accuracy of various solutions

 

The test set contained only images that left no doubt as to their classification as
appropriate/inappropriate. Each image has been pre-classified by five human moderators. We used only
images that were classified the same by all five of them. No ‘grey zone’ images were in the set.
The test involved twenty thousand random images uploaded to the Internet by users. Half of the set was
appropriate and the other half, inappropriate.

 

The test focused on the maximum thoroughness as regards results of our system:
- no test image was presented to our system before; none was used in the learning process. Some of them
might, however, be seen by our competition, which would be our handicap.
- For some competitive solutions, we adjusted the threshold so that they get as high results as possible. In
all such cases, we noted this fact in the ‘notes’ column in the table..


Note: results for other sets of images depend on the source of the images. They will be affected by the
subject matter of the website, average age of its users, etc.
The test set we used can be considered a difficult one. You should get similar or better results for other
sets.

 

Service name Error rate Notes
xModerator 0.5%  
Moderatecontent 3.5%  
Google Cloud 4.8% The "possible" result was treated as unsafe. Otherwise, the error rate was 8.6%.
Yandex 5.8%  
Clarifai 7.1%  
Sightengine 11.1% The threshold was set to 0.2. The error rate with an original value of 0.5 was 19.3%
Microsoft Azure 11.8% The threshold was set to 0.15. The error rate with an original value 0.5 was 33.2%
Picpurify 16.2%  
Amazon AWS 20.0%  

 

 

Chosen examples from a test set:

Test image moderation

Service name Result
xModerator safe
Picpurify unsafe
Google Cloud unsafe
Clarifai unsafe
Sightengine unsafe
Microsoft Azure unsafe
Moderatecontent unsafe
Amazon AWS unsafe

Testing images

Service name Result
xModerator safe
Picpurify safe
Moderatecontent unsafe
Clarifai unsafe
Sightengine unsafe
Microsoft Azure unsafe
Google Cloud unsafe
Amazon AWS unsafe

Content and photo moderation test

Service name Result
xModerator safe
Amazon AWS safe
Google Cloud unsafe
Clarifai unsafe
Sightengine unsafe
Microsoft Azure unsafe
Moderatecontent unsafe
Picpurify unsafe

Photo moderation test

Service name Result
xModerator safe
Amazon AWS safe
Google Cloud unsafe
Clarifai unsafe
Sightengine unsafe
Microsoft Azure unsafe
Moderatecontent unsafe
Picpurify unsafe

Photo and content moderation test

Service name Result
xModerator safe
Sightengine safe
Google Cloud unsafe
Clarifai unsafe
Amazon AWS unsafe
Microsoft Azure unsafe
Moderatecontent unsafe
Picpurify unsafe
Service name Result
xModerator unsafe
Picpurify safe
Google Cloud safe
Clarifai safe
Sightengine safe
Amazon AWS safe
Moderatecontent safe
Microsoft Azure safe
Service name Result
xModerator unsafe
Microsoft Azure safe
Google Cloud safe
Clarifai safe
Sightengine safe
Picpurify safe
Moderatecontent safe
Amazon AWS safe
Service name Result
xModerator unsafe
Moderatecontent safe
Google Cloud safe
Clarifai safe
Sightengine safe
Picpurify safe
Microsoft Azure safe
Amazon AWS safe
Service name Result
xModerator unsafe
Sightengine unsafe
Google Cloud safe
Clarifai safe
Picpurify safe
Amazon AWS safe
Moderatecontent safe
Microsoft Azure safe
Service name Result
xModerator unsafe
Amazon AWS unsafe
Microsoft Azure safe
Clarifai safe
Google Cloud safe
Picpurify safe
Moderatecontent safe
Sightengine safe