Content Moderation and Filtering for Real-time Messaging
Good News! We’ve launched an all new Chat Resource Center.
We recommend checking out our new Chat Resource Center, which includes overviews, tutorials, and design patterns for building and deploying mobile and web chat.
The Internet is an amazing place full of incredible information and knowledge. Unfortunately, it’s also full of trolls who can make life hard for your users. That’s why Community Sift created Sift Ninja. It’s an API to scan chat threads and forum comments for toxic and abusive language. It can identify the topic, measure severity, and moderate based on your own criteria. And with PubNub BLOCKS, now you add Sift Ninja moderation to your own real-time applications.
The Sift Ninja block lets you build this functionality into your PubNub Data Streams, allowing you to monitor and filter unmoderated content as the data is streamed. For example, if you have a chat app or a public forum, you can automatically moderate and filter published messages before it reaches end users.
Getting Started with Sift Ninja Content Moderation
You need to get your own free Sift Ninja account. After you create your account, log in and click on the menu bar icon in the upper right and choose ‘add channel’, then add a new ‘PubNub BLOCK’ channel. Give the new channel a name something like ‘pubnub_block’, then click ‘install now’ to install the block. The next screen will list your credentials.
Also, I’m assuming you already have a PubNub account, but if you don’t you will need to sign up for your free PubNub account. It takes just a moment, and yes it really is completely free.
In a different tab, go to the Sift Ninja block in the BLOCKS Catalog and click on the ‘try it now’ link then follow the prompts to create a fork of the block into your own account.
Now edit the block code to put in your account name, channel name, and api key where indicated at the top. Save the code and start your block. Then press the ‘publish’ button in the lower left to publish a test message with the text “Hello, Ninja! This is a vulgar-severe phrase.”
You will now see an analysis of the message in the BLOCKS console indicating it is a vulgar severe phrase. You can try typing in a different phrase that has actual vulgar messages.
Now you can go back to the Sift Ninja webpage and follow the prompts to the dashboard. There you can see statistics of how many messages in your account have been filtered and what for.
Content Moderation in Real time
That’s all you need to do. Now Sift Ninja will moderate all messages going through your application. From the Sift Ninja dashboard you can customize the ‘sifters’ installed to look for vulgar language, racist and sexist language, and even personally identifying information.
Each time Sift Ninja will instantly report its analysis back to your application. From there you can block the message, filter out the bad parts, send a note to the offending user, or ban them completely. How you use this information is up to you and your application.