Creating a safer digital space with AI content moderation for Elv.ai

Challenge

The project was initiated during the COVID-19 pandemic when a large amount of misinformation and hateful comments began spreading across the internet without any moderation.

The goal of anyone creating content in the online space is to gain the highest possible number of interactions and comments. However, with the increasing number of comments, it becomes significantly more difficult to moderate them manually, as it requires a lot of human resources and time. Such a solution becomes unsustainable and creates an opportunity for automation using AI.

Understanding the problem

We initiated the design phase with an initial research that included interviews with real users. This process helped us identify several critical areas for improvement.

While using the previous tool, users encountered specific difficulties, including:

• Excessive scrolling through a long list of unresolved issues, often leading to unintentional skipping between comments.
• A cluttered and poorly organized user interface.

These issues obscured essential information crucial for moderators when deciding whether to hide or approve a comment, impacting the efficiency and quality of their decisions.

User interface for more efficient work

Based on the information gathered during the research phase, we realized that poorly structured information in the interface could slow users down, so we opted for a clean and minimalist user interface.

We created a so-called focus mode, in which users can address comments from unresolved items one by one without distractions. We also designed a dark mode for the application, which helps reduce eye strain for users working in low-light conditions.

Improved customer experience

The new application also includes a clear client interface, where clients can view comment processing statistics or resolved comments and, if necessary, change decisions.

Testing and iteration

After designing the interface, we conducted usability testing, which helped us better structure the information on the comment card. It was shown that users resolve comments faster in focus mode, thanks to the support of keyboard shortcuts.

Creating an AI solutions

Moderating discussions primarily involves text processing. This field is addressed by NLP (Natural Language Processing). NLP falls under so-called “Weak AI” and includes types of AI such as the well-known ChatGPT.

When searching for a suitable model, we looked for one that performs well in text classification across multiple languages.

Considering that elv.ai currently serves multiple languages and plans to expand to additional foreign markets and add support for new languages in the future, a multilingual model is indeed crucial.

Data

During the training of the AI model, we utilized comments gathered from client profiles on social networks over the course of one year. The AI model decides whether to approve or hide problematic comments, which constitute 25 - 30% of all comments.

It is also crucial to consistently evaluate comments with sufficient accuracy. Therefore, when creating the dataset, it was important to ensure balance, i.e., to include equal representation of both classes. Given that the sample size of the data set was sufficiently large, we decided to perform downsampling on the predominant category for the first version of the model.

Integration of AI into comment moderation

Originally, comments on clients' social media managed by elv.ai were moderated solely by “elves.” However, human moderators struggled to review a sufficient number of comments during their working hours. When integrating artificial intelligence into this process, we considered how to design a new system in which AI could assist the elves as much as possible while ensuring the quality of decisions. We believe that maintaining human involvement throughout the process is crucial. This allows us to individually assess situations and ambiguous comments, taking into account the actual context of the post and the comment itself.

Elves also conduct random checks within the system. In each service, inspectors review AI decisions, with the system itself recommending a sample of comments for them to check. Through this process, we can monitor quality throughout the entire process and intervene if we notice a decline.

Utilization of artificial intelligence for comment filtering

We decided that AI should relieve the elves from handling the worst comments to protect them from toxic content. Additionally, AI can filter out comments that are clearly acceptable. Therefore, we assigned to the elves the comments where AI was not confident enough to classify them into one of the groups.

Optimization of infrastructure for application scalability

Setting up the infrastructure was a significant challenge for us. The models we use are quite large and take some time to load. Thus, we considered how to design the infrastructure to be easily scalable during periods of increased traffic while avoiding unnecessary operational costs. We surveyed various cloud service providers and ultimately chose AWS. Here, we attempted to set up, launch, and test the entire infrastructure. The system comprises several smaller applications, all of which must scale up or down as needed.

The biggest challenge was finding a suitable GPU instance and running the application on it. Our DevOps team put in a lot of work to set this up, but we eventually launched the entire system successfully and conducted stress tests. The application typically processes 45,000 comments per day. On the new infrastructure, we managed to process 45,000 comments in 15 minutes, with operational costs approximately 30% lower than the original setup.

Improved speed and efficiency of moderation with AI

• Speed

  The average processing time for a comment decreased to less than 2 minutes from the original several tens of minutes or even hours.

• Availability

  AI doesn't sleep or take breaks, ensuring continuous support and effective task execution even at night. This significantly reduces the need for human resources and thus the associated operational costs.

• Efficiency

  A human moderator can process an average of 300 comments per hour. Combined with AI, a moderator can process over 1,200 comments per hour.

• Scalability

  Solutions based solely on humans or AI have limitations. Humans cannot handle a large influx of comments, and AI poorly adapts to new situations. Their combination allows for a quick and effective response to both issues.

Elv.ai provides the solution:

• 80 %

  Better consistency

• 4 x

  Faster moderation

• 2 x

  Higher accuracy

Future steps and development

Our long-term goal is to develop a high-quality product. While the first iterations have brought a truly stable and efficient solution, it is crucial to continuously measure and improve its quality. The company is currently expanding into new markets, and part of entering a new market involves fine-tuning the existing model to better understand the language and domain. Additionally, it is essential to regularly add new examples from existing languages to the model to better understand new contexts and situations.

Besides removing inappropriate content, it is essential for our clients to understand and engage effectively with their audience. Therefore, we are gradually adding more features to the application for this purpose. One of the first is the ability to automate responses to comments directly from the application or to add a reply to a comment.