Creating a safer digital space with AI content moderation for Elv.ai

Elv.ai is a technology startup focused on content moderation in the online space, especially on social networks and websites. Major clients include media houses, digital agencies, and public institutions in Slovakia, Czech Republic, and Poland. Every day, the system processes more than 50,000 comments that need to be evaluated. The platform leverages the efficiency of artificial intelligence and the expertise of trained human moderators to create safe and respectful online discussions.

The project was initiated during the COVID-19 pandemic when a large amount of misinformation and hateful comments began spreading across the internet without any moderation.

The goal of anyone creating content in the online space is to gain the highest possible number of interactions and comments. However, with the increasing number of comments, it becomes significantly more difficult to moderate them manually, as it requires a lot of human resources and time. Such a solution becomes unsustainable and creates an opportunity for automation using AI.

We initiated the design phase with an initial research that included interviews with real users. This process helped us identify several critical areas for improvement.

While using the previous tool, users encountered specific difficulties, including: 

• Excessive scrolling through a long list of unresolved issues, often leading to unintentional skipping between comments.
• A cluttered and poorly organized user interface.

These issues obscured essential information crucial for moderators when deciding whether to hide or approve a comment, impacting the efficiency and quality of their decisions.

Based on the information gathered during the research phase, we realized that poorly structured information in the interface could slow users down, so we opted for a clean and minimalist user interface.

We created a so-called focus mode, in which users can address comments from unresolved items one by one without distractions. We also designed a dark mode for the application, which helps reduce eye strain for users working in low-light conditions.

The new application also includes a clear client interface, where clients can view comment processing statistics or resolved comments and, if necessary, change decisions.

After designing the interface, we conducted usability testing, which helped us better structure the information on the comment card. It was shown that users resolve comments faster in focus mode, thanks to the support of keyboard shortcuts.

Moderating discussions primarily involves text processing. This field is addressed by NLP (Natural Language Processing). NLP falls under so-called "Weak AI" and includes types of AI such as the well-known ChatGPT.

When searching for a suitable model, we looked for one that performs well in text classification across multiple languages. 

Considering that elv.ai currently serves multiple languages and plans to expand to additional foreign markets and add support for new languages in the future, a multilingual model is indeed crucial.

Originally, comments on clients' social media managed by elv.ai were moderated solely by "elves." However, human moderators struggled to review a sufficient number of comments during their working hours. When integrating artificial intelligence into this process, we considered how to design a new system in which AI could assist the elves as much as possible while ensuring the quality of decisions. We believe that maintaining human involvement throughout the process is crucial. This allows us to individually assess situations and ambiguous comments, taking into account the actual context of the post and the comment itself.

Elves also conduct random checks within the system. In each service, inspectors review AI decisions, with the system itself recommending a sample of comments for them to check. Through this process, we can monitor quality throughout the entire process and intervene if we notice a decline.

We decided that AI should relieve the elves from handling the worst comments to protect them from toxic content. Additionally, AI can filter out comments that are clearly acceptable. Therefore, we assigned to the elves the comments where AI was not confident enough to classify them into one of the groups.

Setting up the infrastructure was a significant challenge for us. The models we use are quite large and take some time to load. Thus, we considered how to design the infrastructure to be easily scalable during periods of increased traffic while avoiding unnecessary operational costs. We surveyed various cloud service providers and ultimately chose AWS. Here, we attempted to set up, launch, and test the entire infrastructure. The system comprises several smaller applications, all of which must scale up or down as needed.

The biggest challenge was finding a suitable GPU instance and running the application on it. Our DevOps team put in a lot of work to set this up, but we eventually launched the entire system successfully and conducted stress tests. The application typically processes 45,000 comments per day. On the new infrastructure, we managed to process 45,000 comments in 15 minutes, with operational costs approximately 30% lower than the original setup.

Our long-term goal is to develop a high-quality product. While the first iterations have brought a truly stable and efficient solution, it is crucial to continuously measure and improve its quality. The company is currently expanding into new markets, and part of entering a new market involves fine-tuning the existing model to better understand the language and domain. Additionally, it is essential to regularly add new examples from existing languages to the model to better understand new contexts and situations.

Besides removing inappropriate content, it is essential for our clients to understand and engage effectively with their audience. Therefore, we are gradually adding more features to the application for this purpose. One of the first is the ability to automate responses to comments directly from the application or to add a reply to a comment.