updated 06:43 pm EDT, Thu August 21, 2014
Bot system results in 40-percent spam reduction, rule changes issued in seconds
Social microblogging service Twitter offered details on how it battles with spam on the service, showing off a specialized tool that the company made in the process. BotMaker was built by the team, which has so far been responsible for a reduction of 40 percent "in key spam metrics" since its launch. The company went into spill some of the specifics of its creation, something it needed to make since it offers real-time interactions and an exposed developer API.
The spam approach by the social media company was created with three principles in the mind: preventing spam from being created, reducing visible spam and reducing the reaction time to spam. Battling spammers is a complex situation for the company, as it wants to filter items without latency, while spammers essentially know how the company can react based on its API.
The bot system that Twitter created takes a simple approach to how it deals with spam. Rules come down to two parts, conditions that decide whether the bot should act on an event or not, and the actions it will take for the event in question. However, the company aimed to create a system that not only works swiftly, but allows engineers to step in and make changes without an interruption in the quality of service.
Twitter has several types of bots that work at various stages to try and block spam at the earliest opportunities. "Scarecrow" works in real time to prevent spam from even entering the system. "Sniper" operates in near-real time, picking up on the items that make it through Scarecrow, including machine-learning models are not able to be examined in real time. Other models are run through periodic jobs, which look at data offline. Twitter states it doesn't use these types of jobs for all spam detection, as it isn't effective nor scalable.
Another part of the bot system that's important is the ability for a human to interact with it. The company says that it created a system that allowed engineers to quickly deploy and create new models and rules to combat spam in numerous ways. An "intuitive and powerful interface" was created for BotMaker, which includes language highlights of human-readable syntax, the ability to compose complex derived functions, adding rules without changing the code and editing rules that are deployed in a matter of seconds.
Since introducing the tool, Twitter has seen a 40 percent drop in the amount of spam the company tracks internally. It's also reduced the response time to events, since changes now take minutes instead of days in some cases. While the company has turned to BotMaker as its front-line defense against unwanted content, it is also being used in non-spam capacities for distributed systems.
It appears that Twitter has created a rather complex system that isn't overbearing in order to deal with the gross amounts of spam the social media service witnesses. By taking an approach that shows minimal impact on its systems, Twitter can continue to keep its API in circulation for a number of connecting and sharing activities for its users.