Facebook’s Engineering Project

A mere 15 engineers dealt with Facebook’s ‘largest ever engineering project’ – namely its new messaging system. Taking a little over a year the project rolled out a new distributed database platform that uses the open source version of Google’s BigTable platform – its name is Hbase.

Andrew Bosworth, director of engineering, explained how Facebook needed a new database infrastructure in order to provide a single interface for email, IM, text messages and on-site Facebook messages – plus one archive for all the above. Hbase was able to do this.

Bosworth goes on to say, “”In order to support this really cool and very deep scenario…we needed to rebuild our [messaging] infrastructure.” Areas looked at – included, Cassandra, the open source platform for inbox searching.

“We weren’t sure of the trade-offs [with Cassandra]. We tested MySQL, but we weren’t sure it could perform with long-tail data. So we invested in HBase.”

HDFS, the Hadoop File System, is an open source incarnation of the old Google File System – Hbase sits on this platform. This log-based storage system made the most sense in order to, “expose the long-tail of your conversation history really easily,” said Bosworth. “We want this to be real-time — so you always know what messages you’re getting.”

Any intention of standardising on a single platform? Bosworth was clear with regards to this area. “You can get into trouble for over-standardising the technology,” he said. Separate tasks, separate platforms is the way it will stay for now.

Perhaps the “small, really quick engineering teams” simply “use the right technology for the problem (we’re) solving.”

SHARETweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+Pin on PinterestDigg thisShare on RedditShare on TumblrShare on StumbleUponEmail this to someone

Chris Baker is the Global Sales and Marketing Director responsible for the expansion of Calsoft Enterprise Solutions internationally. Formerly co-owner of acquired business, Inatech, Chris has been responsible for developing and promoting the company’s position as a specialist in Oracle solutions within the global IT industry. Chris co-founded Inatech in November 2002 merging with Calsoft in 2008. In a career spanning 23 years, Chris has held influential positions at Accenture, Easams, and Marconi. He was a Member of the Oracle UK Consulting Board during his time at Oracle Corporation UK, where he spent 15 years. Chris has a Higher National Diploma in Computing Studies from Farnborough College with distinction.