Friday, March 18, 2016 - 12:30pm to 2:00pm
Location:ASA Conference Room 6115 Gates & Hillman Centers
Facebook will be on-campus on March 17 and 18. PhD Students: Join us for Tech Talks and/or 1:1 Meetings with some of our researchers. We look forward to seeing you! RSVP: Lunch will be provided so please RSVP here. Tech Talk 1: Facebook's data warehouses operate collectively at exabyte scale, but are distributed in regional deployments. Data is mastered in a single region, but copied to other warehouses to enable full queryability. In the prior generation warehouse metadata service, which was deployed in distinct regions (“regional metastore”), this also meant copying the metadata cross-region, and allowed for inconsistencies when metadata was updated in the master region and the copied metadata was not properly updated. I will discuss how Global Metastore improved metadata consistency by storing metadata for all regions (whether master or copy) together, improved scalability and how we built the system to minimize latency while improving fault tolerance. Logging data to the data warehouse has not always been easy at Facebook. Originally, it required programmers to properly Ctrl-A serialize strings for hive. Later frameworks allowed key-value maps to be logged as JSON, with an option for the schema to be exploded into top-level columns. This either led to really large JSON strings being logged, or potentially degenerate column explosion (50k-200k). Beyond the data warehouse, both Scuba and ODS expect data to be sent to them in different formats. Logger is a client-side library that handles log serialization for the warehouse, Scuba and ODS. It requires programmers to generate one config with a strongly typed schema, generates a custom logging class and provides log-time data quality validators. It is built with push-safety in mind, and to allow programmers to call log() once and have their data logged correctly to all backends they have specified. Switching to Logger both improves data quality and leads to data storage and query engine compute efficiencies. Tech Talk 2:In this talk, we'll discuss the first large-scale study of flash-based SSD reliability in the field. We analyze data collected across a majority of flash-based solid state drives at Facebook data centers over nearly four years and many millions of operational hours in order to understand failure properties and trends of flash-based SSDs.