DESIGN
Building a culture centered around learning
INTRODUCTION
Shifting a culture
At the conclusion of our research phase, we determined that our area of opportunity was to incentivize engineers to share knowledge. To do so, we needed to improve the capture, retrieval, and use of information to enable engineers to learn and perform at a higher level.
​
In other words, we wanted to find a solution that would shift Meta from a task-centric environment to a learning-centric one.
Design Principles
Thanks to our research we had a good idea of what our solution should look like. We used this knowledge to develop a set of design principles to guide our solution. These acted as goals that we could continually reference when evaluating the success of our prototypes. We also identified metrics that could help us measure each principle.
​
With these principles as guideposts as we began prototyping.
Rapidly prototyping
Early in the design phase, we rapidly prototyped many solutions to evaluate the effectiveness of various strategies. These included things like policy interventions, and AI chatbot, a conversational user interface, and a headset equipped with AR.
​
Reevaluating assumptions
AR seemed like an initially promising solution, and all our desk research pointed to it being the most successful. However, when we began prototyping, participants said the glasses were more distracting than helpful.
Meta also brought up concerns about the cost and capability of current AR technology. We ultimately decided the technology would not bring enough value to make it worthwhile, and adjusted our solution accordingly.
​
It's always challenging to have our assumptions proven wrong, but we considered actual user feedback more valuable than any desk research.
Study Design
One of the main challenges our team faced when prototyping was creating testing conditions that mimicked the conditions of data centers.
​
We identified analogous tasks by defining key traits of data centers. For example, data centers are large spaces with many homogenous object. We were able to replicate parts of this environment in the CMU library because it is full of similar objects (books.)
​
The majority of our tasks in later testing involved debugging an Arduino. Arduino mimicked the technical nature of data center repairs and allowed for multiple methods of repairs, such as debugging via the physical components (hardware) or code (software.)
Multimodal smart assistant
After careful consideration, we concluded that a multimodal smart assistant was the most promising route.
In testing, participants indicated that they found an AI chatbot the most helpful because it did not require them to dramatically change the way they worked.
Through additional research we found that multimodal information capture created a smoother documentation process, so we thought it was important our assistant have two modes of input: voice and text. We expected voice to primarily be used for information capture and text for information retrieval, but flexibility throughout the system allows for individual user preference.
​
The smart assistant uses an LLM to process and summarize information provided by the user. As users input more information in the form of documentation, the pool of information grows and the assistant becomes even smarter.
Meta is already developing an LLM that is trained on internal documentation. We propose that this could be a new application for that technology.
Why LLM?
Through research we knew that one of the primary issues in the current documentation system is that information is distributed across many disparate systems. How could we consolidate all this knowledge?
​
In user testing, we found that participants were able to complete tasks 16% faster when the steps were consolidated and summarized for them, rather than referencing raw documentation. This led us to the conclusion that summarizing information was also important.
​
LLM emerged as the logical tool to use because it is able to effectively consolidate and summarize large pools of information.
​
An LLM can also learn from user input; meaning our system can evolve as the knowledge pool does.
​
Meta is already developing an LLM that is trained on internal documentation. We propose that this could be a new application for that technology.
Early iterations
We quickly created mockups and tested them with our stakeholders (product designers working in the data center space at Meta) and data center technicians. Through several iterations we identified key changes and pain points:
Iteration 1: Initial ticket assesment:
Iteration 3: Presenting the user with troubleshooting options:
Iteration 3: Final review of documentation captured by system:
High level insights from testing and critique:
1. The system should be preemptive. Like a helpful co-worker, the system should assist engineers before they even ask for it.
2. Demonstrate the systems’ value immediately. The system should prove that it is helpful by surfacing relevant information from the very first interaction.
3. Show what the system knows. Engineers will be distrustful of the system at first. In order to build trust, the system should be transparent about when it is and is not listening, what it has recorded, and what it may not know.
FINAL ITERATION
Meet Nova
Nova is an AI-based smart assistant that supports technicians by providing them with fast, real time and relevant support throughout their ticket resolution process
Because it utilizes an LLM, Nova is capable of taking in many different sources of information and surfacing relevant pieces to engineers.
Back to our design principles
Nova is a complex system and it was easy to get distracted by considering every single use case.
As we built out the final iteration of our prototype and considered which flows would be the most valuable to demonstrate, returning to our design principles helped us to focus our solution.
DESIGN PRINCIPLE #1:
To allow for effective retrieval, information in our system must be relevant, digestible, and searchable.
DESIGN PRINCIPLE #2:
To enable the capture of knowledge, the documentation process must have as little friction as possible.
Multi-level impact
We believe Nova has the potential to make significant positive impact on both the business and individual users.
