Imagination Meets Automation With BrickGPT

Fusing artificial intelligence and imagination, researchers from Carnegie Mellon University's School of Computer Science have developed a tool that uses text prompts to help people — and even robots — bring ideas to life with Lego bricks.

BrickGPT takes a simple text prompt, such as "guitar," and creates a brick-by-brick guide for a person or a robot to build a physically stable model of that object. The tool is currently focused on building with Lego bricks, but turning a text prompt into something physically stable goes beyond play.

"It's the perfect blend of the virtual world and the physical world. We are fusing AI and robotics, which can further unleash creativity for humans and impact other industries," said Changliu Liu, an associate professor in CMU's Robotics Institute (RI). "This could be a huge benefit to the manufacturing world. It takes a long time to turn ideas into a physical design and prototype. But if you can integrate generative AI into the process, it can significantly improve efficiency and reduce the roadblocks to kicking off projects."

Currently, the BrickGPT demo can produce step-by-step guides for humans or robots to build 21 models out of Lego bricks, including a birdhouse, sofa and piano. If someone wanted to generate a sofa, they would type "sofa" into BrickGPT, which generates a 3D model. Then, an algorithm transforms the 3D model into brick structures and BrickGPT checks to ensure the structure is stable. A person or robotic arm can follow the steps and build the sofa.

"This research paves the way toward generative manufacturing, which is when people can use a generative model to design everyday objects they can build themselves," said Jun-Yan Zhu, an assistant professor in the RI. "They can build a chair, a sofa or a kids' toy. This is a new frontier, a new usage of these models beyond creating social media videos or photos. These brick toy pieces are a simple medium, and it's a starting point."

To train BrickGPT, researchers generated StableText2Brick, a dataset containing over 47,000 brick structures made from more than 28,000 unique 3D objects accompanied by detailed captions. Researchers took an existing dataset of 3D shapes, ShapeNetCore, and converted these shapes into a grid of small cubes, a process the study describes as voxelizing. They then trained an autoregressive large language model (LLM), which predicts future values based on previous ones. For example, in BrickGPT, the LLM predicts the next brick based on the previous one, ensuring that the structure is stable and won't fall over. If there's an error along the way, BrickGPT goes back and eliminates unstable points to guarantee the structure's stability.

Along with Liu and Zhu, the SCS research team includes Ava Pun, a doctoral student in the Computer Science Department; Kangle Deng and Ruixuan Liu, doctoral students in the RI; and Deva Ramanan, a professor in the RI.

"If a structure is unstable, there's a rollback process," said Pun. "During that step, the model determines which bricks were wrong or unstable and we roll back to the point before that. We detect instability with our physics reasoning algorithm, which generates a stability score for each brick in the structure. If the score is high enough, that means that brick was stable."

Researchers hope to scale up this model, enabling it to generate more than the current 21 objects. They also hope to expand the diversity of their library pieces to increase the accuracy and complexity of generated designs.

For more information, or to test the BrickGPT demo, visit the research website.