This video explains the process of AI moving beyond language and image understanding to spatial intelligence, which understands and acts in physical and virtual spaces, following the acquisition and integration of World Labs and SceniX. The key is to build a 'real-to-sim-to-real' pipeline that learns and evaluates in simulation and then transfers to reality by faithfully transferring real-world environments into simulation, rather than having the robot learn indiscriminately in reality. The two teams aim to solve the problem of lack of data for robots and create proven automation cases that can be used in actual industrial sites within the next two years.

1. The next step in AI is spatial intelligence

The video begins by introducing the new frontier of AI as spatial intelligence. For humans and machines to perceive space, reason in it, and actually act in it, simply understanding language is not enough.

World Labs co-founder and CEO Fei-Fei Li describes World Labs as a frontier model research company that is about two years old. The AI ​​created by this company aims to have the following capabilities for both physical and virtual spaces:

• Create space
• Understand the space
• Infer about space
• Ability to interact in space

"The next frontier of AI that we are creating is 'spatial intelligence.'"

The key element that makes this possible is the world model. A world model is a model that helps AI understand not only what the world looks like, but also how it changes over time and what the consequences of certain actions are.

Feifei Li emphasizes that spatial intelligence is not limited to robots. Fields that create and operate within virtual spaces, such as film visual effects, games, and architecture/design, can all be important applications of spatial intelligence.

"The world we live in can become a multiverse, the multiple worlds we create. Technology allows creators and developers to operate in different spaces."

But the ability of AI to act in real physical space is especially important. This is because robots must not only look at space, but also pick up objects, push objects, move objects, and act in accordance with their surroundings. World Labs explains that from the beginning, it has seen robotics as an important application of spatial intelligence and world models.

2. Why World Labs and SceniX met

Yunzhu Li, co-founder of SceniX, introduces his career as a doctoral student at MIT and later researched with Feifei Li at Stanford University. He is currently an assistant professor at Columbia University.

The goals of his research are very practical.

"What I've been doing is very simple: helping robots better perceive the physical world and interact better with that world."

Although Yun Ju-ri is a researcher, she emphasizes her pragmatic inclination to create robots that actually work in the real world. SceniX saw lack of training data and difficulties in evaluation as the biggest bottlenecks in general-purpose robot development.

Language models can be trained on a large scale because of the vast amount of text data available on the Internet. Robots, on the other hand, must move and collect data directly in reality. Because multiple robots must be prepared, remotely controlled by humans, and repeated experiments performed in various environments, data production speed is very slow and costs are high.

The solution proposed by SceniX is a real-to-sim-to-real pipeline.

1. Transfer the appearance and structure of the real environment to digital space.
2. Make real and digital spaces behave as similarly as possible.
3. Generate large amounts of training and evaluation data in digital space.
4. Apply the robot learned in simulation to the real environment.

"If we align what happens in the digital world to what happens in the real world, we can create all the data and assessments we need in the real world in the digital world on a much larger scale."

The important thing here is not simply replicating what reality looks like in 3D. It is necessary to express not only the location and appearance of objects, but also how the environment changes when the robot acts. For example, the robot must be able to reproduce which direction the object moves when pushed, whether it slips when held, and whether it collides with surrounding objects.

3. Marble and robot data bottleneck

The base model being developed by World Labs is codenamed Marble. Marble is a model that takes multiple images or text prompts as input and transforms them into a geometrically consistent 3D world.

"Marble takes one image, multiple images, or text as input and creates a geometrically consistent world."

The world created this way can be expressed using methods such as 3D mesh or Gaussian splatting. The key is not simply to create a believable image, but to ensure that the structure of the space remains consistent even when viewed from multiple perspectives.

This is an important difference from existing image generation models. Visual models can plausibly predict from one scene to the next, but the structure of the space must still be maintained for the robot to act within it.

Ju-Ri Yun explains the problems with existing video prediction models as follows:

"If a robot pushes an object forward and the object suddenly magically disappears, the robot can't really learn what to do."

The biggest problem in the robotics field is the lack of data to apply scaling laws. Like language models, large-scale data is needed in the robotics field to continuously increase data to improve performance, but it is very difficult to collect this in reality.

The combination of World Labs and SceniX is an attempt to address these issues in a complementary manner.

• World Labs has generative model, computer vision, and 3D reconstruction technologies.
• SceniX has expertise in robot learning, simulation, rendering, and real robotic systems.
• By combining the two teams, we can digitize the real environment more quickly and efficiently and use it for robot learning.

Interestingly, SceniX did not initially approach World Labs to discuss an acquisition. The relationship began when SceniX joined World Labs as a customer around the end of 2024, when Marble was first released.

"They initially came to World Labs as customers."

Feifei Li says it wasn't until she looked at the client list that she realized SceniX was the company of her former research colleague Yunju Li. Afterwards, we discovered that there was great synergy between the skills and goals of the two teams, and we naturally evolved into joining forces.

4. Multimodal-based model for robots

In the future, the two teams are open to the possibility of creating a foundation model for robots, that is, a universal foundation model that can be applied to a variety of robots and tasks.

Feifei Lee explains that the latest base model is evolving into an omnimodal model that handles multiple types of input and output, including text, images, and audio. Robot models also need to accept multiple pieces of information together.

• video frame
• Text commands
• image
• Depth information
• Condition of the surrounding environment
• Robot behavior information

In particular, robot models should include action as an important output. Yun Ju-ri explains the robot model from two perspectives.

The first is a model that predicts changes in the environment. When you take a certain action, it predicts how the environment will change after that action.

"If it takes actions as input and predicts how the environment will change, it is a kind of forward simulator."

The second is a policy model that selects actions to achieve a goal. For example, given the goal of "Put the cup on the table," the robot determines what movement it should make.

"Given a goal, a policy model predicts what actions to take in a real environment to get closer to the goal."

These models can simultaneously support a robot's understanding of its environment and action planning, and can then be further trained for specific robotics or industrial tasks.

5. Difference between video model and consistent 3D world

The host asks what is different from the 3D/simulation-oriented approach pursued by World Labs and SceniX, as the approach commonly used by robotics companies these days is based on video models.

Yunju Lee answers that to create a world where robots can learn, consistency is needed first. The consistency referred to here is not simply the level where the screen flows naturally.

• Must be spatially consistent
• Must be consistent in time
• Must be consistent at different times
• Interactions based on robot behavior must also be consistent.

"The worlds we create must be consistent across space and time, across different viewpoints and across different interactions."

Existing image prediction models may be strong at making the image itself believable, but they have limitations in expressing how an object will physically react when a robot actually performs an action. If an object disappears or changes shape suddenly, the robot may learn incorrect behavior.

However, this does not mean that video models are completely excluded. The two teams envision a data flywheel where simulation-based models and real robot policy models exchange data with each other.

1. Initially, utilize the laws of physics and 3D structures to create a consistent simulation.
2. Learn the robot policy from simulation.
3. Run on a real robot and collect new data.
4. Reflect the data back into the model.
5. Over time, physics-based models and learning-based models evolve together.

In other words, in the end, it does not purely use physical laws, nor does it purely use data. It's about finding a middle ground that preserves the fabric of the world while also leveraging the scalability of data and computing.

Yun Ju-ri says her goal is simple.

"My North Star is making robots work. I want robots that actually work in real environments."

A previous survey he and Feifei Li conducted collected about 1,000 tasks that people want robots to do, about a third of which were cumbersome and messy, such as cleaning.

"People really don't want to do such dirty and troublesome work."

So the goal of the research is not just to show off cool demonstrations, but to enable robots to reliably perform the repetitive, demanding tasks that people actually need.

6. Why simulation is needed

The host points out that simulations are bound to be different from reality and asks which is more important: collecting real-world data or simulations.

Yun Ju-ri replies that it is not a matter of choosing between the two. Simulations do not replace real-world data; they are used alongside and complement real-world data.

"It's not a dichotomy between simulation and real data. Together, they make the robot work."

A simulation is a model of the world that predicts how the environment will change based on your actions. This model can be a pure physics simulator, but it can also combine physics-based elements with elements learned from data collected from the real world.

Initially, more physics-based approaches will be used to secure the basic structure of space and interaction. Afterwards, as more and more real-world data is collected, the proportion of learning-based models can be increased.

Fei Fei Li explains the philosophical meaning of simulation by comparing it to human thinking.

"Think about human intelligence. We do a lot of simulations in our heads."

Humans imagine things that haven't happened yet or things that are difficult to test in reality. Before a soccer game, we anticipate various situations and think about how certain actions might change the outcome. This is counterfactual reasoning.

"An important role that differentiates simulation from real data is counterfactual inference. It allows you to run events that have not yet happened, are difficult to happen in reality, or for which there is not enough real data."

Even robots cannot directly experience all possibilities in a real environment. Risky, rare, costly, and situations that have not yet occurred must be learned in advance through simulation.

This is the same reason why self-driving company Waymo is known to use billions of hours of simulation. Pei Fei Li notes that cars are relatively simple among robots, and emphasizes that simulation plays a very important role in robot learning.

The benefits of simulation can be broadly divided into two categories:

Reliability

In order for a robot to operate stably in reality, it must sufficiently experience various states and changes. In a simulation, you can learn by systematically changing the following elements:

• lighting
• friction
• Shapes and types of objects
• Structure of space
• Physical parameters
• Robot speed and movement

This allows you to cover a wide range of situations that are difficult to experience randomly in real life.

Efficiency

In reality, if a person remotely controls a robot to collect data, it may be slower than a human's speed. However, many customers want robots that can work faster than humans.

Simulations allow the robot's behavior to be systematically and quickly replayed. Rather than simply making the robot move faster in reality, you can learn it by taking into account changes in gravity and the environment when the speed increases.

"Simulation can increase both reliability and efficiency."

7. How real customers use it: training and evaluation

SceniX's platform is largely used for robot training and robot evaluation.

First, evaluation refers to measuring how well a particular robot model or policy works. For example, see which model succeeds at a task 90% of the time, whether it succeeds 99.9% of the time, how long it takes to complete a task, etc.

Language models can be evaluated and iteratively improved relatively quickly. However, robots must actually move in the real world, so evaluation is much slower.

• The experiment takes a long time
• There is a risk that the robot may break down or injure people.
• Requires equipment and manpower
• The cost of repeat experiments is high.

"The iteration speed of robot evaluation is orders of magnitude slower than the iteration speed of language models."

If the digital environment is sufficiently aligned with reality, a model that performs better in simulation is likely to perform better in reality as well. This allows customers to evaluate robots in a safe, fast and scalable way.

Simulations are also very useful in terms of training. This is because various conditions that are difficult to collect in reality can be adjusted as desired.

"In simulation, you can control everything and know exactly how much of the distribution you have covered."

These platforms do not create specific robots themselves, but provide an environment to test and learn the brains and hardware of robots made by other companies.

Our customers use different types of robots.

• Single robotic arm
• Fixed robot arm
• Mobile operating robot
• Various grippers
• Complex end effectors
• Two-armed robot or humanoid robot

SceniX's platform aims to be embodiment-agnostic, a structure that is not dependent on a specific body structure or specific model, that is, a structure that is not limited by the shape of the robot. Additionally, rather than forcing a specific AI model, it also supports learning a new model from scratch or additionally learning an existing vision-language-behavior model.

World Labs was already receiving a lot of inquiries from robotics companies after releasing Marble, but at the time, it wasn't able to fully address the demand. Combining with SceniX is a move to connect this demand with actual products and services.

8. Semi-structured environment before humanoids

The host asks whether the prospects for humanoid robots are somewhat exaggerated and that their use is likely to initially spread in limited environments such as warehouses or factories.

Ju-Lee Yun explains that robotics technology generally develops in the following sequence:

1. Fully structured environment
2. Semi-structured environment
3. Unstructured environment

A fully structured environment is a factory or automobile manufacturing line, where the space and location of objects are virtually fully known and controllable. These fields have already been automated for decades.

Semi-structured environments include Amazon warehouses, hotels, and restaurants. Although you can't control the entire environment, you can organize your workspace to some extent and limit the tasks your robot will perform.

On the other hand, home is a representative unstructured environment. The positions of objects constantly change, objects with irregular shapes such as clothes or toys, and even people and pets move together.

"Home is the biggest challenge, especially since I have three dogs and a five-year-old child."

A robot's robustness comes from having enough experience with the situations it might encounter. Therefore, rather than trying to handle every assumption perfectly from the beginning, it is realistic to start with a semi-structured environment that is relatively controllable.

Feifei Li explains that the fact that humanoids mimic the human body is also an important issue. The human body has evolved to survive in a variety of environments, but it may not be optimal for any single task.

"The human body is capable of doing everything to some degree, but it is not the best at any one task."

From a business and technology perspective, specialized robots tailored to specific tasks can be more efficient. However, the challenge that SceniX must solve is not to create a system tailored only to specific body structures, but to create a general-purpose infrastructure that supports a variety of robots and multiple semi-structured environments.

9. How long will it take to reach human-level energy efficiency?

The host points out that the human brain and body are very efficient at moving through 3D space and picking up objects, and asks whether robots will one day be able to perform simple labor with human-level energy efficiency.

Yun Ju-ri replies that it is difficult to reach that level in the near future. This is because a robot operating in the real world is not a single part but a complex system.

• Hardware
• Software
• Robot control system
• AI model
• Motors and sensors
• Coefficient of friction of fingers
• Physical interaction with the environment

All of these elements must fit together.

"A robot that operates in the real world is ultimately a system. Not only the hardware, software, and brain, but also details such as the coefficient of friction of the fingers must all be considered."

Still, he says the pace of progress in the field of robotic learning is much faster than he expected when he started his PhD. It is clear that much progress will be made in the future, but it is also emphasized that it will take a long time to reach human-level efficiency and capabilities.

Feifei Li points out that even today's language models are not as efficient as the human brain. The human brain operates on about 30 watts of energy, but current AI systems require much more power.

"Even today's language models cannot match the efficiency of the human brain, which runs on 30 watts."

However, for certain narrow tasks, AI has the potential to approach humans in terms of power-to-performance ratio. However, the field of robots that move and manipulate objects in physical space is still far from that level.

Yunju Lee says we need measured optimism rather than being overly optimistic or pessimistic about AI and robots.

"The hardest thing in AI today is having the right level of measured optimism."

10. Integration is not rushed, but customer-centered

World Labs and SceniX will work closely together technically, but we do not plan to merge all code and teams immediately.

Feifei Li emphasizes that SceniX is already a team with its own technology stack, customers, and products. Therefore, rather than hastily mixing everything together, we decided to integrate areas that have strong synergies, such as simulation and behavioral condition-based models.

"We will proceed with integration, but we will not rush to mix the entire team into one big salad."

SceniX already uses World Labs' Marble as an internal customer. The two sides will start collaborating around simulation technology and base models for robots, and plan to gradually combine them while maintaining their respective strengths.

Geographically, World Labs will develop into a cross-strait company. The headquarters will be located in San Francisco, and a base will also be established in New York to utilize SceniX's human resources. We plan to install real robots in both offices to test them remotely and develop remote operation technologies that our customers need.

11. Success after 2 years

The host asks specifically what success will look like two years after the acquisition.

PayPay Li's goal is not to make large-scale, vague promises, but to verify performance with real customers in a small number of important industries.

"The goal is for real customers in several important industries to validate that our systems and infrastructure truly help them automate."

These customers become representative reference customers, or 'lighthouse customers', who will lead future business expansion. The idea is that if they achieve results in the actual field, they can expand to other customers and industries.

There is no set time for a customer to engage with World Labs.

Some customers only need a real-to-sim step to digitize the real-world environment and evaluate the robotic system. Other customers want a full real-to-sim-to-real pipeline that trains policies in simulation and runs them on real hardware.

The platform can be used flexibly depending on the customer's situation as follows:

• Moving real work into digital space
• Evaluate the robot model
• Train the model with simulation data
• Additional learning of existing base models
• Execute policies on real hardware
• Collect new data from the real world and improve the model again

The customers we are currently collaborating with are dealing with specific tasks that are closer to real-world deployment rather than a research demo phase. There are many tasks that can immediately create value if automated successfully, and each customer has dozens to hundreds of tasks that are candidates for automation.

Finally, the two say that if you're a robotics company, it's never too early or too late to contact World Labs.

"It's never too early. We want to hear about your use cases."

"If you have a robotics project or a robotics company, contact World Labs. We are open."

Conclusion

The key to this conversation is that the future of robotics isn't just about better hardware or bigger AI models. For robots to operate reliably in the real world, they need a digital environment where they can understand the world, predict the consequences of their actions, and repeatedly learn from various situations.

World Labs' spatial intelligence and Marble and SceniX's robot learning and simulation technology are different pieces to solve this problem. The two teams aim to build a real-to-sim-to-real infrastructure that connects the real and virtual worlds, making robot training and evaluation faster, safer and more scalable.

The ultimate goal is clear, not grand future predictions.

"My North Star is making robots work."