Hacker News

As a World Labs customer, I suspect they are using existing 3D assets when creating their environments. For one thing, a vendor of 3d models for AI mentioned Word Labs as a customer on a webinar.

I don't know if that undercuts what you're saying entirely. But if they're just protecting the splat in real-time based on the camera path through a 3d world, would the model truly understand that it's a floor?

What is the scenario you're thinking about? This is kinda what happens with VLAs, the latent knowledge of an LLM can help in action generation pipelines, but it's also kinda slow.

Yeah where I've been previously skeptical for using AI in creative ways, basically when these tools get to the point where you have a tight iteration loop, it enables a great creative flow.

You still need the technical proficiency to be able to make edits beyond that, but I think studios would be foolish to let go of their experts in the first place.

It's an overloaded term that different people use for different things. At World Labs we think about three major categories of world models:

- Renderers output pixels or views of a world, either the real world or a generated world

- Simulators predict how the world responds to actions

- Planners predict actions for an agent to make to affect some change in the world

Under this taxonomy, Atlas is somewhere between renderer and simulator. It predicts high-quality views of a consistent 3D world (renderer) but it can also predict explicit 3D to aid in simulation. Atlas doesn't currently predict actions to directly drive a robot, so it isn't natively a planner; but its simulation abilities can help train another model to be planner.

Source: https://www.worldlabs.ai/blog/taxonomy-of-world-models

It's an overloaded term for AI models that have spatial reasoning LLMs currently lack.

Best definition I've heard is: AI systems that can build an internal map of their surroundings to anticipate what happens next and make decisions based on their predictions about the consequences the different actions they can take would have.

There's a bunch of different approaches people are trying:

- World labs (linked in this post) is going down the route of neural 3D representation work (NeRFs, 3D Gaussian Splatting)

- Yann LeCun is pretty famously betting on JEPA architectures (check out the excellent Welch Labs videos for more)

- Google is betting on generative video

- Karl Friston was pursuing 'active interference,' which is just traditional RL techniques with different reward functions

It means everything to everyone.

However essentially a world model is something that has the understanding of 3d world and can generate novel view point given either text or image input.

The use I have seen is for robotics. You feed in the current view and describe the action you want it to do, and then it plans the arm movements. (really useful for softbody manipulation.

There are other meanings. but essentially a world model is able to reason in 3d, rather than text.

It means it builds internal representation of the world it understands (can do physics on/predict/modify) and then renders it.

In addition to frozen time, Atlas can handle some scene motion - see for example the candy New York City in the Camera-Controlled Generation section where cars are moving, or some subtle motion of the waves toward the end of the 1-minute video example.

But this is certainly one area we plan to improve going forward.

Could this be used to replace photogrammtry when accuracy is needed? Photogrammetry requires lots of images and can be brittle, and is slow to compute.

Are y'all using any sort of self-distillation similar to https://self-evo.github.io/ to sharpen representations?

How does it compare to Google's Genie 3 or other similar models? At first glance it does look much more impressive visually

How long does it take to process from input images to model output?

spacial context feature is cool - what are the limitations, if any? What would it take to geo and rotation tag every photo ever taken , combine it into a mass spatial context, run it through atlas and build an entire 3D model of the world?

Please give us access :) We are doing super useful things for museums and travelers and this would be a huge unlock!

Also, big fan of Dr.Li!

It can operate in different modes for different applications.

If you only want to reconstruct the content in the input images without any imagination, Atlas can do sparse 3D reconstruction where it predicts a depth map for each input pixel, without predicting anything occluded or not visible in the input views. This could be used to implement your "fog of war" view.

But sometimes you want the model to stitch together views and imagine a coherent completion; this is a powerful tool for building and authoring generative worlds. For example the 1 minute video we show in the post was built by positioning seven unrelated input images in space and prompting the model to interpolate between them along a hand-specified camera path.

Yes, one of the main goals of this model was to keep 3D consistency as the camera moves around; it can do this even without relying on an explicit 3D point cloud or Gaussian splat representation of the scene.

I believe so. This is not a model that generates pixels frame by frame from user input like genie 3. Instead, there’s an actual 3D scene / structure generated (point cloud, 3dgs) from the input images.

Synthetic training data is easier to acquire