Nvidia has launched GauGAN360, an experimental new online art tool that turns rough sketches into 360-degree environments that can be used in 3D scenes.
The demo – an updated version of Nvidia’s GauGAN AI painting app – lets users paint in the overall form of a landscape, and have GauGAN 360 generate a matching cube map or equirectangular image.
Turn quick doodles into synthetic 360-degree environments for use in 3D scenes
Nvidia describes GauGAN360 as the “next evolution” of GauGAN, its popular AI painting demo.
Launched in 2019, GauGAN uses an AI model based on Generative Adversarial Networks to generate photorealistic images matching users’ rough sketches.
GauGAN does pretty much the same thing, but instead of a conventional 2D image, the result is a cubemap or equirectangular image that can be used as an environment map inside 3D software.
Nvidia’s demo shows its own Omniverse Create software, but the maps could be used in other 3D apps like Blender, or in game engines like Unity and Unreal Engine.
Different brush colours let you paint plants, ground, landscape features and skies
Workflow in GauGAN360 is similar to GauGAN: you paint the rough form of the image you want to generate onto the online template, click the button, and wait for the AI to do the rest.
Different brush colours represent different landscape elements: there are 25 available, divided up into ground elements like mud and sand; land and sky features like rivers, mountains and clouds; and plants.
The controls are pretty basic – a choice of three brush shapes, and a brush size slider – but that’s kind of the point: it’s designed to generate images from very quick doodles.
You can also upload a real image to use as a source, with GauGAN350 generating a synthetic image with the same overall composition as the original.
Limited functionality in the current online demo
In our quick tests, we found that the best results came from blocking out broad areas of a landscape: painting finer strokes to suggest smaller features resulted in some oddly distorted-looking trees, for example.
The current demo also only generates natural landscapes: while there were brush colours for roads and pavements, there were no options to generate buildings or other man-made features.
The output was an 8-bit PNG file, so while it could be used as a background environment, it wouldn’t be suitable for environment lighting and, at 1,024 x 768px, it was fairly low-resolution.
More advanced features to come?
However, Nvidia’s teaser video (embedded at the top of the story) suggests that there may be more features to come: the footage shows options to generate HDRIs as well as low dynamic range panoramas.
The interface shown in the video also includes a Building category not available in the current online demo, with brush colours corresponding to man-made features like houses, bridges and walls.
In addition, the description of the video on YouTube says that GauGAN360 generates 8K panoramas: a much higher resolution than the images we were able to download.
We’ve contacted Nvidia to ask when these options will be available, and will update if we hear back.
However, even in its current form: GauGAN360 is a lot of fun to play around with: as with GauGAN itself, there’s something very satisfying about seeing your rough scribbles turned into fantasy environments.
The technology behind GauGAN also underpins Canvas, Nvidia’s free downloadable AI painting app, so if that gets updated in the same way, the same features could also become available in offline tools.
System requirements and licensing
Tags: , , , , , , , , , , , , , , , , , , , , , , , , ,