If you have been reading my articles on Medium, you know I like to run experiments using AI for creative endeavors and write about my experiences. I have covered areas like image generation, creative writing, and music composition. For the first time, I expanded my research into the third dimension. I looked into using commercial and open-source AI tools to create new physical objects and printed them using a 3D printer. In this article, I will show you the steps I took to design and print four different 3D meshes using various commercial and open-source tools. You will find all four objects in the 3D Gallery in the appendix.
Overview
In the following sections, I will walk you through four experiments I ran using different tools for 3D-object generation and show the results. The first was using commercial tools: Midjourney for creating a 2D image and a website called 3dMaker.ai for extracting a 3D mesh, a file that contains the geometry for display and printing in 3D. The second experiment used an open-source AI model from OpenAI called Shape-E [1]. The third used an open-source model called MVDream [2], and the fourth used a combination of MVDream and another open-source project called threestudio [3].
In all cases, I started with a text prompt, generated a 3D mesh, and used Blender, an open-source desktop application, to modify and clean up the 3D mesh. I then used desktop "slicer" apps, Ultimaker Cura and PrusaSlicer, to prepare and preview the 3D meshes before printing them at local libraries. If you are interested in printing these objects, you can check them out on my Thingiverse profile page.
After walking through the four examples, I’ll touch on the societal impact and ethics of generating 3-D objects with AI. I will discuss policies of ownership rights from the various services and systems. And I will end with a summary of what I learned from my experiments.
Creating 3D Objects with Midjourney and 3dMaker.ai
For my first experiment, I started with Midjourney, a commercial service that creates images based on text prompts. For more information on this service, check out my previous article.
To create a 2D image, I logged into my Midjourney account and typed in this prompt, "a simple geometric, 3D-printed sculpture, solid white plastic, on a small white plastic pedestal, gray background". It created four thumbnail images, shown below.
All four images are really good. The system knows about 3D Printing; all forms are interesting and printable. I liked the "Figure 8" image in the top-left the best. I then created a larger version of the image using the upscale feature in Midjourney.
Removing the Background with Clipdrop
The next step in creating a 3D mesh from a 2D image is to remove the background from the scene. Many AI models can do this, but I found a free service from stability.ai called Clipdrop that does this well. It’s accessible via a web browser.
As you can see from the sequence of images above, Clipdrop is very easy to use. I went to the site, uploaded my source image, and clicked the "Remove background" button. The system did a great job removing the background, including the shadow but kept all the details of the sculpture. I then downloaded the resulting image.
Creating a 3D Mesh from a 2D Image using 3dMaker.ai
To print a 3d model, I needed to convert the 2D image created by Midjourney into a 3d mesh first. Several open-source models can do this, but I found a commercial service called 3dMaker.ai that does this well for a fee.
To create the 3d mesh, I went to 3dMaker.ai, created an account, and uploaded the 2d image with the background removed. There are two choices for creating a 3d mesh: standard quality for US$25 and high quality for US$40. The site says that standard quality is for hard-surface, organic, highly detailed inputs, and high quality is for organic, really highly detailed inputs. I chose standard quality and clicked the "Generate" button. The results took about 30 minutes, and I downloaded the model in OBJ format, a common format that describes 3D geometry. The results were excellent! It captured the detail of the original image and faithfully constructed the back part of the image, too.
Amazingly, 3D Maker AI created the details of the back of the sculpture based solely on one image of the front view. You can see another example of output from this service later in the article.
3D Printing Services and Local Libraries
The final step for this experiment was printing the 3D mesh. I don’t own a 3D printer, but I was pleasantly surprised to find that many public libraries in the Boston area offer free or low-cost 3D printing services. Residents can use the printers to construct useful things, like cellphone cases and cookie cutters, as well as small decorative items.
The libraries provide different types of access. Some have 3D printers in "makerspaces," collaborative workspaces where residents gather to create, invent, and learn. Other libraries have an online printing service where residents can upload a 3D mesh for printing. The libraries also vary by the services they offer, like maximum print sizes and choices of colors for the printing material.
Printing the 3D Object
In order to print the 3D mesh, I had to prepare a file first. The standard file format for 3D printing is STL, but 3dMaker.ai doesn’t export in that format. So, I used Blender, an open-source desktop application for 3D modeling, to read the OBJ file and export it as STL. I then launched a slicer app called PrusaSlicer and imported the STL file to see a preview of printing.
I printed this using a dark gray filament at the Waltham Public Library’s Makerspace. In 3D printing, filament is the thermoplastic feedstock for fused deposition modeling (FDM) printers. It comes in spools of various materials and is melted and extruded through the printer’s nozzle to build objects layer by layer. Residents are free to print objects using the Waltham Library’s 3D printers. I printed the sample above on their Prusa MK3S printer with a 0.6mm nozzle.
I also printed the piece using white filament at the Woburn Public Library using their online form. They allow prints up to 11" tall, but I printed mine at 6.5" tall because of the print time. They don’t accept jobs that take longer than 10 hours to print. Unlike the other libraries, Woburn charges for the material used in jobs. But they issue a monthly US$5 credit. My total cost after applying the credit was US$3.63. Here are the two printed pieces.
Both printed objects came out great! The one printed in Waltham is only 4.75 inches tall but shows many details from the original Midjourny image. You can see some stratification artifacts due to printing at a relatively small scale. The free print service in Waltham is restricted to creating objects for up to 8 hours. The one printed in Woburn looks better at 6.5 inches tall, with fewer stratification artifacts. But there are a few dark marks on the surface of the piece.
Creating 3D Objects with Shape-E
For my next experiment, I used an open-source AI model from OpenAI called Shap-E [1] that creates 3D meshes from text prompts. The name Shap-E is a play on their DALL-E AI model that creates 2d images from text prompts. Here are some example images from their paper.
The Shap-E Model
In May of 2023, OpenAI released a text-to-3D model called Shap-E. The system was designed to create 3D meshes by generating parameters for implicit functions, enabling the rendering of detailed 3D visuals. They employed a two-step training process, first mapping 3D assets to the parameters and then refining the results with a conditional diffusion model. The approach allows for the efficient production of varied 3D models from text prompts. Each sample takes roughly 13 seconds to generate on a GPU [1]. OpenAI released the source code and weights for the trained models on GitHub under the MIT open-source license.
The system uses three AI models described on OpenAI’s model card.
transmitter– the encoder and corresponding projection layers for converting encoder outputs into implicit neural representations.decoder– the final projection layer component oftransmitter. This is a smaller checkpoint than thetransmittersince it does not include parameters for encoding 3D assets. This is the minimum required model to convert diffusion outputs into implicit neural representations.text300M– the text-conditional latent diffusion model.
There is also an image-conditional latent diffusion model, but I didn’t use it for this project.
Running Shap-E in Python
I used a Google Colab to create a 3D mesh from a prompt. Here’s the Python code that shows how I initialized the system.
from shap_e.diffusion.gaussian_diffusion import diffusion_from_config
from shap_e.models.download import load_model, load_config
import torch
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
xm = load_model('transmitter', device=device)
model = load_model('text300M', device=device)
diffusion = diffusion_from_config(load_config('diffusion'))The code downloads three models: the transmitter, the decoder, and the text diffusion model. All the models are loaded onto the GPU if there is one.
Using Shap-E to Create a 3D Object
I used this code to send a text prompt into Shap-E to generate the latent parameters for a 3D shape.
from shap_e.diffusion.sample import sample_latents
from shap_e.util.notebooks import create_pan_cameras, decode_latent_images
import numpy as np
import random
prompt = "a dolphin"
seed = 0
batch_size = 1
guidance_scale = 15
render_mode = 'nerf'
size = 512
torch.manual_seed(seed)
np.random.seed(seed)
random.seed(seed)
latents = sample_latents(
batch_size=batch_size, model=model, diffusion=diffusion,
guidance_scale=guidance_scale, model_kwargs=dict(texts=[prompt] * batch_size),
progress=True, clip_denoised=True, use_fp16=True, use_karras=True,
karras_steps = 64, sigma_min=1e-3, sigma_max=160, s_churn=0)
cameras = create_pan_cameras(size, device)
for i, latent in enumerate(latents):
images = decode_latent_images(xm, latent, cameras, rendering_mode=render_mode)
display(gif_widget(images))You can see how I used the text prompt "a dolphin" to sample latent parameters using Shap-E. I initialized the random seed to 0, which will always render the same mesh. Changing the random seed to 1, 2, or any other integer would create different variations of the mesh. Here is the resultant image.
It’s pretty basic and has a lot of empty space. But it certainly looks like a dolphin. Here’s the code to export the 3D mesh.
from shap_e.util.notebooks import decode_latent_mesh
for i, latent in enumerate(latents):
t = decode_latent_mesh(xm, latent).tri_mesh()
with open(f'example_mesh_{i}.ply', 'wb') as f:
t.write_ply(f)This code uses the latent parameters to generate a mesh in yet another file format for 3D objects called PLY. Here’s what the mesh looks like in Blender.
It still looks like a dolphin, but the mesh has several problems. You can see layered steps on the object’s body, and the fins appear broken into horizontal slices. Also, the mesh doesn’t have a pedestal to display as a printed object. To address the layering problem, I used a Displacement modifier in Blender, followed by a Smooth modifier, to fatten the shapes and smooth them down. To create a pedestal, I reran Shap-E with the prompt, "a cylindrical pedestal in the style of an ocean wave," and exported the mesh. I then positioned and joined the two objects in Blender. Here is the modified mesh.
You can see how I smoothed out the sides of the dolphin and fixed the fins. Shap-E did a good job rendering the pedestal, which needed minimal modifications in Blender. I only added a thin cylinder to the pedestal’s base to help the whole thing sit flat on a surface. Next, I saved the model as an STL file.
Printing the 3D Object
Because there is a lot of negative space under the dolphin, the slicer app added temporary support columns to allow the upper layers to be printed. You can see these in the images below.
I also printed this at the Waltham Public Library’s Makerspace. I printed it on their Prusa MK3S printer. Here is the final piece.
This one came out well, too. Because it was printed small, only 5 inches wide, you can see the stratified lines of the printing process. Also, the dorsal fin at the top is very thin and needs to be better defined.
Creating 3D Objects with MVDream
For my next experiment, I used an open-source project called MVDream, where MV stands for "multiple views." It’s a variant of the Stable Diffusion text-to-image model that creates multiple 2D views of a 3D object given a text prompt. The authors describe the model in their paper [2].
We introduce MVDream, a multi-view diffusion model that is able to generate consistent multi-view images from a given text prompt. Learning from both 2D and 3D data, a multi-view diffusion model can achieve the generalizability of 2D diffusion models and the consistency of 3D renderings. We demonstrate that such a multi-view prior can serve as a generalizable 3D prior that is agnostic to 3D representations. It can be applied to 3D generation via Score Distillation Sampling, significantly enhancing the consistency and stability of existing 2D-lifting methods. – Yichun Shi, et al. [2]
Note that a "prior" in Machine Learning refers to the knowledge the model has before it receives any specific input. MVDream’s multi-view diffusion model acts as a 3D prior that is "agnostic" to specific 3D representations. This means it can generate 3D-like imagery without being tied to any particular format for the 3D data, making it versatile in generating 3D representations from various angles based on textual prompts. Here are some example objects generated by MVDream with the specified prompts.
The images illustrate MVDream’s ability to create multi-view 3D objects from text prompts, showing consistent and detailed renderings from different angles. The examples, including an astronaut on a horse and a carved eagle, reveal the practical application of the model for generating diverse 3D images with accuracy.
I ran MVDream in a Google Colab using the following Python code.
from mvdream.camera_utils import get_camera
from IPython.display import display
prompt = """a 3d-printed Cubist-styled sculpture of a male bust,
in light-gray plastic, on a simple light-gray pedestal,
dark-gray background"""
num_views = 4
seed = 12
set_seed(seed)
img = t2i(model, prompt=prompt, uc=uc, sampler=sampler, step=100, scale=10,
batch_size=num_views, ddim_eta=0.0, device=device, camera=camera,
num_frames=num_views, image_size=256, seed=seed)
images = np.concatenate(img, 1)
pil_image = Image.fromarray(images, 'RGB')
display(pil_image)The code shows how I used MVDream to generate multiple 2D views of a 3D object from my text prompt. I specified the desired attributes of the object to be generated (a Cubist-styled sculpture of a male bust) and configured the model to produce four different views. It then processed these views to create a single image that combines them side-by-side, displayed using IPython’s display function. You can see the results here.
The images show four angles of a 3D-printed, Cubist-style male bust, each view consistent with the other, highlighting the model’s ability to interpret and visualize the text prompt precisely. The sculpture’s texture suggests a granularity similar to 3D printing, and the placement on a pedestal gives it a display-ready appearance.
Creating a 3D Mesh from Four 2D Images using 3dMaker.ai
Once again, I used the 3D Maker AI service to generate a 3D mesh from the four images. Here are four views of the mesh rendered in Blender.
The output from 3D Maker AI is close to the original form, but there are some noticeable differences. The output from MVDream shows a head with more defined, angular planes, particularly around the facets of the face, which is characteristic of the Cubist style mentioned in the prompt. In the rendered mesh from the 3D Maker AI service, the head form appears smoother, with more gradual transitions between the facial features and the planes of the head. To create an artistic effect, I used the polygon reduction setting in the Prusa slicer to make the head more angular. Here are the results.
It’s subtle, but you can see that the head has more distinct angular triangles than the version shown above. I made this artistic choice to make the piece look more Cubic-styled.
Printing the 3D Model
I printed this mesh at the Watertown Free Public Library’s Hatch Makerspace. They use a different slicer, Unitmaker Cura, but the printer is the same, a Prusa i3 MK3. I used tree supports connected to the buildplate, which minimizes the amount of attachment points on the finished piece.
The tree supports look especially creepy with this 3D piece. However, the software needed a way to support the overhang of the brow while it was being built. It was easy to remove the supports because of the minimal attachment points. Here’s the final piece.
This came out well. The 3D-printed head presents a Cubist style through its angular and faceted surfaces. Cast in a uniform blue, the sculpture emphasizes geometric form over detail, with sharp planes defining the contours of the face. It rests on a simple base.
Creating 3D Objects with threestudio and MVDream
The MVDream model works well for rendering multiple views of a 3D object from a text prompt, but it doesn’t generate a 3D mesh. That’s where threestudio comes in. threestudio is an open-source project that provides a modular framework that allows users to experiment with various text-to-3D and image-to-3D components, including MVDream. Here’s what the authors say in their paper.
We introduce threestudio, an open-source, unified, and modular framework specifically designed for 3D content generation. This framework extends diffusion-based 2D image generation models to 3D generation guidance while incorporating conditions such as text and images. We delineate the modular architecture and design of each component within threestudio. Moreover, we re-implement state-of-the-art methods for 3D generation within threestudio, presenting comprehensive comparisons of their design choices. This versatile framework has the potential to empower researchers and developers to delve into cuttingedge techniques for 3D generation, and presents the capability to facilitate further applications beyond 3D generation – Ying-Tian Liu et al. [3]
Here’s a flow diagram from the paper that shows the major components.
The threestudio pipeline for generating 3D content from text or images using AI consists of several key components. The process begins with generating random camera parameters for optimization, including extrinsic and intrinsic properties and lighting conditions. Geometry defines the 3D object or scene through representations like Implicit Signed Distance Function (SDF), Implicit Density Field, and others. Materials determine the object’s appearance under specific conditions, employing Diffuse and Physically Based Rendering (PBR) types. Background creation options include Neural Environment Maps, Textured Maps, or Solid Colors. Rendering is handled by various rasterizers, considering the geometry and materials to produce the final image. Guidance from diffusion models like DeepFloyd-IF and Stable Diffusion uses text or image inputs to direct the optimization process toward generating the desired 3D content. This structured approach allows for the modular generation of 3D representations from textual or visual inputs [3].
Using threestudio and MVDream to Create a 3D Object
I ran threestudio with MVDream in a Google Colab, but the model required a GPU with more than 16 Gigs of VRAM. So, it only runs with a Colab Pro subscription to access an A100 GPU.
After installing threestudio and the MVDream extension, I used this code to create a 3D object from a text prompt.
prompt = """a 3d-printed abstract sculpture with geometric shapes,
in light-gray plastic, on a simple pedestal"""
!python launch.py --config custom/threestudio-mvdream/configs/mvdream-sd21.yaml
--train --gpu 0 system.prompt_processor.prompt="$prompt" seed=42I defined the prompt and ran the launch.py script, indicating using the configuration file to use MVDream. The operation is to "train" the system, which means it runs an optimization loop to create a checkpoint file that defines the 3D geometry based on the text prompt. I set the seed to 42 to get the same output consistently. Changing the seed number will create different variations. It took around 40 minutes to run the script.
Creating a 3D Mesh from the Trained Checkpoint
While running training optimization, the system renders images showing the status of the shape being formed. Here is the resultant 3D object from the prompt, "a 3d-printed abstract sculpture with geometric shapes, in light-gray plastic, on a simple pedestal."
Wow, this is pretty cool! The threestudio system using MVDream for guidance produced a sculpture with three stacked triangular forms placed precariously on a pedestal. The texture of the surfaces seems weathered.
Printing the 3D Object
The next step in the process is exporting a 3D mesh that defines the shape. Here is the command I used for this step.
!python launch.py --config "$save_dir/../configs/parsed.yaml" --export
--gpu 0 resume="$save_dir/../ckpts/last.ckpt"
system.exporter_type=mesh-exporter
system.geometry.isosurface_method=mc-cpu
system.geometry.isosurface_resolution=256
system.exporter.save_texture=False system.exporter.fmt=objThis Python script runs the export command using the previously generated checkpoint file. I told it to use the mesh exporter. The "MC" method is for the "marching cubes" method of rendering, which produces a higher resolution mesh to get a more detailed model. I also indicated that I only wanted the OBJ file and not a texture map to make the process faster. It took 81 seconds to run this step. Here is the resultant 3D mesh rendered in Blender.
It looks like the rendering above but is bumpier and has some extra material at the bottom. I cleaned up the mesh in Blender, removing most of the pedestal and adding a tapered cube as a replacement.
Printing the 3D Model
I printed this mesh at the Hatch Makerspace in Watertown using their Prusa i3 MK3. I used regular supports attached to the mesh’s lower part.
The images above show the settings I used for the Cura Slicer, the build in progress, and the finished piece before I removed the supports. Here is a picture of the finished object.
This came out well, too. The texture is quite bumpy, and you can see some of the residual remnants of the supports. But overall, it looks good. If I were to print it again, I would use tree supports only connected to the buildplate.
Now that I have walked you through how I created four 3D objects using AI, I’ll discuss two important aspects: the societal impact of 3D object generation using AI and the ownership rights of the systems I used.
Societal Impact of 3D Object Generation Systems
The development of 3D object generation marks a significant step in technology, allowing individuals to convert their ideas into physical models. This process, facilitated by commercial and open-source tools, brings a new dimension to creative expression and manufacturing. Yet, alongside its opportunities, it’s essential to consider the societal impacts. These technologies offer a clear example of how innovation can have positive and negative consequences, highlighting the need to carefully consider their use and implications.
Societal Impact of Shap-E
In their paper, OpenAI discusses biases in their training dataset that will likely impact the behavior of the models they developed. For example, they examined bias within their text-to-3D model by providing ambiguous captions in which specific details, such as body shape or color, are left unspecified. They observed that some samples generated by their model exhibited common gender-role stereotypes in response to the ambiguous prompts. You can read more about it in Appendix C of OpenAI’s paper [1].
Societal Impact of MVDream
The MVDream creators discussed the societal impact of their model in their paper.
The multi-view diffusion model proposed in this paper aims to facilitate the 3D generation task that is widely demanded in gaming and media industry. We do note that it could be potentially applied to unexpected scenarios such as generating violent and sexual content by third-party fine-tuning. Built upon the Stable Diffusion model [4], it might also inherit the biases and limitations to generate unwanted results. Therefore, we believe that the images or models synthesized using our approach should be carefully examined and be presented as synthetic. Such generative models may also have the potential to displace creative workers via automation. That being said, these tools may also enable growth and improve accessibility for the creative industry. – Yichun Shi, et. al [2]
In navigating the societal impacts of text-to-3D models, we are reminded of the careful balance between harnessing AI’s potential for innovation and addressing the ethical, cultural, and economic ramifications accompanying its rise.
Ownership Rights of 3D Object Generation Systems
Ownership rights are a crucial aspect of media generation, necessitating a thorough review of the terms of service for each tool to comprehend the rights granted to creators.
Ownership Rights of Midjourney Users
Midjourney recently changed its policy on the ownership of images generated using its service. The company used to require a paid subscription for users to own the images they created. But this has been relaxed for individual users. Here is their updated policy.
You own all Assets You create with the Services to the fullest extent possible under applicable law. There are some exceptions:
- Your ownership is subject to any obligations imposed by this Agreement and the rights of any third-parties.
- If you are a company or any employee of a company with more than $1,000,000 USD a year in revenue, you must be subscribed to a "Pro" or "Mega" plan to own Your Assets.
- If you upscale the images of others, these images remain owned by the original creators. – Midjourney
So, individual users will own the images they create. But if you work for a company that makes more than US$1M, you have to pay US$60 per month to own your images. Details on pricing are here.
Ownership Rights of 3dMaker.AI Users
The ownership rights when using 3dMaker.AI are simple. The FAQ on the site says, "The models generated from 3dMaker.AI are 100% yours." You don’t need a lawyer to parse that! 😊
Summary
In my project, I experimented with transforming ideas into 3D-printable objects using commercial and open-source AI tools. Starting with generating 2D images via Midjourney and converting these into 3D models with 3dMaker.ai, I explored the creative process from digital conception to physical creation. Open-source models like Shape-E, MVDream, and threestudio further expanded the possibilities, allowing for direct text-to-3D transformations.
The process involved refining the generated models in Blender, preparing them for printing, and then bringing them to life using 3D printers at local libraries. This journey showcased the technological advancements in AI and 3D printing and highlighted the importance of considering the societal impacts and understanding the ownership rights of these emerging tools.
As I navigated through creating and printing 3D objects, the project underscored the blend of innovation and accessibility in the field, prompting reflections on the ethical and practical implications of using AI in creative processes.
Source Code
I posted this project’s source code on GitHub. And I posted the 3D designs on both Sketchfab and Thingiverse. I am releasing the code and designs under the Creative Commons Attribution Sharealike license.
Acknowledgments
I thank Jennifer Lim for reviewing this article and providing feedback. I would also like to thank the staff at the Waltham Public Library Makerspace, the Watertown Free Public Library’s Hatch Makerspace, and the Woburn Public Library for their assistance in printing the objects for this article.
References
[1] Heewoo Jun and Alex Nicho, Shap·E: Generating Conditional 3D Implicit Functions (2023)
[2] Yichun Shi et al., MVDream: Multi-view Diffusion for 3D Generation (2023)
[3] Ying-Tian Liu et al., threestudio: a modular framework for diffusion-guided 3D generation (2023)
[4] Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models (2022)
Appendix A: 3D Gallery
Here are the 3D objects that I created for this article. Feel free to interact with them here or download the STL files.
