Or Dance of the Sugar Plum Fairies?

Day 23 of Advent of Code had us simulate a group of elves spacing themselves out to plant fruit. This one is a little slapdash, but it’s still fun to look at. The ground and view could have been nicer, but the holidays always limit what I can attempt in a reasonable amount of time.

Day 12 of Advent of Code is a path finding problem, which is ripe for visualization. I used the A* algorithm for my solution, which should get to the closest path as fast as possible. It’s always crazy to watch these algorithms in action as they search and narrow down the solution near the end.

Lessons Learned

This is the first visualization with a large amount of nodes. For just the terrain, there are 5,904 nodes. The renderer uses one giant buffer for storing constants and allows a max of 3 renders in flight at a time. This means I can only use 1/3 of the buffer per render pass. In my original implementation, I was breaking past the 3MB buffer, which at best causes artifacts, and at worst causes slow downs and lock ups. To fix this, I added:

1. The ability to specify the buffer size at initialization.
2. A check after a render pass to fatally crash the app if more than the maximum buffer allowance is used.

Day 10 of Advent of Code had us determine which pixels should be enabled on a broken display. These pixels make a string that is the final input. Some times challenges like this can be interesting to look at, because the puzzle has the display go through a series of iterations before the string comes to form. This puzzle was much more straight forward.

Lessons Learned

My renderer never had a means to update the perspective matrix, leaving me stuck with a near / far ratio of 0.01 to 1000 and a field of view of 60º. I added updatePerspective to allow modification of these values at any time.

I noticed in my previous visualization that memory usage was extremely high. I didn’t think too much of this until this visualization also consumed a lot of memory for no real reason. This is a simple render comparatively. The last time this happened, I was bitten by CVMetalTextureCache taking a reference to the output texture as a parameter:

CVMetalTextureCacheCreateTextureFromImage(
    kCFAllocatorDefault, 
    textureCache, 
    pixelBuffer, 
    nil, 
    .bgra8Unorm, 
    CVPixelBufferGetWidth(pixelBuffer), 
    CVPixelBufferGetHeight(pixelBuffer), 
    0, 
    &tMetalTexture
)

In the above code, the function takes a reference to currentMetalTexture as output. This would cause Swift to never release any previous value in currentMetalTexture, effectively leaking every texture made. Assigning nil to currentMetalTexture was the fix in that case.

But this was not the issue. It felt like another texture leak, because the size was growing quickly with every frame. A look at the memory graph debug should 100,000+ allocations in Metal, so I was on the right track.

Most of the objects still in memory are piles of descriptors and other bookkeeping objects, but they were all stuck inside of autorelease pools. Since the rendering function is just one long async function, anything created inside of an autorelease pool in the function will never get released until the function eventually ends. Wrapping the function in an autoreleasepool closure solved the issue and brought memory consumption on both this visualization and the previous one under control.

Day 5 of Advent of Code revolves around a crane moving crates around to different stacks. This was a great opportunity to try my new 3D renderer for generating visualizations.

What Was Missed?

This was the first attempt at using the renderer, so a proper implementation was going to expose what features I didn’t know I needed.

Animation is a bit weird if you don’t have easing functions. I implemented a small set of functions on the 3D context, so that I can ease in and ease out animations as the crates go up, over, and down.

Rendering text was an easy implementation when using CoreGraphics and CoreText, but for 3D renderers, it gets more complex. I built a createTexture function that generates a CoreGraphics context of a given size, uses the given closure to let you draw as you need, and then converts that to a texture that is stored in the texture registry. There is a bit of overlap here with the 2D renderer, but for now, the utilities exist as copies between the two implementations.

Ooops!

There are a couple of rough edges if you manage to watch through the whole 1 hour video. I try not to rewrite too much of my original solution when I’m creating the visualized variant. I typically add the structures and logic from the initial project and slightly adapt it to work across both the console and visualized versions. Because of that, I’m typically stuck with weird state. If you watch the crates go up and over, they use the height of the tallest stack, even if that stack isn’t traversed. Crates travel further than they need to.

Also, because the movement is generated from a state when the moving crates are removed and not placed in their destination, you’ll some times see crates travel down through their own stack and move across at the wrong height. I’ll chalk that up to a quirk and leave it.

In my previous post, I detailed how I combined CoreGraphics, AVFoundation, and Metal to quickly view and generate visualizations for Advent of Code. With this new set up, I wondered, could I do the image generation part completely in Metal? I have been following tutorials from Warren Moore (and his Medium page), The Cherno, and LearnOpenGL for a while, so I took this opportunity to test out my new found skills.

If you’d like to follow along, the majority of the code is in the Solution3DContext.swift file of my Advent of Code 2022 repository.

Subtle Differences

When using CoreGraphics, I had a check-in and submit architecture:

• Get a CoreGraphics context with nextContext()
• Draw to this context using CoreGraphics APIs.
• Submit the context with submit(context:pixelBuffer)

With 3D rendering, you typically generate a scene, tweak settings on the scene, and submit rendering passes to do the work for you.

Before rendering, meshes and textures need preloaded. For this I created the following:

• loadMesh provides a means to load models files from the local bundle.
• loadBoxMesh creates a mesh of a box with given dimensions in the x, y, & z directions.
• loadPlaneMesh creates a plane with the given dimensions in the x, y, & z direction.
• loadSphereMesh create a sphere with a given radius in the x, y, & z direction.

The renderer uses a rough implementation of Physically Based Rendering. Each mesh is therefore composed of information about base color, metallic, roughness, normals, emissiveness, and ambient occlusion. The methods above exist in two forms: one that takes raw values and one that takes textures.

With the meshes available above, a simplistic node system is used to define objects in the scene. Each node has a transformation matrix and points to a mesh and materials. The materials are copied at initialization, so a mesh can be created with some defaults, but then modified later.

With a scene in place, the process of generating images becomes:

• Modify existing node transformations and materials.
• Use snapshot to render the scene to an offscreen texture and then submit it to our visible renderer and encoding system.

If I wanted to render a scene of spheres of different material types, I can use the following:

try loadSphereMesh(name: "Red Sphere", baseColor: SIMD3<Float>(1.0, 0.0, 0.0), ambientOcclusion: 1.0)

let lightIntensity = SIMD3<Float>(1, 1, 1)

addDirectLight(name: "Light 0", lookAt: SIMD3<Float>(0, 0, 0.0), from: SIMD3<Float>(-10.0,  10.0, 10.0), up: SIMD3<Float>(0, 1, 0), color: lightIntensity)
addDirectLight(name: "Light 1", lookAt: SIMD3<Float>(0, 0, 0.0), from: SIMD3<Float>( 10.0,  10.0, 10.0), up: SIMD3<Float>(0, 1, 0), color: lightIntensity)
addDirectLight(name: "Light 2", lookAt: SIMD3<Float>(0, 0, 0.0), from: SIMD3<Float>(-10.0, -10.0, 10.0), up: SIMD3<Float>(0, 1, 0), color: lightIntensity)
addDirectLight(name: "Light 3", lookAt: SIMD3<Float>(0, 0, 0.0), from: SIMD3<Float>( 10.0, -10.0, 10.0), up: SIMD3<Float>(0, 1, 0), color: lightIntensity)

updateCamera(eye: SIMD3<Float>(0, 0, 5), lookAt: SIMD3<Float>(0, 0, 0), up: SIMD3<Float>(0, 1, 0))

let numberOfRows: Float = 7.0
let numberOfColumns: Float = 7.0
let spacing: Float = 0.6
let scale: Float = 0.4

for row in 0 ..< Int(numberOfRows) {
    for column in 0 ..< Int(numberOfColumns) {
        let index = (row * 7) + column
        
        let name = "Sphere \(index)"
        let metallic = 1.0 - (Float(row) / numberOfRows)
        let roughness = min(max(Float(column) / numberOfColumns, 0.05), 1.0)
        
        let translation = SIMD3<Float>(
            (spacing * Float(column)) - (spacing * (numberOfColumns - 1.0)) / 2.0,
            (spacing * Float(row)) - (spacing * (numberOfRows - 1.0)) / 2.0,
            0.0
        )
        
        let transform = simd_float4x4(translate: translation) * simd_float4x4(scale: SIMD3<Float>(scale, scale, scale))
        
        addNode(name: name, mesh: "Red Sphere")
        updateNode(name: name, transform: transform, metallicFactor: metallic, roughnessFactor: roughness)
    }
}

for index in 0 ..< 2000 {
    let time = Float(index) / Float(frameRate)
    
    for row in 0 ..< Int(numberOfRows) {
        for column in 0 ..< Int(numberOfColumns) {
            let index = (row * 7) + column
            
            let name = "Sphere \(index)"
            
            let translation = SIMD3<Float>(
                (spacing * Float(column)) - (spacing * (numberOfColumns - 1.0)) / 2.0,
                (spacing * Float(row)) - (spacing * (numberOfRows - 1.0)) / 2.0,
                0.0
            )
            
            let transform = simd_float4x4(rotateAbout: SIMD3<Float>(0, 1, 0), byAngle: sin(time) * 0.8) *
                simd_float4x4(translate: translation) *
                simd_float4x4(scale: SIMD3<Float>(scale, scale, scale))
            
            updateNode(name: name, transform: transform)
        }
    }
    
    try snapshot()
}

Or, I can go a bit crazy with raw objects, models, and lights:

Additional Notes

To make the encoding and muxing pipeline work, you must vend a CVPixelBuffer from AVFoundation to later submit it back. Apple provides CVMetalTextureCache as a great mechanism to create a Metal texture that points to the same IOSurface as a pixel buffer, making the rendering target nearly free to create.

Rendering pipelines tend to use semaphores to ensure that only a specific amount of frames are in-flight and don’t reuse resources that are being modified. This code uses Swift Concurrency, which requires that forward progress must always be made, which goes against a semaphore that may hang indefinitely. Xcode is complaining about this for Swift 6.0, but I’ll cross that bridge once I get there.

Model I/O is both amazing and infuriating. It can universally read models like OBJ and USDZ files, but what you discover is that everyone makes their models a little bit differently. As noted above, each material aspect could come from a texture, or from a float value, or from float vector. Even though you get the translation for free, the interpretation of the results can turn in to a large pile of code.