This post is part of a series we wrote after wrapping up the IRIS project.
Design notes part 1: Digital impacting Physical
When building a mixed-reality setup, the digital components impose some limits on the physical aspects. Ideally, you would like to move all components to the realm (digital/physical) in which they are the easiest and/or most effective to implement, but due to the restrictions of technologies this is not always possible. Here are some examples of how the technologies we used impacted our design choices.
Pepper’s Ghost screens
The technology has to work effectively in the box you design. As the Pepper’s Ghost effect works with a 45-degree-angled screen, this screen needs to fit the apartment and its furniture. Furthermore, the effect works best when viewed from the front. Although it has a reasonable viewing angle, it will not work at all when viewed from the sides as you will look onto the edge of the screen. This way, the box needs to be built ‘peep box style’, with only visibility from the direct front.
Most practical, the size of the box is determined by the size of the screen you use for projecting the Pepper’s Ghost. In our case we went for a 15,6“ screen, which meant the effective width of the play area inside the box could be no wider than 34,4 cm.
Lighting
During development we found that lighting can be done both through the digital layer and the physical layer. Atmospheric lighting can be generated by the laptop screen, but also by LEDs in the box. However, too many lights within the box will wash out the visibility of the digital layer.
The box also needs to prevent too much ambient light to flood in from outside the box, as this will reduce visibility as well. In order to do so, we designed the physical box so it is fully enclosed form all sides except the front from where the players look inside. All the lights inside the box are precisely dimmed so that even when they simulate daylight conditions, they are bright enough to light the scene, yet they are dim enough not the wash out the peppers ghost effect. This part took some trial and error to figure out.
Even with these precautions, the effect works best when you avoid any light from coming into the box from behind the player. Therefore the front of the box should not face windows or lamps as this could result in reflections on the screen, spoiling the effect.
This post is part of a series we wrote after wrapping up the IRIS project.
Design notes part 2: Physical impacting Digital
Even before the lockdown measures, the design of the apartment was done digitally to help with rapid prototyping. The designs were then translated to physical form either through model-building techniques or 3D printing. Digitally modelling the physical box meant we could quickly estimate the proportions of the physical box, without having to fully build it first.
The box was built in such a way that all the electronics could be fitted inside the box’ cavities without laying a finger on any material. Besides being a really rapid way of prototyping and conducting ‘form studies’ it also is a great way to save material otherwise spent on making different iterations of the box in real life. Sidenote: it’s a fun fact that prototyping in the physical realm often saves time when building a digital solution or game. In this case it were the digital tools that provided us with quick ways to test our assumptions and prototype ideas before spending time on the physical building process.
The Apartment
The setup was built ‘peep-box‘ style in order to effectively use the Pepper’s Ghost effect. In a Peppers Ghost setup, the character is best able to move along the surface of the screen: along a single 2-dimensional plane parallel to the peppers ghost’ horizontal axis. We can get some some faking of depth into the third dimension (the Z axis) because of everything except the digital character being in 3 dimensions. When scaling the digital content up or down, we are generating the illusion of movement on the Z axis and thus giving ourselves some play area within the given constraints of the physical box. While doing this, you will still have to take into account the occlusion by objects around the screen. As this depends on the viewing angle of the player it takes some trial and error to make it work effectively.
Because of the peep box style setup we needed to place all elements and rooms of the apartment next to each other, which you wouldn’t do when building fully digital. For the same reason we chose to go for a single-room apartment, with the bathroom and the hallway the only other rooms. Those rooms are only used by the character in certain circumstances, most of the action is located in the living room. This meant that the apartment layout is not realistic, but actually almost 1-dimensional.
Controls and UI
A project that would be fully digital would have a separate controller, either a mouse or gamepad. As our box was designed to be fully digital as well as mixed-reality, we chose to have the same button interface in the digital version as in the physical version.
In this way we tried to mimic the physical box experience as much as possible, so the frame/interface of the physical box & its buttons where literally translated to the digital version. Even if you as a player are using a mouse controller, you would still push virtual, ‘physical‘ buttons. This would still (indirectly) give the feel of really interacting with a miniature world in a doll-like house setup.
When testing the digital overlays like UI in the physical box, it quickly became apparent that for the Pepper’s Ghost effect to work properly, certain aspects needed to be exaggerated compared to a fully digital version. For visibility, the icons that denote the needs of the character were made much larger and with thicker lines than we did later in the digital version.
Character design
In the digital version everything had to become more subtle that way. While creating the digital version from the basis of the mixed-reality setup, we discovered that the character style we had developed for overlaying the physical environment did not match the fully digital environment style at all.
On one hand, this is because of the fact that the 3d model of the character we used for the Pepper’s Ghost effect was very basic, based on a crude block-out. This was done because this resulted in the best visual readability in the physical box.
On the other hand, as in the mixed reality setup the character is small and transparent, the expressions and movements had to be exaggerated to be clearly readable to the player. When transitioning to the fully digital environment, these exaggerations came across as too cartoony and overacted. We found that people could no longer connect with the character on an emotional level any more as well.
At this point we could have put a lot of time and effort into adjusting the environment and the assets to match the cartoony style of the character. Instead, decided to completely replace the character with a more emotionally immersive one that would also fit the modelled environment. That second route to us was the obvious choice, as it would take way less time and generate a relatable and attractive character that would fit the needs of a digital version. In the end we used a concept model that we developed into the 3d model.
Cartoon-style character used in the mixed-reality version
More realistic character for the fully digital version
This post is part of a series we wrote after wrapping up the IRIS project.
When the IRIS-box project started, we aimed at developing a mixed-reality box and intended to design both the physical and digital components in parallel. Then we hit the Coronavirus lockdown-measures. This meant a couple of things for our design process, as we couldn’t…
…collaborate on the physical box in person
…playtest a physical box with the target audience
…show a prototype to our project partners.
We had to make a decision on how to continue the project. As it was unclear how the situation would develop, we chose to assume that the duration of the lockdown would be longer than the project itself. We therefore switched to a fully digital representation of the physical box. Still with the intent to move to a mixed-reality setup when possible and before the end of the project. In this way we could worry about the development of the physical components later, but would still be able to playtest with the target audience by sharing the digital version online. It also meant that we could show a fully digital prototype to the project partners, that would still give a pretty good approximation of the final product.
Moving to fully digital
During further development the digital version developed into a complete branche of the project, a digital twin of the physical object. Due to the success of the digital version and the prolonged lockdown measures, the digital version became a deliverable in its own right. It existed in the project alongside the physical version, instead of being just a digital derivative.
As mentioned, one of the biggest advantages of the fully digital version was that we could very easily share it with our target audience.
While building the fully digital version, we were able to quickly implement some designs that might not have come up in the physical development. Like the TV being on, switchable lighting inside the room and ambient lighting from outside falling into the apartment and emphasizing the day/night cycle. Some of these elements, like the TV being switched on, is definitely a bridge too far for the physical box. The fidelity of a fully 3D-rendered digital version can not be matched with the Pepper’s Ghost technology. We believe we squeezed out every bit of technological possibility we could.
Even when we went fully digital for our intermediate iterations, we still kept the physical box in mind. Ideally, the digital version should still reflect what the clients can expect from the physical experience. So keeping a close check with the actual physical box in mind is crucial. In the way we designed it, it should in theory not be a problem for players to switch from an online digital version to the physical mixed-reality box since all elements match 1:1.
When executed well the illusion of Pepper’s Ghost continues to impress, even in the light of recent improvements in AR. However, because of the boundaries between the physical and the virtual realities Pepper’s Ghost images tend to look superimposed in a very artificial way. One way to create more cohesion between the physical and digital is to add uniform lighting.
How scene lighting, projection and a physical model of an apartment can come together to create a scene.
So for a project we’re currently working on we wanted to mix physical and digital space to create an arcade-like interactive box. One of the more simple ways to essentially create a hologram that’s placed inside a physically built environment using a technique known as Pepper’s Ghost.
What’s a Pepper’s Ghost?
Pepper’s Ghost is an illusion technique named after English scientist John Henry Pepper. It used a flat sheet of glass that’s ideally invisible to the viewer to project a ghostly version of a scene or image on the other side of the glass.
So, our way of utilising this technique is essentially pretty simple. We built a box without a top and an opening in the side to look into. Then we use a computer monitor and lay it on top of the box, the screen facing down into the box. Then we place a transparent and reflective plate in the box at a 45 degree angle to the screen to project the images displayed on the monitor into the physical space. At it’s core, there isn’t much more to it than that!
It’s basically just a box, a monitor and a plate of glass!
One interesting thing about this technique is that anything you project this way is placed on a specific plane. This plane starts at the intersection point of the glass plate and the monitor screen and goes straight down as long as the plate is placed properly at 45 degrees.
We want to place this projection plane somewhere in the middle of the room to create depth. The box had to go deeper than the projection plane. In our case. That meant we didn’t use about a third of the monitor for the projection, so we decided to use it in a different way.
Dynamically lighting the scene
A clear example how we can use the part of the screen that isn’t projecting for dynamic scene lighting
We can use the part of the screen that isn’t projecting to light the back of the box in whatever way we want. Because it’s just a monitor we can change the color, strength and placement of the light dynamically. Both the lighting of the space and the projection are digital in this case. This means we can relatively easily create cohersion between the physical space and the digital projection by having the lights dynamically influence the projected models or characters.
Here you can see how the 3D models being projected are influenced by the moving lights behind it.
Masking and other clever tricks
Mixing the physical and the digital elements of the box is key to create cohersion between the two. To do this we want the digital to be aware of the physical in more ways than just “lighting”.
To try to do this, we can make the illusion the projection moves behind and inside of things:
Two things are used in this case: masking and the physical object’s position in comparison to the projection.
To create the idea the character is walking behind the blue cube, we digitally mask the projection to appear black when the character moves there. The projection normally appears on top of everything, so it won’t be perfect, but it creates the right idea.
By putting a physical object on the projection plane, but still letting it appear, it will look like the character is inside the physical object, as if it were a separate room.
Things to keep in mind
I think there’s a lot of potential in this technique, but there’s a few challenges that are important to keep in mind. The main one here is that the effect starts to fade if there is too much external light coming in. In our experiments it still worked fine in an averagely lit room, but if it’s in direct sunlight the projection won’t show up.
The same goes for contrasting lighting or colors in the background of the projection. Too much contrast directly behind a projected image makes the fact that it’s always a little see-through very clear.
In this feature film, the characters, the environments are great examples in expressing how the human state of consciousness of the Blade Runner world has shaped its own habitat. As an example I’ll explore the ‘Niander Wallace’ office building in an attempt to reveal the deeper symbolic structure that lies beneath, how it is presented & what design principles are used to accomplish this expression. For the environment is not only expressing / mirroring the psychological makeup of the inner self of its owner & his place in the world, but it is also expressing his place within the hierarchy on cosmological scale which is rudimentary part of any good story which explores the human condition.
Niander Wallace
was a scientist, replicant manufacturer, technologist and the founding CEO of the replicant manufacturing company, Wallace Corporation.
Symbolically refers to being blind for the ‘natural’ state of being or the natural world. He can therefor only ‘see’, ‘view’ & ‘interpret’ the world through artificial eyes. This is symbolically presented by hovering fishlike drones equipped with technical camera lenses which serve as his eyes to the world. This symbolic ‘blindness’ for the natural world is THE main reason why Wallace is not able to find & unlock the last piece in the puzzle of creating life itself; fertility, which resides in the domain of the natural world which he can not perceive. That is why his artificial creations remain barren & sterile, not able to reproduce themselves. He must therefore track down and get hold of Rachel & Deckard’s hybrid miracle child; the bridge between the ‘artificial’ & ‘natural’ world.
example of the ‘fishlike’ camera drones
The name ‘Niander’ refers to ‘Neander’ which means: New Man. So the name reflects Niander Wallace’s intentions & grand vision of creating a new- yet synthetic composed human being, by means of dethroning God as creator of the natural composed ‘Man’. To accomplish his ultimate goal he must ‘conquer’, ‘unlock’ the last stronghold, the entrance to the garden of Eden; the seat of life itself which resides in the womb of the natural woman. All this in order to finally wield the power of making his sterile creations fertile so they can populate the universe.
Side note: In today’s world we often interpret the word ‘Man’ for as being the masculine part of humanity. However from its original linguistic meaning, it is a plural word incorporating both male & female. Above a picture of entering the synthetic ‘womb’ of Niander Wallace’s residence displaying a lineup of various models of his artificially created humans.
The ‘Wallace’ Building
Interior Design Language
Introduction
To understand the fundamental design language of the Wallace building interior designs & structure, it’s important to be aware of the underlying symbolic cosmology in which the world of Blade Runner is set. The world of Wallace is manifesting itself by being the artificial counterpart or mirror of the natural world. So to give some concrete examples on how this plays out in terms of design choices:
The ‘Womb’
Within the cosmological structure of the natural world, creation of life is represented by the coming together of spirit & body, (heaven & earth). This through conception, nourishment & birth of a child in the womb of the female. Symbolically speaking, the seed from the man is the ‘spirit’ which is instilled in the women’s womb or in other words; is fused with ‘matter’ the earth. Being the counterpart / mirror of this natural world, Wallace’s artificial creation of life therefore must use the same analogies to match the cosmological structure. How this is expressed in the interior design can be seen in below picture of the ‘birth chamber’:
Also notice how the ‘artificial’ use of materials are evident counterparts of the ‘natural’ world (use of stone, plastic, etc.)
The sterile square ceiling opening; mimics the organic private female body part.
The light shining down from the opening; refers to & mimics the divine womb, where life is conceived & nourished.
The sterile plastic & transparent shaft; mimics the organic female birth canal.
The square spot on the floor; mimics & marks the ‘space’ the child will inhabit when entering the reality of the natural outside world.
Use of Color
The yellowish color pallet in the Wallace ‘Womb’ space is clearly used as a design element to express ‘artificiality’ throughout the whole film. Consequently therefore the interior spaces in the Wallace building complex are all lit with these yellow color tones to support consistency in the film’s symbolic narrative. The natural world counterpart is expressed with the color green, which is also consistently and purposefully used throughout the film to underline the symbolic meaning. Here an example picture:
Like in the first Blade Runner movie, the film opens with a close up shot featuring a green colored eye, which again refers to the natural world looking into a world which has taken over or being saturated by artifice.
The ‘Heaven’ realm
The place where the ‘Creator God’ & its ‘Angels’ reside. Which again are analogies to the cosmological structure of the natural world which also must be expressed in the artificial counterpart / mirror world of Wallace. As in the natural world, the heaven realm has a strong hierarchical structure which consolidates order of importance, divine power, providence & ranking.
The ‘Throne Room’ island, above the water
Stairs leading up to the ‘Throne Room’, below the water
‘Office Spaces’, below the water
In the middle picture we see one of the hallways leading to the office spaces & Wallace his ‘throne room’. Again the spaces are square, abstract & sterile in shape & compositional language which contrasts the organic natural world. Notice how the hierarchical cosmological structure of higher to lower order is expressed in going up & down the stairs leading to the throne room, which indicates Wallace’s ranking position within the order; he is the prime creator so his residence is the highest placed room within the total structure.
But that is just one of the many elements the designers used to firmly underline this symbolic cosmological order. The island on which the throne room is set, is placed ‘above’ the waterline & the other spaces are situated ‘below’ the waterline which marks the order of divinity & power in opposition to Wallace’s slave ‘angels’.
Also notice the caustic lighting effects in the lower office rooms is a reminder to support this notion of hierarchical structure of cosmological spaces. But also again underlining the ‘womb’ idea where artificial life is being generated.
But that’s not all; this notion of ‘above’ water & ‘below’ water has strong roots in ancient cosmology. If we dig deeper and frame a picture of a symbolic hierarchical structure of the cosmos, we could get something in the likes of this:
From above cosmic construct we could say that the ‘realm’ in which Wallace dwells is actually not ‘heaven’ or the highest form of ‘order’ but a place right between ‘order & chaos’, ‘semi-light’ & ‘semi-dark’, ‘gas & fluid’, ‘air & water’, etc. His ‘placement’ on the cosmological scale therefore actually makes perfect sense if we take into account what we talked about earlier, of him being a twilight person; very perceptive to the artificial world but blinded to the natural world. Which makes him unable to enter or conquer the Eden / heaven of the natural world. Or in other words, despite his aspirations of being a God, he remains ‘human’.
Though, there is still another interesting thing about the use of water as a symbolic design element in the Wallace building interior designs. If we study the ranking of different natural elements in the scale, we can see that it goes from fully ‘solid’ to fully ‘transparent’. Spiritually speaking, it indicates that the higher you travel on the scale, more will be revealed (enlightened), or the lesser your view is ‘obscured’ or ‘distorted’. So let’s draw an analogy to the office spaces below the water-line & see how that notion is expressed. Standing in the office space, looking upwards through the opening in the ceiling, the image we then see will be distorted by the refraction of the water. So Wallace’s ‘angels’ do not have access to the full ‘clear picture’ of his artificially created reality or heaven. So prohibition is enforced on autonomy, knowledge, self-determination, power, etc. This is only reserved / accessible to the ‘God’ of this twilight realm; Niander Wallace himself. However if we take his own ‘blinded’ spiritual constraint of perception/limitation into consideration and see how that is expressed in the throne room, we come to the following observations:
The island is situated just above the water-line, indicating that despite his genius abilities, brilliance & intelligence in artifice, he has barely managed to create ‘land’ to establish the 1st state of ‘order’. Because cosmologically speaking; only when spirit & matter meet, order can be established from chaos. This notion of order from chaos is symbolized & expressed by land rising from the water to build a house upon. So by artifice Wallace has gained dominion over artificial land & declared himself God over this land. But Wallace strives for a higher goal; to take over heaven by artifice; trying to bypass the natural order of the universe, creation of life & eventually God.
This cosmological notion of not yet having access to the natural Eden or heaven realm, is also expressed by the fact that the throne room is enclosed by stone slab; creating a kind of cave like environment. So even though Wallace considers his world as the artificial heaven, in its expression it is conveying more of an ‘underworld’ realm. Which is obscured & hiding from the natural heaven & its God. A dualistic symbolic notion of both rejection & protection; Wallace not able to reach the natural heaven because it is closed off from him by God, but yet also asif he is hiding from the natural world / heaven & needs protection to shield him from it.
This twilight dwelling between above & below, knowing & not knowing, order & chaos, etc., is also expressed by the constant shifting of light & shadow in the throne room. Which is again a strong symbolic reference to Wallace’s state of consciousness, his place on the cosmic hierarchical scale & the artificial reality of his world. On top of that, the caustic light is also shining from below the water-line, reflecting on the stone slab. Which again is a symbolic reminder of the fact that even though Wallace has managed to arise above the waters of chaos, still he has a distorted view on the higher realm of reality.
This isn’t the title of an upcoming Dan Brown novel, it’s an exploration of a fractal type that’s also referred to as the Apollonian limit set.
“A disco for parasites” is a real-time raymarched Kleinian group fractal.
Fractals have long been the domain of mathematicians and computer graphics programmers, however with the recent advances in procedural content generation and real-time image rendering techniques such as raymarching, their recursive and often mesmerising quality would seemingly lend them well for making art and video game content. For this experiment I chose the Apollonian gasket, a limit set of the Kleinian group that in my opinion produces quite natural looking patterns. And here we already stumble upon the first barrier one often encounters when looking into how fractals work: the theory is jargon ridden. With this article I aim to provide an accessible entry point to understanding the Apollonian gasket which afforded me to create “a disco for parasites”.
The fractal-like pattern called the Apollonian gasket used to create “a disco for parasites” is generated by recursively applying Möbius transformations to a set of circles located in some particular positions. The initial configuration of circles is bounded by some sort of shape and must satisfy the condition that they’re externally tangential, in other words their edges touch. Every subsequent iteration fits circles inside the negative space left by the previous iteration.
The curvature of a circle is equal to the reciprocal of its radius (1 / r) and from this follows that the larger the circle is, the smaller the curvature (a curvature of 0 would look like a straight line).
float ApollonianGasket(vec3 p)
{
float scale =1.0;
for (int i =0; i < ITERATIONS; i++)
{
p =2.0* clamp(p, -vec3(1.0), vec3(1.0)) - p;
float sqrRadius = dot(p, p);
p /= sqrRadius;
scale /= sqrRadius;
}
return0.2* abs(p.y) / scale;
}
[Shadertoy] Basic Apollonian gasket that results in the image on the left. Please note that the fractal is mirrored across the X-axis here.
Using a very basic raymarching approach the fractal can be represented in 3D.
[Shadertoy] Raymarching + surface normal calculation.
The final representation distorts the fractal to break the symmetries and make it look more organic. I got interesting results by offsetting the radius, in the case of “a disco for parasites”:
This resulted in a distorted 3D Apollonian gasket representation, and by experimentally rotating and offsetting the origin and direction vectors a photogenic view was found.
Texturing and lighting
Texturing proved to be more straightforward than expected. Commonly used strategies for texturing signed distance fields are triplanar mapping and cubemap lookups, however notwithstanding some stretching and mirroring artifacts it turned out that the components of vector p coud just be scaled and used for lookup in tiling 2D diffuse maps for a quite convincing result.
Physically based shading was used for lighting and the red component sampled from the diffuse map using the aforementioned texture coordinates was used for the roughness attribute of the material.
Literature:
Hill, Stephen., et al. (2013). Physically Based Shading in Theory and Practice. SIGGRAPH. [Available here]
Yáñez Escanciano, Jorge. (2017). An introduction to the limit set of Kleinian groups Master Thesis, Universidad Nacional de Educación a Distancia (España). Facultad de Ciencias [Available here]
