Speaking of cutting-edge research progress in the field of VR (Virtual Reality), Meta, the world's most popular VR equipment supplier, definitely leads the market trend.
Meta CEO Mark Zuckerberg recently revealed Meta's latest layout in the metaverse in an interview. Meta will launch a new VR headset, the Meta Quest Pro, in October.
It is reported that the biggest highlight of the headset is that it has face and eye tracking capabilities, which can allow virtual digital avatars to make eye contact. Hardware analyst Brad Lynch believes that the Meta Quest Pro will use an improved XR2 processor that will provide a 30 percent year-over-year performance increase over the existing XR2 processor. Analyst Ming-Chi Kuo predicts that the Quest Pro will use a single-eye 2160×2160 resolution panel.
In the interview, Zuckerberg also talked about the social metaverse. He believes that from text, and images, to video, the carrier of social media is evolving towards a larger information capacity, and the next one can carry more than video. The carrier of the amount of information is in the field of VR.
Earlier, Zuckerberg personally recommended his own social platform Horizon Worlds and posted a VR selfie. Although the photo was ridiculed by the entire Internet group, Zuckerberg still brought a wave of Internet popularity to Horizon Worlds.
Meta has been doing a lot of movement in the metaverse recently. In June of this year, Meta's VR division, Reality Labs, launched four VR headset prototypes, showing four key technological evolution directions for future VR devices: high resolution, high dynamic range (HDR), thin and light form factor, and reduced dizziness. Sensitive zoom technology optimization. At the SIGGRAPH conference held in August, Douglas Lanman, head of display system development at Meta, gave a speech, introducing us to ten challenges to break the bottleneck of VR technology.
At the SIGGRAPH 2022 conference, foreign media Auganix reporter Sam Sprigg specially experienced the Starburst, a VR prototype that features HDR. The Oculus Quest 2 is the pinnacle of Meta's all-in-one VR headset to date, with over 10 million units shipped last year, making it the best-selling VR product in history, and the Starburst's peak brightness is 200 times that of the Quest 2.
So, what exactly is the metaverse layout of Meta? What did the Meta prototype feel like? Where has Meta's tens of billions of R&D been invested? Let's take a look at Meta's latest developments in the VR field with these questions in mind.
In order to explore four research areas, Meta released four prototypes
In June, Meta Reality Labs unveiled four prototype VR headsets in different technology areas.
Butterscotch explores resolution technology, Starburst explores HDR technology, Holocake 2 explores thin form factor technology for VR headsets, and the Half Dome series explores zoom technology. The four prototypes represent the latest advancements Meta has made in the development of four VR headsets.
Previously, Meta CEO Mark Zuckerberg proposed the concept of the Visual Turing test. To put it simply, the evaluation standard of the visual Turing test is whether the virtual image can be confused with the real one so that the human eye cannot distinguish the virtual image from the real image. Zuckerberg said the visual Turing test has two focuses, photorealism and presence.
One of the ways to make an image more realistic is to increase the resolution of the image as much as possible. To this end, Meta developed a prototype Butterscotch, which features "retina-level" resolution.
Butterscotch's PPD reaches 55, which is close to the "retina resolution" level of 60 pixels per 1-degree angle, which is 2.5 times the resolution of the Quest 2, said Yang Zhao, an optical scientist on the DSR team in Meta Reality Labs' scientific research division, Butterscotch's custom lens mold. In the group, there are two high-precision glass lenses, which have a higher surface quality but are also heavier. In addition, there is a special lens that diffracts light. The combination of the glass lens and the diffractive lens can present a clear image.
 |
| Starburst prototype |
The Starburst prototype features high HDR (High Dynamic Range Imaging) capabilities with a backlight that can reach a maximum brightness of 20,000nit, which is 200 times that of the existing VR headset Quest 2. The Meta Reality Labs team developed Starburst to restore full interior lighting in a virtual environment and to explore the ideal brightness for the user to achieve the best immersion experience.
Starburst replaces the original LCD screen backlight with COB LEDs (LEDs commonly used in car lights), which greatly improves the LCD backlight brightness, but also generates more heat. Therefore, the Starburst also comes with a fan and a 3D-printed cooling chassis. However, the overall device is relatively bulky and consumes a lot of power. Of the four technologies, this one is the furthest from the market.
 |
| Holocake 2 prototype |
The Holocake 2 prototype is characterized by being thin and light, which can improve the wearing comfort of the VR headset. Holocake 2 is a PC VR device. Although it is thin and light, it requires additional space for computing and battery components and adopts a split design.
Half Dome is a prototype VR headset based on the Oculus Rift, featuring a zoom lens. Starting from the third generation, the lens of Half Dome has been upgraded from a mechanical system to an electronic zoom system, reducing mechanical noise and vibration. Half Dome 3 has 32 switchable focus planes. According to experiencers, when playing games such as "Lucky's Tale" through Half Dome's dynamic zoom, the focus of the image in the field of view can change according to the distance the handle moves.
The peak brightness of Starburst is 20000nit, which is 200 times that of Quest 2
At SIGGRAPH 2022, the top computer graphics conference held this month, Sam Sprigg, a reporter from foreign media Auganix, especially experienced the Starburst, a VR prototype that features HDR. The Oculus Quest 2 is the pinnacle of Meta's all-in-one VR headset to date, with over 10 million units shipped last year, making it the best-selling VR product in history, and the Starburst's peak brightness is 200 times that of the Quest 2.
At the show site, two Starbursts were hung on a frame for participants to experience. The Starburst weighs about 2.5kg, which is a bit too much for a headset. "It's so heavy that you can't put it on your head, you have to hold it in front of your face," says Sam Sprigg. Also, the Starburst needs a lot of power to power its high-brightness display, so it's affixed to the frame.
The brightness of the Starburst is measured in "nits," and Sam Sprigg claims that the brightness of the indoor environment exceeds 10,000 nits, while the peak brightness of the Starburst is 20,000 nits. Meta has said that the peak brightness of the Quest 2 is about 100nit, and the peak brightness of the Starburst is 200 times that of the Quest 2.
And the peak value of 20000nit can also explain Starburst's "clunky" to a certain extent. High brightness also generates high heat, and to dissipate heat, the Starburst is equipped with a large aluminum heat sink. Aluminum heat sinks makeup at least 50 percent of a device's weight, says Nathan Matsuda, a research scientist in Meta's Display Systems Research (DSR) group. Additionally, there's a large fan on top of the unit that absorbs heat from the heatsink.
However, the Starburst has a lower resolution, around 2560x1620 pixels, slightly lower than the Quest 2. However, this is also acceptable. Because Starburst mainly shows HDR solutions, as for high resolution, that is the direction that Butterscotch should explore.
Starburst uses both color and monochrome LCDs (liquid crystal displays). That's because a single LCD has poor contrast in low light, while two-color LCDs absorb most of the light, degrading the display, Matsuda said. Therefore, the best display effect can be obtained by using color and monochrome LCDs in series.
In order to focus on exploring HDR technology, Starburst made compromises in the refresh rate and field of view of the picture. However, Sam Sprigg said that this compromise did not affect Starburst's performance in terms of HDR and brightness.
The first part of the Starburst demo is to set up a light source in virtual space and place some spheres to show the reflected light. Sam Sprigg said that at the 10,000nit display range, this is the same experience as the Quest 2, nothing special.
However, when the brightness rises to 10000nit, he can feel more obvious brightness. When showing 10000nit of light, the demo switched to an ocean sunset view. The sun pierced through the clouds and the light hit the waves of the ocean, making the water sparkle. Then, the light rose to a peak of 20,000nit, and there was a virtual sun in the picture. Sam Sprigg said he felt as if he saw the real sun because the light was so bright. Finally, as the sun went down, the waves gave off a faint reflection. This demo is impressive.
Sam Sprigg said that staring directly at a 20,000nit light source in Starburst may feel like staring at a 60-watt light bulb. However, the light source in VR is only inches from his eyes, which is close. He believes that this brightness display can bring users a more realistic and immersive experience.
However, Sam Sprigg also pointed out that power, heat, and form factor are important obstacles that limit the development of HDR technology in VR headsets. He believes that we may still need a long time to wait for a mature product that can output 20,000nit.
There are many difficulties, and VR headsets face ten technical challenges
At present, although Meta has made some technical progress on VR headsets, it still encounters many technical bottlenecks that cannot be solved at present. In his speech at SIGGRAPH, Douglas Lanman, Head of Display Systems Development at Meta, gave us some pointers - there are ten challenges that need to be solved in order to take the VR experience to the next level!
Whoever can take the lead in breaking these challenges is expected to gain the upper hand in the increasingly fierce Metaverse Hardware Entry Competition.
Resolution: Meta develops a Butterscotch prototype to improve resolution
To increase user immersion and make virtual worlds look more realistic and clear, VR headsets must have higher resolutions. The resolution of current VR headsets is very far from the level of human vision.
Meta says their initial goal is to achieve 8K resolution per eye and 60PPD (pixel density). But at present, Quest 2 has only reached 2K resolution and 20PPD per eye, which is still a long way from this goal.
To increase the resolution of VR headsets, Meta developed a Butterscotch prototype that focuses on exploring resolutions. In this regard, the development and production of high-resolution displays are not difficult, the difficulty lies in how to equip high-resolution displays with the corresponding computing power. Lanman pointed out that two technologies, Foveated Rendering and Cloud streaming, may help to improve computing power, but the development of these two technologies is also very difficult.
Field of view: The field of view of the VR headset is twice as small as that of the human eye
The average human horizontal field of view is 200 degrees, while most current commercial VR headsets have a horizontal field of view of 100 degrees. This means that the field of view of a VR headset is narrower than what we see in a real environment. In addition, there is room for improvement in VR headsets in terms of vertical field of view.
The wider the field of view, the more pixels the VR headset can render. This requires higher power and more waste heat. In addition, when the line of sight is widened, the image at the edge of the line of sight is easily distorted. Therefore, a wider line of sight requires better lens technology support. And, the researchers also want to make sure that the lenses and display are built in a way that doesn't increase the VR headset's form factor.
Ergonomics: VR headsets are not comfortable enough to wear
Most current VR headsets tend to be slightly "clunky", such as the Quest 2, which weighs over 500 grams and has a device thickness (protruding from the face) of 8cm. Such a device will obviously not bring a good wearing experience to the user. The ideal VR device should be thinner and lighter, allowing users to wear it comfortably for long periods of time.
Pancake lenses and holographic lenses may help reduce the size of the headset, and Meta's Holocake 2 prototype is currently exploring this direction, but it has not yet developed to the level of mass production.
Vision correction display: No low-cost vision correction display has yet been found
Better VR headsets should have vision correction features that allow users to not have to wear regular glasses under the VR headset. This problem can be solved with special lens attachments, but this is not an ideal solution. The best solution is for the headset's display to have its own vision correction function so that the headset designer no longer needs to consider issues such as fitting glasses.
But the challenge is how to find a solution that is cheap to manufacture and doesn't add extra weight to the headset.
Zoom: Meta prototype Half Dome can achieve human eye zoom
In the natural environment, in order to see pictures at different distances, the human eye can naturally zoom. But in VR headsets, because the distance between the human eye and the screen does not change, the human eye cannot zoom or see the difference between near and far objects. There is a vergence-accommodation-conflict problem. So sometimes users experience eye fatigue, even headaches, and vomiting.
To solve this problem, researchers at Meta have developed Half Dome, a series of prototypes that support "progressive vision." Half Dome can simulate different focal planes, showing different degrees of blur, thus helping the human eye to adjust the focus in the virtual world.
Eye tracking: it is difficult to adapt to everyone's eye conditions
Eye tracking is a key technology in virtual reality. It is the basis for many other important VR technologies, such as zoom, distortion correction, and more. Through this technology, users can achieve eye contact with other users in the virtual world and have a more realistic social experience.
The problem is that everyone has a different pupil shape, and everyone's eyelids and eyelashes grow differently, so current eye-tracking technology isn't well suited for all users. Therefore, the Lanman team will extensively collect more user data to upgrade the eye-tracking technology and strive to make it suitable for more people.
Distortion correction: Meta developed a distortion simulator to speed up algorithm iteration
The movement of the pupil can distort the image, thereby reducing the user's sense of immersion, especially when used with zoom technology. To develop the correction algorithm, researchers must test the adjustment on a physical headset, but producing headsets can take weeks or even months, significantly delaying algorithm iterations.
To solve this problem, the Meta research team developed a distortion simulator. With this simulator, researchers can test correction algorithms without producing test headsets and special lenses.
High Dynamic Range (HDR): The peak brightness of the Starburst prototype is 20000nit
Physical objects and environments are much brighter than VR headsets, and to address this, Meta created the Starburst prototype. The peak brightness of Starburst is 20000nit, which is 200 times that of the existing Quest 2.
Starburst can more realistically simulate enclosed spaces and lighting conditions at night, making virtual environments look more realistic. However, currently, Starburst is heavy and consumes a lot of energy.
Meta believes that HDR contributes more to visual realism than resolution and zoom, but the technology is the most distant from practical use.
Visual realism: Passthrough technology is not yet mature
Better VR headsets should also allow VR users to stay engaged with the real world. VR headsets can record real-world environment information and display it as a video image in virtual reality. This technique, called "passthrough," has been implemented in commercial VR headsets, but the quality is poor. For example, the Quest 2 offers a black-and-white perspective mode, while Meta's upcoming high-end VR headset "Quest Pro" aims to pursue higher resolutions and color perspective capabilities.
However, VR headsets have yet to achieve a perfect reconstruction of the physical environment. The perspective of the picture captured by the perspective technology spatially deviates from the eye, and users may feel uncomfortable when using it for a long time. So, Meta is working on AI-assisted line-of-sight synthesis, which generates perspective-correct viewpoints in real-time with high visual fidelity.
In addition, Meta is working on a "Reverse Passthrough" feature that would allow a real-world user to see the VR user's glasses and face to make eye contact. But currently, the technology is visually incongruous, unnatural, and far from ready for the market.
Facial reconstruction: Meta conducts research on the "virtual avatar" project
Meta hopes that in the future people will meet in a virtual environment as real as in reality. To that end, Meta is working on a research project called Codec Avatars, which is dedicated to creating 3D digital avatars for people in virtual environments.
Currently, VR headsets can read the facial expressions of VR users in real-time and transfer them into virtual reality. It is reported that the upcoming Quest Pro may be the first device from Meta to offer face-tracking.
Meta's VR technology has made breakthroughs, but it still needs to continue to overcome technical problems
Lanman presented the top ten challenges currently facing VR headsets at SIGGRAPH 2022, where we can see the progress and unsolved problems of the current Meta. Meta has released four prototypes that aim to solve the four problems of VR headsets in terms of resolution, zoom, thin and light form factor, and HDR. These four prototypes have their own strengths and have achieved preliminary technical progress in their respective fields. In addition, Meta worked on the avatar project to assist the facial reconstruction function of VR headsets and developed a distortion simulator to accelerate algorithm iteration.
These prototypes show the highest level of technology that Meta can achieve in VR today, but these technologies are far from enough to create a full-fledged VR headset. At present, there are still some technical problems that Meta cannot overcome temporarily. For example, the field of view of the VR headset is narrower than the human eye, the VR headset is not comfortable enough to wear, the VR headset is not equipped with a vision correction function, and the eye-tracking technology of the VR headset is not suitable for everyone, and the VR headset is not suitable for everyone. The "see-through" technology is not yet mature and so on.
Counting the current exploration of Meta on VR headsets, we can find that Meta has come a long way on the way to exploring VR headsets. However, Meta is far from the end of this road. At present, various technologies related to VR headsets are under active development, but these technologies are far from mature, and there is still a long way to go before they are commercially available. From this perspective, the explosion of VR is still far away.
0 Comments