Connect with us

Tech

Speech recognition system trains on radio archive to learn Niger Congo languages

Published

on

speech recognition

Join Transform 2021 this July 12-16. Register for the AI event of the year.


For many of the 700 million illiterate people around the world, speech recognition technology could provide a bridge to valuable information. Yet in many countries, these people tend to speak only languages for which the datasets necessary to train a speech recognition model are scarce. This data deficit persists for several reasons, chief among them the fact that creating products for languages spoken by smaller populations can be less profitable.

Nonprofit efforts are underway to close the gap, including 1000 Words in 1000 Languages, Mozilla’s Common Voice, and the Masakhane project, which seeks to translate African languages using neural machine translation. But this week, researchers at Guinea-based tech accelerator GNCode and Stanford detailed a new initiative that uniquely advocates using radio archives in developing speech systems for “low-resource” languages, particularly Maninka, Pular, and Susu in the Niger Congo family.

“People who speak Niger Congo languages have among the lowest literacy rates in the world, and illiteracy rates are especially pronounced for women,” the coauthors note. “Maninka, Pular, and Susu are spoken by a combined 10 million people, primarily in seven African countries, including six where the majority of the adult population is illiterate.”

The idea behind the new initiative is to make use of unsupervised speech representation learning, demonstrating that representations learned from radio programs can be leveraged for speech recognition. Where labeled datasets don’t exist, unsupervised learning can help to fill in domain knowledge by determining the correlations between data points and then training based on the newly applied data labels.

New datasets

The researchers created two datasets, West African Speech Recognition Corpus and the West African Radio Corpus, intended for applications targeting West African languages. The West African Speech Recognition Corpus contains over 10,000 hours of recorded speech in French, Maninka, Susu, and Pular from roughly 49 speakers, including Guinean first names and voice commands like “update that,” “delete that,” “yes,” and “no.” As for the West African Radio Corpus, it consists of 17,000 audio clips sampled from archives collected from six Guinean radio stations. The broadcasts in the West African Radio Corpus span news and shows in languages including French, Guerze, Koniaka, Kissi, Kono, Maninka, Mano, Pular, Susu, and Toma.

To create a speech recognition system, the researchers tapped Facebook’s wav2vec, an open source framework for unsupervised speech processing. Wav2vec uses an encoder module that takes raw audio and outputs speech representations, which are fed into a Transformer that ensures the representations capture whole-audio-sequence information. Created by Google researchers in 2017, the Transformer network architecture was initially intended as a way to improve machine translation. To this end, it uses attention functions instead of a recurrent neural network to predict what comes next in a sequence.

Above: The accuracies of WAwav2vec.

Despite the fact that the radio dataset includes phone calls as well as background and foreground music, static, and interference, the researchers managed to train a wav2vec model with the West African Radio Corpus, which they call WAwav2vec. In one experiment with speech across French, Maninka, Pular, and Susu, the coauthors say that they achieved multilingual speech recognition accuracy (88.01%) on par with Facebook’s baseline wav2vec model (88.79%) — despite the fact that the baseline model was trained on 960 hours of speech versus WAwav2vec’s 142 hours.

Virtual assistant

As a proof of concept, the researchers used WAwav2vec to create a prototype of a speech assistant. The assistant — which is available in open source along with the datasets — can recognize basic contact management commands (e.g., “search,” “add,” “update,” and “delete”) in addition to names and digits. As the coauthors note, smartphone access has exploded in the Global South, with an estimated 24.5 million smartphone owners in South Africa alone, according to Statista, making this sort of assistant likely to be useful.

“To the best of our knowledge, the multilingual speech recognition models we trained are the first-ever to recognize speech in Maninka, Pular, and Susu. We also showed how this model can power a voice interface for contact management,” the coauthors wrote. “Future work could expand its vocabulary to application domains such as microfinance, agriculture, or education. We also hope to expand its capabilities to more languages from the Niger-Congo family and beyond, so that literacy or ability to speak a foreign language are not prerequisites for accessing the benefits of technology. The abundance of radio data should make it straightforward to extend the encoder to other languages.”

VentureBeat

VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative technology and transact.

Our site delivers essential information on data technologies and strategies to guide you as you lead your organizations. We invite you to become a member of our community, to access:

  • up-to-date information on the subjects of interest to you
  • our newsletters
  • gated thought-leader content and discounted access to our prized events, such as Transform 2021: Learn More
  • networking features, and more

Become a member

Continue Reading
Advertisement
Click to comment

Leave a Reply

Your email address will not be published. Required fields are marked *

Tech

Magic: The Gathering’s Adventures in the Forgotten Realms delves into Dungeons

Published

on

Magic: The Gathering's Adventures in the Forgotten Realms delves into Dungeons

Where does your enterprise stand on the AI adoption curve? Take our AI survey to find out.


I once feared Magic: The Gathering would kill Dungeons & Dragons. Wizards of the Coast ended up saving it, and now, the granddaddy of trading card games is heading to the Forgotten Realms of Faerûn — and its Dungeons.

Today, Wizards of the Coast is showing off more cards from the Adventures in the Forgotten Realms set, which launches July 8 on Magic: The Gathering — Arena and July 23 in paper. In addition to bringing the likes of Drizzt Do’Urden, Tiamat, and Lolth the Spider Queen to Magic, Adventures in the Forgotten Realms introduces Dungeons to the card game.

Senior game designer James Wyatt (who also worked on two of my favorite D&D books, the 3rd Edition City of the Spider Queen and Draconomicon) and worldbuilding designer Meris Mullaley showed off a handful of the set’s cards to the press last week. And the three Dungeons and their Venture mechanic showed how the Magic team is approaching fitting Realmslore into the set.

Dungeon delving

Above: The dungeons of Adventures in the Forgotten Realms.

Image Credit: GamesBeat

The Dungeons are adaptations of existing D&D modules and campaigns that have appeared in 5th Edition (among others):

“Whenever you have a card that tells you to Venture into the Dungeon, what you do is you pick one of these Dungeons, and you put a marker at the very top room. And every time you Venture, then you can move down a level — farther into the Dungeon — by one room,” Wyatt said in a video briefing.

Each player has their own Dungeons, so they could be exploring the Lost Mine of Phandelver at the same time (so, two people could be doing so in a 1-on-1 game, or three of four players could be in a Commander match). You can have one, two, or all three active at once. When you Venture, you could either go deeper into one or begin exploring another.

These Dungeons offer choices. You choose which one you want to delve into and which path you take. The Tomb of Annihilation has you sacrificing cards, artifacts, and life to gain a horrific benefit (which fits the theme of the lich Acererak’s deathtrap). I also find adding this dungeon interesting because Acererak was a card in Spellfire, which was D&D‘s failed answer to Magic back in the 1990s.

Halaster’s dungeon gives you more choices, but it takes longer to get through it (as befits the numerous levels of Undermountain).

“If you choose Dungeon of the Mad Mage, you’re really in this dungeon for a long time exploring the holes of Undermountain,” Wyatt said. “You need seven Ventures to get all the way through, but you have lots of choices to make as you go along the way.”

Dungeons are a neat way to capture the flavor of D&D within Magic. Undermountain has been a mainstay of the Realms since The Ruins of Undermountain boxed set in 1991; since then, TSR or Wizards of the Coast has published several campaign sets, adventures, game books, and even a board game about these halls.

The Magic team is using its existing combination of creatures, artifacts, and spells to take advantage of these Dungeons.

FR Venture Dungeon cards

Above: These cards work with Dungeons, giving you benefits or helping you get through them.

Image Credit: GamesBeat

“There are a variety of cards that interact with Venture in interesting ways, including all the way down to Common [rarity] with things like Shortcut Seeker, hitting that classic trope of ‘look, there’s a trapdoor under the rug,’” Wyatt said. “Venture is a strong theme across all rarities, so there’s lots of opportunity for players to experience the thrill of exploring Dungeons.”

I asked if the Dungeons had special loot attached to them, such as a Sphere of Annihilation for the Tomb of Annihilation. A Wizards spokesperson on the call said we’d have to wait and see on that.

Give me land, lots of land

Another way to capture the flavor of the Forgotten Realms is with lands. The Basic lands all have some art or text reference to Faerûn, even if it’s not obvious at first glance.

What’s really interesting are some of the alternate land cards. One example is Evolving Wilds, a Magic staple. This treatment captures the style of classic D&D modules such as The Keep on the Borderlands (it even has the lavender-ish coloring).

FR lands

Above: The Basic lands reference the Realms in their art and their text.

Image Credit: GamesBeat

The set will have nine of these lands, eight of them with new names.

“We’re calling this the Classic Module land frame. These are borderless module lands featuring art that is reminiscent of the cover art from classic Dungeons & Dragons adventure modules,” Mullaley said. “They’re all lands. There’s nine of them. This one is Evolving Wilds, but the other lands are new, with names that were created to sound like adventures.”

Seeing some of the Basic lands did raise a concern. The Forest doesn’t scream Forgotten Realms to me, and the text doesn’t add any flavor; it looks like it could fit into any other Magic set.

“We did a full concept push for this set, like we do for any Magic set. Obviously there’s already a ton of art exploring what the Forgotten Realms looks like. There’s not necessarily a ton of of art or color art establishing the look of specific geographical regions like the Evermoors, or the Spine of the World, or the High Forest,” Wyatt said. “So all of these lands — almost all of these lands — do actually point to specific places that we developed in the world guide, though I think that forest right there is an example of elven architecture, rather than a specific place, so that was also one of the areas we explored in the world guide.

“If I’m remembering right, the cycles of lands include one of each land type in the Underdark, one that shows a settlement of various peoples of the Realms, one that is just a wilderness area, and one that includes some ruins of ancient civilizations. So there’s definitely a lot of Realms flavor, sometimes not obvious in there, but in there.”

Who’s the set for?

FR card treatments

Above: Card treatments for Adventures in the Forgotten Realms include borderless art cards, special art cards that look like D&D stat blocks, and illustrations that hark back to 1st and 2nd Edition styles.

Image Credit: GamesBeat

As Mullaley and Wyatt showed off this batch of cards, I wondered (as did others on the briefing) who this set was for. Is it for Magic players, enticing them into something new? Is it for Realms fans who Wizards wants to push into Magic? Or folks like me, who enjoy both of Wizards’ big properties?

“I think that for someone who is familiar with Magic and not familiar with Dungeons & Dragons, it will be like encountering a completely new plane that we’ve created for the first time for a Magic set,” Mullaley said. “It’s for Standard play, so it’s built to work with all of the other sets in Standard. And while we created a few new mechanics that were kind of inspired by Dungeons & Dragons play for this set, for the most part, it plays like a Magic set, and it’s got the creature types you’ve come to expect and be the Standard exciting Magic gameplay, and the flavor of the world happens to be Dungeons & Dragons.

“So we’re hoping that, as you’re playing this, what might be a deep cut reference for a friend of yours might be something that sparks a bit of curiosity for you.”

One card that worries me is a Legendary character, the Dragonborn knight Nadaar, Selfless Paladin. They’re a character created for this set. But why would you need to make characters when you have official material going back to the “Grey Box” set of 1987 and Realms fans want characters they’ve come to love over the years, such as The Simbul, the dastardly wizard Manshoon, or even gods such as Bhaal?

“Hopefully, we can do both,” Wyatt said on mixing known and new characters together. “We have a lot of goals, putting Legends into a set, including hitting nostalgia, but also hitting various diversity milestones, trying to make sure that that we’re reflecting our audience and the game as it is now, not as it was 25 years ago. So, yeah, we definitely trying to do both.”

Yesterday, Magic head designer Mark Rosewater posted a blog with a number of hints and teases that addresses my concerns. These include:

  • a Legendary creature that makes a Legendary Hamster creature token (this must be Minsc & Boo, the beloved duo from the Baldur’s Gate games)
  • a card that creates a Legendary creature token named Vecna (while Vecna is more associated with Greyhawk than the Realms, the lich is a popular figure in the D&D community and was part of Critical Role’s story)
  • a creature with a death trigger that makes an equipment token (this could be a Gelatinous Cube, with the remains of an adventurer inside it)
  • Spend this mana only to cast Dragon spells or activate abilities of Dragons (this could be from an Orb of Dragonkind)
  • Creature — Bird Bear (this must be an Owlbear)
  • Creature — Elf Spider (this must be a Drider, the drow that Lolth curses to be part elf, part spider, and all horror)
  • Legendary Creature — Devil God (this must be Asmodeus, who’s been playing with the Realms for some time now)
  • Legendary Creature — Beholder

Also yesterday, Wizards of the Coast put out a list of folks who will have card previews and the date they’re showing them off.

GamesBeat

GamesBeat’s creed when covering the game industry is “where passion meets business.” What does this mean? We want to tell you how the news matters to you — not just as a decision-maker at a game studio, but also as a fan of games. Whether you read our articles, listen to our podcasts, or watch our videos, GamesBeat will help you learn about the industry and enjoy engaging with it.

How will you do that? Membership includes access to:

  • Newsletters, such as DeanBeat
  • The wonderful, educational, and fun speakers at our events
  • Networking opportunities
  • Special members-only interviews, chats, and “open office” events with GamesBeat staff
  • Chatting with community members, GamesBeat staff, and other guests in our Discord
  • And maybe even a fun prize or two
  • Introductions to like-minded parties

Become a member

Continue Reading

Tech

Tonkean raises $50M to expand its workflow automation platform

Published

on

Tonkean

Where does your enterprise stand on the AI adoption curve? Take our AI survey to find out.


Tonkean, a software startup developing a no-code workflow automation platform, today announced that it nabbed $50 million in a series B round led by Accel with participation from Lightspeed Ventures and Foundation Capital. CEO Sagi Eliyahu says that the proceeds will be put toward scaling up the company’s hiring efforts across engineering and go-to-market teams.

San Francisco, California-based Tonkean was founded in 2015 by Eliyahu and Offir Talmor. At age 18, Eliyahu and Talmor met in the Israel Defense Forces (IDF), where they spent four years working on software technologies and challenges. Before founding Tonkean, Eliyahu was the VP of engineering at Jive Software, but many of Tonkean’s R&D early hires in Israel came from Eliyahu’s and Talmor’s IDF unit.

Eliyahu argues that the value proposition of Tonkean’s platform is twofold. It gives businesses and teams within those businesses the ability to tailor workflows to systems, employees, and processes. At the same time, it solves challenges in a way that doesn’t require many customizations.

“As [Jive] scaled, [we] encountered problems that large businesses often see as inevitable: a tech stack that balloons to include hundreds if not thousands of applications and inefficiency that ran rampant throughout the organization,” Eliyahu told VentureBeat via email. “Tonkean was built to solve the fundamental challenges of enterprise software to allow department and operational experts to actually deliver software with the flexibility to streamline business processes without introducing yet more apps.”

Workflow automation

Tonkean’s workflow designer features adaptive modules that can be added or removed in a drag-and-drop fashion. Customers can use it to proactively reach out and follow people via email, Slack, or Microsoft Teams to deliver data and actions to them or to keep track and manage performance across processes, people, and systems. Moreover, they can automate manual steps such as triaging finance requests, routing items to team members, and chasing status updates. Or they can dive into live details of individual jobs and see aggregate views of metrics and KPIs like turnaround time, turnover rate, and cycle times for tasks.

“In many cases, Tonkean is reducing the need for internal custom development by IT and business technology teams or the need to purchase multiple packaged solutions to support needs from various business units,” Eliyahu said. “Tonkean operates at the cross-section of automation platforms like robotic process automation, integration platform as a service, and business process automation, often replacing but also often extending the value of these platforms by allowing enterprises to orchestrate more complex, human-centric processes and reducing the technical skill sets needed to leverage capabilities provided by technology platforms.”

Above: A screenshot of Tonkean’s workflow automation platform.

Image Credit: Tonkean

Tonkean says it already has “a few dozen” customers, mostly at the Fortune 1000 level — including Grubhub and Crypto.com.

“Tonkean’s AI-powered coordination engine can intelligently and proactively reach people by learning individual or team preferences, like what communication medium is preferred, and route alerts, data, or actions to the right place at the right time,” Eliyahu said. “Tonkean is the operating system for business operations, and as such can be used to deliver use cases in any business operations function including revenue operations, legal operations, HR operations, finance operations, IT operations, and more.”

Tonkean has raised $81 million in venture capital to date with this latest funding round, which also had contributions from Zoom CEO Eric Yuan, Atlassian co-CEO Scott Farquhar, former Google CEO Eric Schmidt, and executives from UiPath. The company, which has over 60 employees, plans to expand the size of its workforce to over 100 within the next year.

VentureBeat

VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative technology and transact.

Our site delivers essential information on data technologies and strategies to guide you as you lead your organizations. We invite you to become a member of our community, to access:

  • up-to-date information on the subjects of interest to you
  • our newsletters
  • gated thought-leader content and discounted access to our prized events, such as Transform 2021: Learn More
  • networking features, and more

Become a member

Continue Reading

Tech

Varjo Reality Cloud lets you virtually experience a real place via ‘teleportation’

Published

on

Varjo is unveiling its way to teleport to virtual spaces today.

Where does your enterprise stand on the AI adoption curve? Take our AI survey to find out.


Varjo is unveiling its Reality Cloud platform for virtual teleportation. That means one person can capture the reality of a space in a particular location and share that reality in extreme detail for a remote person to experience, virtually.

The Varjo Reality Cloud shares the details of a room in photorealistic detail, showing someone remotely located a view of the room in real time. Yes, you heard that. Varjo lets one person scan a 3D space and another person experience it virtually at almost the same time, as it can transfer the necessary data in compact streams of 10 megabits to 30 megabits per second with almost no time delays, the company said.

It’s a pretty amazing technology that comes from the pioneering work that Varjo has done in creating high-end virtual reality and mixed reality headsets for enterprises such Volvo, which uses it to design cars in virtual environments. The caveat, of course, is if the tech really works as envisioned.

“We are introducing Varjo Reality Cloud, and this is something very different from what you’ve seen from Varjo before,” said Timo Toikkanen, CEO of Varjo, said in an interview with GamesBeat. “We have been working on a software platform that is the first in the world that enables virtual teleportation.”

The earlier VR and mixed reality tech that Varjo introduced in the past couple of years now uses cameras on a Varjo VR-3 virtual reality headset to capture the environment around a person. Then it transmits that slice of reality to someone else who uses a headset to experience the exact same physical reality, but in a virtual way. If Varjo can deliver the Varjo Reality Platform with the same quality it shows in its videos, then it will feel like you’re “teleporting” from your real location to a virtual location.

“You can you can be anywhere in the world,” Toikkanen said. “You can scan your surroundings, not just a 3D object or something like that. You can digitize the world around you if you like. And do that in super high fidelity, through Varjo Reality Cloud, so anybody anywhere in the world can join you in that location and see it exactly the way you see it, in perfect color, with lights and reflections, and so forth.”

It’s no joke, as Varjo has been working on this for years and it has raised $100 million to date from investors including Volvo (via the Volvo Cars Tech Fund), Atomico, NordicNinja, EQT Ventures, Lifeline Ventures, Tesi, and Swisscanto Invest by Zürcher Kantonalbank.

“It’s a sci fi dream come true. But we are fully grounded in reality. So we have been looking at the at the experience. How can we enable people to have similar interpersonal experience as you do in real life, and do that remotely,” Toikkanen said. “What really accelerated for us during last year was the realization how world will never be returning to the same after COVID and travel will forever be changed. And we saw that this is one of those moments when world is more ready than ever for the transformation of this nature in the way the communication and interaction is done. This is the right time to begin that change.”

A realistic metaverse

Above: Varjo is unveiling its way to teleport to virtual spaces today.

Image Credit: Varjo

Toikkanen said that this capturing and sharing of reality is like a true-to-life metaverse, the universe of virtual worlds that are all interconnected, like in novels such as Snow Crash and Ready Player One.

He said that you will be able to see in real-time what your friend is seeing in another place through the cloud-based platform. One person can map their reality by looking around in a room, and that view is transported to the cloud and rebuilt as a room. The person that you share this reality with can view it and feel like they’re there, Toikkanen said.

“It’s a metaverse grounded in reality,” he said. “It really is like the science fiction, beaming yourself to the other end of the world and back. And we think we think this is a really big deal. If you think of the economical and ecological drivers in the world today, something like this makes travel unnecessary.”

He said it could pave the way for a new form of human interaction and universal collaboration.

“You can engage on a completely different level than you have ever been in the history of communications,” Toikkanen said. “It really does change things in a big way. Both for businesses as well as for private individuals. You can teleport to other people, to your family,  or you can teleport to a work project.”

The system lets anybody scan their surroundings, turning them into 3D imagery using a Varjo XR-3 headset and then transport that 3D space to another person. That person gets to see the exact physical reality, completely bridging the real and the virtual in true-to-life visual fidelity, said Urho Konttori, chief technology officer at Varjo in Helsinki, Finland.

“It’s super important that the latency is kept low enough so that you have you feel that the interaction is logical, and that you don’t have like motion-related latency,” said Konttori. “We have put immense amount of effort into making it so that human-eye resolution, fully immersive stream, from the cloud, can be sent in 10 to 30 megabits per second speeds.”

This real-time reality sharing will usher in a new era in universal collaboration and pave the way for a metaverse of the future, transforming the way people work, interact, and play, Konttori said.

For the past five years, Varjo has been building and perfecting the foundational technologies needed to bring its Varjo Reality Cloud platform to market such as human-eye resolution, low-latency video pass-through, integrated eye tracking and the Lidar ability of the company’s mixed reality headset.

The company has already delivered these building block technologies in market-ready VR products that enterprises use to design their products. And now Varjo is uniquely positioned to combine them with Varjo Reality Cloud to empower users to enjoy the scale and flexibility of virtual computing in the cloud without compromising performance or quality.

Using Varjo’s proprietary foveated transport algorithm, users will be able to stream the real-time human-eye resolution, wide-field-of-view 3D video feed in single megabytes per second to any device. This ability to share, collaborate in and edit one’s environment with other people makes human connection more real and efficient than ever before, eliminating the restrictions of time and place completely.

Dimension10 acquisition

Varjo has been working on the Varjo Reality Cloud for years.

Above: Varjo has been working on the Varjo Reality Cloud for years.

Image Credit: Varjo

To further accelerate bringing the vision for Varjo Reality Cloud to life, Varjo today also announced the acquisition of Dimension10, a Norwegian software company that pioneers industrial 3D collaboration.

“We’re big fans of the company and have been for a long time,” Toikkanen said. “They have been pioneering collaboration, 3D models. And we think collaboration is at the heart Varjo Reality Cloud and us joining forces with them expedites progress.”

The Dimension10 virtual meeting suite is designed for architecture, engineering and construction teams and will become a critical component to making virtual collaboration possible within Varjo Reality Cloud. Dimension10 adds 14 people to Varjo’s team.

Additionally, Varjo added Lincoln Wallen to the company’s board of directors. Wallen currently serves as the CTO at Improbable, and he is a recognized scholar in computing and AI.

Previously, Wallen has worked as CTO of Dreamworks where he transitioned global movie production to the cloud, including the development of a cloud-native toolset for asset management, rendering, lighting, and animation.

Varjo Reality Cloud will first be available to existing customers and partners in alpha access starting later this year. For more information about Varjo’s new cloud platform and its vision for the metaverse, tune into a live, virtual event today, June 24, 2021, at 9 a.m. Pacific time via varjo.com.

Tech demos

varjo Press Image for Varjo Reality Cloud 4

Above: Varjo lets one person scan a 3D space and another person experience it virtually.

Image Credit: Varjo

In a video tech demo, Varjo showed a simplification to show how the world can be captured and streamed in real time as a 3D representation. It shows a time-lapse capture of a scene captured in real-time from a Varjo XR-3 headset. The video is converted into a 3D space that someone with a viewer and access to the Varjo Reality Cloud can use to see that room from any 3D angle.

In the beginning of the video, the user scans the room and then stops to watch Konttori give a talk. While Konttori is speaking, you see the naturalness of the movement, captured with just a Varjo XR-3 headset in the room, no additional cameras or recording devices. The camera is able to move freely as it’s all in 3D and not a flat video.

In a second video, Varjo teleports Konttori to the company’s Varjo HQ in Helsinki in mixed reality. A user wearing the headset sees the teleported Konttori mixed into a physical space at the headquarters. Later they mix the teleported surroundings together with the physical space in the headquarters.

Cool technology

Volvo is using Varjo headsets to design cars.

Above: Kia is using Varjo headsets to design cars.

Image Credit: Varjo

Varjo was founded in 2016, when other headsets like the Oculus Rift and the HTC Vive first appeared. But instead of targeting entertainment, Varjo went after enterprises with no-compromise technology.

It debuted its first VR headset, the XR-1, in early 2019 with human-eye resolution, or 1,920 pixels x 1,080 pixels per eye and an 87-degree field of view. That headset cost $10,000, but the company followed it up December 2020 with its XR-3 and VR-3 headsets that combined VR and augmented reality in the same headset.

That generation had twice the performance of the previous generation, with “human-eye resolution” of 1,920 pixels x 1,920 pixels per eye and a 115-degree field of view. It was also cheaper, ranging from $3,195 to $5,495 and it was available for cheaper enterprise subscriptions.

Now these headsets can be the jumping off point for the Varjo Reality Cloud, as they can connect to the datacenter and upload the scanned environment that someone can see via the cameras that are on the headset. The quality of the headset capture enables high-quality imagery in the cloud, Konttori said.

“We have innovated for the last five years on making that high fidelity possible,” Toikkanen said. “It links directly to the investment we have made on the headset side into gaze tracking, eye tracking, if you like, because that enables innovation. We have also invested in transporting the data between the locations, to the cloud and back, so that we can do this ensure high quality or super low latency. So that’s essentially what we are. We think of it as nothing less than the next form of human interaction.”

The hard part

Varjo is targeting professionals such as product designers with its XR/VR headsets.

Above: Varjo is targeting professionals such as product designers with its XR/VR headsets.

Image Credit: Varjo

“Nobody else is at the place that they have the hardware even near the quality that we have, let alone the software stack that allows us to actually pull this off,” Toikkanen said. “And we have of course be developing this simultaneously. And now is the culmination of all that work.”

Gaze tracking is important because if you can track where someone’s eyes are moving, then you know what they’re looking at and you can transport that view with low latency. That allows the company to create foveated transport algorithms, which means it only sends the data that you can see and that you are looking at, rather than other data that isn’t needed in real time at that moment.

“It’s a huge undertaking, and so we developed a year and a half ago a new way of doing that transport,” Konttori said. “The video stream focuses at the place that you’re looking at. That’s where we have the full resolution in the video stream. And then the degrades gradually from that towards the edges of the screen. And does that very quickly. It means that we can send the data that we send at the moment on cables from the computer to the headset, which is running at like 20 gigabits per second, and we can send that with our new compression technology at 10 megabits to 30 megabits per second.”

That means it works that you can share imagery with someone 2,000 miles away, Toikkanen said.

Enterprise applications

Varjo's XR-3 and VR-3 headsets.

Above: Varjo’s new XR-3 and VR-3 headsets.

Image Credit: Varjo

It’s a level of quality that is 10 times the resolution difference of other headsets out there, Konttori said.

“You get real-time presence because when we’re scanning, we’re just not just making a 3D model of the surroundings that you’re in and make that a teleport location,” Konttori said. “We’re actually updating that in real time.”

You could have a manager on a factory floor put on a headset. They can create a teleport node, and people from other countries can join and see what the manager sees. It’s all updated in real time and people get a sense they are truly at that location. They can fix the things that the manager is looking at, and then take off a headset and be at home.

“If you want to visit your family, it’s the same thing,” Konttori said. “You can share that physical location and people can instantly perceive the world as if they were actually there themselves.”

Once you scan a place, you don’t have to scan it again, Toikkanen said. And you can use any headset to teleport to a location, or use a phone and still have the freedom of movement to look around. But the Varjo XR-3 is the only device that can be the teleportation node that broadcasts and streams the 3D space to someone else.

Toikkanen said it’s like moving from the telephone to a video conference, and moving from that to something that is even more transformative.

“We think there are going to be a billion people using this kind of service over the next 10 years or 20 years,” he said. “We are in the alpha phase with real customers and partners this year.”

A cousin of the Omniverse

BMW Group is using Omniverse to build a digital factory that will mirror a real-world place.

Above: BMW Group is using Nvidia’s Omniverse to build a digital factory that will mirror a real-world place.

Image Credit: Nvidia

I asked if this would be a way to scan the world into Nvidia’s Ominverse, the metaverse for engineers that lets them simulate photorealistic details in a virtual world to test how they will work in reality. BMW is using the Omniverse for creating a “digital twin,” or a car factory it can design in a virtual space before it builds an exact copy in the physical world.

Toikkanen said that both tools are useful for the metaverse and they are complimentary.

“They’re both part of the like, movement towards metaverse, and this teleport functionality is adding a completely new node into the sphere of discussion of a metaverse, which is that one part of that can be the real world itself,” Toikkanen said. “And we make it so that you get the benefits of a metaverse also in real world setting. And we think that’s at least equally transformative as the metaverse which is typically seen only in virtual reality.”

GamesBeat

GamesBeat’s creed when covering the game industry is “where passion meets business.” What does this mean? We want to tell you how the news matters to you — not just as a decision-maker at a game studio, but also as a fan of games. Whether you read our articles, listen to our podcasts, or watch our videos, GamesBeat will help you learn about the industry and enjoy engaging with it.

How will you do that? Membership includes access to:

  • Newsletters, such as DeanBeat
  • The wonderful, educational, and fun speakers at our events
  • Networking opportunities
  • Special members-only interviews, chats, and “open office” events with GamesBeat staff
  • Chatting with community members, GamesBeat staff, and other guests in our Discord
  • And maybe even a fun prize or two
  • Introductions to like-minded parties

Become a member

Continue Reading

Trending