Connect with us

Tech

Roblox goes public at $41.9 billion valuation in direct listing

Published

on

Roblox goes public at $41.9 billion valuation in direct listing

Join Transform 2021 for the most important themes in enterprise AI & Data. Learn more.


Roblox, the platform for Lego-like user-generated games, went public today in a direct listing, where employees and investors finally get a chance to sell shares on the market. The initial price was far higher than the reference price of $45 a share, as the first trade was about $64.50 a share, giving the company a valuation of $41.9 billion.

The event isn’t an initial public offering, as the San Mateo, California-based company decided not to raise money for the company itself. But the direct listing tells the world just how valuable the public markets believe the company has become with its 32.6 million daily active users.

Roblox‘s employees and shareholders sold more than 388.2 million shares out of a total 650 million available. The shares are trading under the RBLX stock symbol on the New York Stock Exchange. Shares are already trading above $70 a share.

“It’s a nice moment to step back,” said Roblox chief business officer Craig Donato in an interview with GamesBeat today. “We don’t normally take these moments to touch the trophy. We’re so always focused on what’s next. It’s nice to just take the moment for today to reflect on that.”

Though it chose not to raise money today, Roblox itself still has plenty of cash, as it brought in $520 million in private funds last month at a $29.5 billion valuation, which was seven times its value from a funding round from nearly a year before. The reason for that skyrocketing valuation is that the company saw huge increases in players during the pandemic. And since the company gets a 30% cut from the revenues generated by its user-generated games, the company’s bookings for 2020 were $1.9 billion, double what they were the year before.

“This will raise the visibility to the company, which is super important as we try to hire more and more folks to the company,” Donato said. “Raising the profile of the company and supporting our recruiting efforts is a great thing. As you know, we have big aspirations for where Roblox can go. And one of the things that we need to make sure that we are able to do is really to step into that vision, and that requires a lot more people. So hiring remains a big focus for us.”

With the higher-than-expected valuation, Roblox will be able to raise money in the future at a much higher price than it could have just a short time ago. And if any other company wanted to acquire Roblox, it will be far more expensive after today.

If the direct listing goes well, then Roblox’s value will be high, and it will be able to raise money for itself in the future without worrying if it is leaving money on the table by mispricing its stock, as the market will have established its stock price. And it will be another vote of confidence for game platforms, where a company creates the foundations for games rather than making games itself, said Roger Lee, the managing director at Battery Ventures (which is not an investor), an interview with GamesBeat.

Above: Roblox CEO and founder Dave Baszucki rings the opening bell at the New York Stock Exchange.

Image Credit: Roblox

But there are some risks, as losses are going up as the company expands. Roblox recently reported that it lost $253.3 million on revenues of $923.9 million for the year ended December 31.

“I think the unit economics and our business are strong, they have been strong, and they’ll continue to be strong,” Donato said. “That said, I think our inclination is to invest in long term success. We will always be looking at can we pay the creator community more money. Can we invest in our infrastructure to enable the experiences to be more sophisticated, more advanced, or involve new use cases? While we do pretty much care about the bottom line, I’d say we’re in the mode of very much investing in our long-term success.”

The company’s free cash flow, or money coming in the door, was $411.2 million in 2020, significantly more than its net losses.

“We’re fortunate in that our business has been cashflow positive for a number of years,” Donato said. “We are saving more money than we’re spending. That said it’s nice to be able to raise some additional money to have an increase in the size of our cash reserves for investments that we want to make. We’re a thoughtful company in terms of how we spend our money.”

The company will use the proceeds from the previous $520 million to hire more people. Right now, Roblox has more than 1,000 people, including more than 500 that were hired during COVID-19.

“I think we’re in the early innings of a very long game,” Donato said. “So we know we want people that are invested in us, the long-term opportunity.”

The metaverse

How to make a metaverse

Above: How to make a metaverse

Image Credit: Roblox

As I wrote on Friday, I see its pending success or failure as a stock as a kind of referendum on the metaverse, the universe of virtual worlds that are all interconnected, like in novels such as Snow Crash and Ready Player One. Is that a reason to buy the company’s stock next week?

As much as anyone, Roblox CEO Dave Baszucki is a kind of James Halliday figure, the fictional billionaire who built the Oasis, or the version of the metaverse in Ernest Cline’s sci-fi novel Ready Player One. (OK, maybe he’s more like Ogden Morrow, the cofounder of the Oasis in the novel). Like Halliday, Baszucki is the visionary who dreamed of building the metaverse in the early days of his company 17 years ago.

Baszucki and his cofounder, Erik Cassel, played around with physics in a virtual world, and they coded a platform where kids and adults could interact in 3D simulated virtual environments. The materials were physically accurate blocks, like a world full of virtual Legos, and users could build whatever they wanted. They wound up building games, and those user-generated games became the foundation for a business behemoth. Sadly, Cassel passed away a while ago, but Baszucki never neglects to mention him when talking about the company’s early days.

Roblox’s games have become so popular that people have played the best titles billions of times. On average, 32.6 million people come to Roblox every day. More than 1.25 million creators have made money in Roblox. In the year ended December 31, 2020, users spent 30.6 billion hours engaged on the platform, an average of 2.6 hours per daily active user each day.

The next steps are building more features that can enable the metaverse, where people would spend more time and engage in a wider variety of activities.

“I think that the next big chapter for us is really expanding the different types of use cases that happen on our platform,” Donato said. “We’ve already started on that. Obviously, a lot of people come to Roblox to share adventures and hang out with friends. But we’ve invested heavily in in music and other forms of socially consumed entertainment. We’ve done concerts and launch parties, as well as events with video providers like our working friends Warner Bros around DC Comics. So there’s all sorts of things we’re going to do there.”

While that growth is good, it raises Roblox’s costs. The company has more than 2,300 paid moderators, who serve as contractors, and their job is to ensure civility.

“We’ve been investing in safety and moderation systems since day one. We continue to constantly be pushing what we do in those areas,” Donato said. “Our chat filter for years has been age sensitive. So we’re filtering not just on the age of the speaker, but the person that sees on the other end. We’re double filtering it based on age. So we’ve invested a lot of infrastructure, not just doing basic level mom moderation, but context sensitive moderation. And that level investment will actually serve us well, as we move into the next chapter.”

Much like Disney, Roblox is just beginning to see its audience get older and cross multiple generations.

“As our audience grows up and ages up, I actually think it’s an advantage because a lot of the infrastructure that we need to put in place to maintain safety and civility is actually very well served and needed for people of all ages,” Donato said. “It just needs to be built in a way that it’s age and context sensitive.”

He added, “I’m excited about the work we’re going to be doing in education. We will invest more in educational experiences on our platform. And we’ve been doing some work-related experiences.”

A venture capitalist’s view

Roblox's annual figures

Above: Roblox’s annual figures

Image Credit: Roblox

Lee said that he saw Roblox through the lens of his children, who are in their younger years.

“My older and middle child were spending tons of time on Roblox, and that was my first view of how special and innovative the company was,” he said. “It was the experience of watching my children in the Roblox community that opened my eyes to what they were doing. And Dave is a spectacular leader, who has been very clear with his vision from day one. It’s very rare to have such clarity.”

He added, “I’ve always been a believer in the metaverse. I thought the first incarnation of Second Life was the 1.0 version of what the metaverse could be. And then we went off into social media land for the next 15 years and we had very lightweight versions of it, with Facebook and Instagram, and Snap. And I think the emergence of Roblox has very quietly, very steadily become the real metaverse where it’s not this centrally controlled platform or environment. It’s this distributed platform that the creators can build on their own.”

As for the frenzy in the investment and acquisitions market for games, Lee said, “I think one of the things people will appreciate over time is actually how sticky the consumers are with these games. There’s a belief that the revenue for these games is not very durable. And when you actually look at the revenue for these franchises, and how loyal the players are, they’re much stickier, they’re much more engaged, and they’re much more loyal than I think most people appreciate.”

A Roblox investor’s view

Roblox opened trading higher than expected on the NYSE.

Above: Roblox opened trading higher than expected on the NYSE.

Image Credit: Roblox

Neil Rimer, a partner at Index Ventures (one of Roblox’s biggest backers) said in a blog post that he views Roblox as a new type of media company, much like Disney was when it emerged as a high growth company in the 1950s.

“In many ways, Roblox … could be considered a 21st-century Disney: a category-defining media company that capitalized on major demographic and technological changes to create totally new forms of entertainment and human interaction,” Rimer wrote. “Unlike Disney which has to find a common denominator to justify the cost of its productions and attractions, Roblox offers every group of visitors their own personal amusement park to explore that is tailored to their age and interests on any given day.”

He added, “Roblox is probably the greatest contemporary example of a founder taking a very long view and never — not for a millisecond — doubting the relevance of the ultimate vision as they patiently build the broad foundation to underpin it. To arrive where it is today, Roblox had to resist the temptation to jump on the latest ‘hot’ game genre and try to build its own version of Minecraft, Clash of Clans or Fortnite. Instead, Dave remained humbly focused on ensuring that Roblox would be the place the next version of these games could be built.”

A developer’s view

Yonathan Raz-Fridman, the CEO of Supersocial, believes enough in Roblox that he has created a development studio to make games for Roblox. While Roblox has eight million creators and 1.25 million who have made money from making games, Raz-Fridman is betting that bigger studios can raise the quality bar and provide better experiences inside the world of Roblox than outside of it.

“We’re seeing a lot of just overall growth in the developer community, both studios coming in, as well as the studios that started in Roblox getting bigger and bigger,” Donato said. “The common use case [in the past] was a single developer. Now, all of our top experiences are built by teams of 10 or more people.”

GamesBeat

GamesBeat’s creed when covering the game industry is “where passion meets business.” What does this mean? We want to tell you how the news matters to you — not just as a decision-maker at a game studio, but also as a fan of games. Whether you read our articles, listen to our podcasts, or watch our videos, GamesBeat will help you learn about the industry and enjoy engaging with it.

How will you do that? Membership includes access to:

  • Newsletters, such as DeanBeat
  • The wonderful, educational, and fun speakers at our events
  • Networking opportunities
  • Special members-only interviews, chats, and “open office” events with GamesBeat staff
  • Chatting with community members, GamesBeat staff, and other guests in our Discord
  • And maybe even a fun prize or two
  • Introductions to like-minded parties

Become a member

Continue Reading
Advertisement
Click to comment

Leave a Reply

Your email address will not be published. Required fields are marked *

Tech

Understanding the differences between biological and computer vision

Published

on

Understanding the differences between biological and computer vision

Join Transform 2021 this July 12-16. Register for the AI event of the year.


Since the early years of artificial intelligence, scientists have dreamed of creating computers that can “see” the world. As vision plays a key role in many things we do every day, cracking the code of computer vision seemed to be one of the major steps toward developing artificial general intelligence.

But like many other goals in AI, computer vision has proven to be easier said than done. In 1966, scientists at MIT launched “The Summer Vision Project,” a two-month effort to create a computer system that could identify objects and background areas in images. But it took much more than a summer break to achieve those goals. In fact, it wasn’t until the early 2010s that image classifiers and object detectors were flexible and reliable enough to be used in mainstream applications.

In the past decades, advances in machine learning and neuroscience have helped make great strides in computer vision. But we still have a long way to go before we can build AI systems that see the world as we do.

Biological and Computer Vision, a book by Harvard Medical University Professor Gabriel Kreiman, provides an accessible account of how humans and animals process visual data and how far we’ve come toward replicating these functions in computers.

Kreiman’s book helps understand the differences between biological and computer vision. The book details how billions of years of evolution have equipped us with a complicated visual processing system, and how studying it has helped inspire better computer vision algorithms. Kreiman also discusses what separates contemporary computer vision systems from their biological counterpart.

While I would recommend a full read of Biological and Computer Vision to anyone who is interested in the field, I’ve tried here (with some help from Gabriel himself) to lay out some of my key takeaways from the book.

Hardware differences

In the introduction to Biological and Computer Vision, Kreiman writes, “I am particularly excited about connecting biological and computational circuits. Biological vision is the product of millions of years of evolution. There is no reason to reinvent the wheel when developing computational models. We can learn from how biology solves vision problems and use the solutions as inspiration to build better algorithms.”

And indeed, the study of the visual cortex has been a great source of inspiration for computer vision and AI. But before being able to digitize vision, scientists had to overcome the huge hardware gap between biological and computer vision. Biological vision runs on an interconnected network of cortical cells and organic neurons. Computer vision, on the other hand, runs on electronic chips composed of transistors.

Therefore, a theory of vision must be defined at a level that can be implemented in computers in a way that is comparable to living beings. Kreiman calls this the “Goldilocks resolution,” a level of abstraction that is neither too detailed nor too simplified.

For instance, early efforts in computer vision tried to tackle computer vision at a very abstract level, in a way that ignored how human and animal brains recognize visual patterns. Those approaches have proven to be very brittle and inefficient. On the other hand, studying and simulating brains at the molecular level would prove to be computationally inefficient.

“I am not a big fan of what I call ‘copying biology,’” Kreiman told TechTalks. “There are many aspects of biology that can and should be abstracted away. We probably do not need units with 20,000 proteins and a cytoplasm and complex dendritic geometries. That would be too much biological detail. On the other hand, we cannot merely study behavior—that is not enough detail.”

In Biological and Computer Vision, Kreiman defines the Goldilocks scale of neocortical circuits as neuronal activities per millisecond. Advances in neuroscience and medical technology have made it possible to study the activities of individual neurons at millisecond time granularity.

And the results of those studies have helped develop different types of artificial neural networks, AI algorithms that loosely simulate the workings of cortical areas of the mammal brain. In recent years, neural networks have proven to be the most efficient algorithm for pattern recognition in visual data and have become the key component of many computer vision applications.

Architecture differences

Above: Biological and Computer Vision, by Gabriel Kreiman.

The recent decades have seen a slew of innovative work in the field of deep learning, which has helped computers mimic some of the functions of biological vision. Convolutional layers, inspired by studies made on the animal visual cortex, are very efficient at finding patterns in visual data. Pooling layers help generalize the output of a convolutional layer and make it less sensitive to the displacement of visual patterns. Stacked on top of each other, blocks of convolutional and pooling layers can go from finding small patterns (corners, edges, etc.) to complex objects (faces, chairs, cars, etc.).

But there’s still a mismatch between the high-level architecture of artificial neural networks and what we know about the mammal visual cortex.

“The word ‘layers’ is, unfortunately, a bit ambiguous,” Kreiman said. “In computer science, people use layers to connote the different processing stages (and a layer is mostly analogous to a brain area). In biology, each brain region contains six cortical layers (and subdivisions). My hunch is that six-layer structure (the connectivity of which is sometimes referred to as a canonical microcircuit) is quite crucial. It remains unclear what aspects of this circuitry should we include in neural networks. Some may argue that aspects of the six-layer motif are already incorporated (e.g. normalization operations). But there is probably enormous richness missing.”

Also, as Kreiman highlights in Biological and Computer Vision, information in the brain moves in several directions. Light signals move from the retina to the inferior temporal cortex to the V1, V2, and other layers of the visual cortex. But each layer also provides feedback to its predecessors. And within each layer, neurons interact and pass information between each other. All these interactions and interconnections help the brain fill in the gaps in visual input and make inferences when it has incomplete information.

In contrast, in artificial neural networks, data usually moves in a single direction. Convolutional neural networks are “feedforward networks,” which means information only goes from the input layer to the higher and output layers.

There’s a feedback mechanism called “backpropagation,” which helps correct mistakes and tune the parameters of neural networks. But backpropagation is computationally expensive and only used during the training of neural networks. And it’s not clear if backpropagation directly corresponds to the feedback mechanisms of cortical layers.

On the other hand, recurrent neural networks, which combine the output of higher layers into the input of their previous layers, still have limited use in computer vision.

visual cortex vs neural networks

Above: In the visual cortex (right), information moves in several directions. In neural networks (left), information moves in one direction.

In our conversation, Kreiman suggested that lateral and top-down flow of information can be crucial to bringing artificial neural networks to their biological counterparts.

“Horizontal connections (i.e., connections for units within a layer) may be critical for certain computations such as pattern completion,” he said. “Top-down connections (i.e., connections from units in a layer to units in a layer below) are probably essential to make predictions, for attention, to incorporate contextual information, etc.”

He also said out that neurons have “complex temporal integrative properties that are missing in current networks.”

Goal differences

Evolution has managed to develop a neural architecture that can accomplish many tasks. Several studies have shown that our visual system can dynamically tune its sensitivities to the common. Creating computer vision systems that have this kind of flexibility remains a major challenge, however.

Current computer vision systems are designed to accomplish a single task. We have neural networks that can classify objects, localize objects, segment images into different objects, describe images, generate images, and more. But each neural network can accomplish a single task alone.

Gabriel Kreiman

Above: Harvard Medical University professor Gabriel Kreiman. Author of “Biological and Computer Vision.”

“A central issue is to understand ‘visual routines,’ a term coined by Shimon Ullman; how can we flexibly route visual information in a task-dependent manner?” Kreiman said. “You can essentially answer an infinite number of questions on an image. You don’t just label objects, you can count objects, you can describe their colors, their interactions, their sizes, etc. We can build networks to do each of these things, but we do not have networks that can do all of these things simultaneously. There are interesting approaches to this via question/answering systems, but these algorithms, exciting as they are, remain rather primitive, especially in comparison with human performance.”

Integration differences

In humans and animals, vision is closely related to smell, touch, and hearing senses. The visual, auditory, somatosensory, and olfactory cortices interact and pick up cues from each other to adjust their inferences of the world. In AI systems, on the other hand, each of these things exists separately.

Do we need this kind of integration to make better computer vision systems?

“As scientists, we often like to divide problems to conquer them,” Kreiman said. “I personally think that this is a reasonable way to start. We can see very well without smell or hearing. Consider a Chaplin movie (and remove all the minimal music and text). You can understand a lot. If a person is born deaf, they can still see very well. Sure, there are lots of examples of interesting interactions across modalities, but mostly I think that we will make lots of progress with this simplification.”

However, a more complicated matter is the integration of vision with more complex areas of the brain. In humans, vision is deeply integrated with other brain functions such as logic, reasoning, language, and common sense knowledge.

“Some (most?) visual problems may ‘cost’ more time and require integrating visual inputs with existing knowledge about the world,” Kreiman said.

He pointed to following picture of former U.S. president Barack Obama as an example.

ObamaPicture

Above: Understanding what is going on it this picture requires world knowledge, social knowledge, and common sense.

To understand what is going on in this picture, an AI agent would need to know what the person on the scale is doing, what Obama is doing, who is laughing and why they are laughing, etc. Answering these questions requires a wealth of information, including world knowledge (scales measure weight), physics knowledge (a foot on a scale exerts a force), psychological knowledge (many people are self-conscious about their weight and would be surprised if their weight is well above the usual), social understanding (some people are in on the joke, some are not).

“No current architecture can do this. All of this will require dynamics (we do not appreciate all of this immediately and usually use many fixations to understand the image) and integration of top-down signals,” Kreiman said.

Areas such as language and common sense are themselves great challenges for the AI community. But it remains to be seen whether they can be solved separately and integrated together along with vision, or integration itself is the key to solving all of them.

“At some point we need to get into all of these other aspects of cognition, and it is hard to imagine how to integrate cognition without any reference to language and logic,” Kreiman said. “I expect that there will be major exciting efforts in the years to come incorporating more of language and logic in vision models (and conversely incorporating vision into language models as well).”

Ben Dickson is a software engineer and the founder of TechTalks. He writes about technology, business, and politics.

VentureBeat

VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative technology and transact.

Our site delivers essential information on data technologies and strategies to guide you as you lead your organizations. We invite you to become a member of our community, to access:

  • up-to-date information on the subjects of interest to you
  • our newsletters
  • gated thought-leader content and discounted access to our prized events, such as Transform 2021: Learn More
  • networking features, and more

Become a member

Continue Reading

Tech

How Legacy Games still has a good business selling CD games at Walmart

Published

on

How Legacy Games still has a good business selling CD games at Walmart

Join Transform 2021 this July 12-16. Register for the AI event of the year.


Legacy Games has been publishing and distributing casual PC games at retail since 1998. And believe it or not, it’s still in business and its founder Ariella Lehrer is back in charge of the company that targets women who are 40 years old or older.

Lehrer started the Los Angeles company 23 years ago to make games for women at retail. She left in 2017 to move on to augmented reality game maker Hitpoint. Legacy Games stayed small with just a handful of people, but it kept its relationships with key retailers such as Walmart. And it still has Walmart as a client. Meanwhile, most of its competitors have moved on to more attractive markets. So after three years at Hitpoint, Lehrer returned as CEO of Legacy Games in October and she has started a new indie publishing program.

Legacy has helped game developers find new casual game customers through Legacy’s unique distribution channels, such as Walmart. Now the company is diversifying its game portfolio by working with indie game developers. Lehrer said in an interview with GamesBeat that she is signing up a variety of indie developers who are making PC and mobile games that target casual gamers. Roughly 70% of the customers are older women, and about 30% are men.

“We are signing up cool indie game developers, and that’s overdue,” Lehrer said. “I came back and found it was still kicking, and maybe I can push it toward digital. I’m really focused on bringing Legacy Games into the digital age.”

Going digital and physical

Above: Legacy Games targets its games at women over 40.

Image Credit: Legacy Games

Since coming back, Lehrer launched a digital store and she expects the company triple its digital sales in 2021.

She is signing up developers that have highly rated casual games on Steam, but have otherwise had limited distribution. Many developers have had a hard time in the pandemic. A survey by the Game Developers Conference found that 34% of game developers saw their business decline, and a report from Video Game Insights found more than 50% of indies never make more than $4,000.

“We found there are all these wonderful indie games on Steam, but our customers don’t go on Steam,” she said.

Lehrer distributes the games on the company’s web site. And if any do particularly well on the digital storefront, then the company will see if they will sell at Walmart, where the company sells around 3,000 units a week. Legacy can package the games together in a bundle on DVD discs. Successful digital bundles will then be sold at retail.

“It’s a lovely little business,” she said. “We have been profitable every year except for the Great Recession” in 2008.

legacy 3

Above: Legacy Games was started in 1998.

Image Credit: Legacy Games

It got started with a hit game called Emergency Room, originally created for IBM. Lehrer got the rights back and then sold it at retail at Walmart, and the title sold more than a million units. At its height, Legacy Games had about $5 million in revenues. That was never that exciting to investors. But the company has stayed steady and it did raise money once a while ago from Targus. The company made 20 different games based on television licenses like Law & Order, Criminal Minds, Murder She Wrote, Tarzan, and others. Lehrer kept it going but stayed on

Legacy has 18 of 24 spots on the shelf for casual games at Walmart stores. All of the competitors have loved on to other markets. Lehrer said she values the relationship with Walmart, which is the last national retail company standing when it comes to selling casual game DVD bundles, Lehrer said. Legacy Games also sells its games on retailers’ online websites, such as Walmart.com, Amazon.com, Staples.com, and through the following online distributors: Arvato, Avanquest, and Synnex. Additionally, Legacy Games sells its games through other traditional outlets like Steam, Microsoft Windows, and wherever casual games can be sold profitably.

“Others have said it’s a shrinking market at retail and they are going somewhere else exciting,” said Lehrer. “I think there is an opportunity here. There’s still an opportunity to sell these kinds of games at retail. I had a feeling these women were underserved. They buy their products at Walmart. They love casual games like hidden object games, or match-3, or time management, and they want to play on the PC.”

While Lehrer was gone, three part-time employees ran the company. Then she came back and she has added three more full-time employees. And now the company’s revenues are close to $1 million.

New developers

Lehrer has signed up 15 new game studios this year. These include JumpGate (Project Blue Book), Thomas Bowker (Lyne), Joel McDonald (Prune), Flippfly (Evergarden) and Walkabout (Wanderlust: Travel Stories), Joybits (Doodle God), and BufoProject (Classic Card Games 3D), among others.

“We’re going to try out different genres, different ways of packaging, different pricing and we will see what resonates,” Lehrer said.

Legacy Games has a long history of working with established casual game developers such as Artifex Mundi, Brave Giant, Alawar, Microids, Jet Dogs, Crisp App Studios, and many more. Rivals include Big Fish Games. The company has publishing contracts with more than 50 game developers, and it sells more than 500 individual games. One of the regular hits is the Amazing Games bundle at Walmart, with titles including Supernatural Stories, Fantastic Fables, True Crime, Murder Mystery, Greatest Hits, and Magical Matches.

“There are many fewer retail and digital sites to purchase casual PC games than there were a few years ago,” Lehrer said. “Many of our competitors have switched their focus to mobile. Our customers find Steam overwhelming. I believe there is a significant revenue opportunity for indie developers to reach new customers and generate incremental revenue by partnering with Legacy.”

One of the developers using Legacy’s publishing services is Aaron San Filippo, co-owner of Flippfly, a three-person studio near Madison, Wisconsin. In an interview, he said Legacy reached out to him a couple of months ago to get his game Evergarden, which is a mysterious puzzle gardening title, onto its platform. It will be launching soon in the digital store and it has a chance for physical distribution, San Filippo said.

San Filippo said he launched the game on Steam a few years ago and it didn’t connect well with that audience. Steam was more about hardcore gamers, and so the casual gaming audience of Legacy seemed a lot more appealing. The game also debuted on Linux and iOS, and it did best on iOS.

“It goes to the target market for our games,” San Filippo said. “We’re always looking for more opportunities. This is all about diversifying our income streams. Additional revenue streams are worthwhile, even if it’s small. I’m hopeful this will do well.”

GamesBeat

GamesBeat’s creed when covering the game industry is “where passion meets business.” What does this mean? We want to tell you how the news matters to you — not just as a decision-maker at a game studio, but also as a fan of games. Whether you read our articles, listen to our podcasts, or watch our videos, GamesBeat will help you learn about the industry and enjoy engaging with it.

How will you do that? Membership includes access to:

  • Newsletters, such as DeanBeat
  • The wonderful, educational, and fun speakers at our events
  • Networking opportunities
  • Special members-only interviews, chats, and “open office” events with GamesBeat staff
  • Chatting with community members, GamesBeat staff, and other guests in our Discord
  • And maybe even a fun prize or two
  • Introductions to like-minded parties

Become a member

Continue Reading

Tech

GPT-3’s free alternative GPT-Neo is something to be excited about

Published

on

GPT-3's free alternative GPT-Neo is something to be excited about

Join Transform 2021 this July 12-16. Register for the AI event of the year.


The advent of Transformers in 2017 completely changed the world of neural networks. Ever since, the core concept of Transformers has been remixed, repackaged, and rebundled in several models. The results have surpassed the state of the art in several machine learning benchmarks. In fact, currently all top benchmarks in the field of natural language processing are dominated by Transformer-based models. Some of the Transformer-family models are BERT, ALBERT, and the GPT series of models.

In any machine learning model, the most important components of the training process are:

  1. The code of the model — the components of the model and its configuration
  2. The data to be used for training
  3. The available compute power

With the Transformer family of models, researchers finally arrived at a way to increase the performance of a model infinitely: You just increase the amount of training data and compute power.

This is exactly what OpenAI did, first with GPT-2 and then with GPT-3. Being a well funded ($1 billion+) company, it could afford to train some of the biggest models in the world. A private corpus of 500 billion tokens was used for training the model, and approximately $50 million was spent in compute costs.

While the code for most of the GPT language models is open source, the model is impossible to replicate without the massive amounts of data and compute power. And OpenAI has chosen to withhold public access to its trained models, making them available via API to only a select few companies and individuals. Further, its access policy is undocumented, arbitrary, and opaque.

Genesis of GPT-Neo

Stella Biderman, Leo Gao, Sid Black, and others formed EleutherAI with the idea of making AI technology that would be open source to the world. One of the first problems the team chose to tackle was making a GPT-like language model that would be accessible to all.

As mentioned before, most of the code for such a model was already available, so the core challenges were to find the data and the compute power. The Eleuther team set out to generate an open source data set of a scale comparable to what OpenAI used for its GPT language models. This led to the creation of The Pile. The Pile, released in July 2020, is a 825GB data set specifically designed to train language models. It contains data from 22 diverse sources, including academic sources (Arxiv, PubMed, FreeLaw etc.), Internet webpages (StackExchange, Wikipedia etc.), dialogs from subtitles, Github, etc.

Source: The Pile paper, Arxiv.

For compute, EleutherAI was able to use idle compute from TPU Research Cloud (TRC). TRC is a Google Cloud initiative that supports research projects with the expectation that the results of the research will be shared with the world via open source code, models, etc.

On March 22, 2021, after months of painstaking research and training, the EleutherAI team released two trained GPT-style language models, GPT-Neo 1.3B and GPT-Neo 2.7B. The code and the trained models are open sourced under the MIT license. And the models can be used for free using HuggingFace’s Transformers platform.

Comparing GPT-Neo and GPT-3

Let’s compare GPT-Neo and GPT-3 with respect to the model size and performance benchmarks and finally look at some examples.

Model size. In terms of model size and compute, the largest GPT-Neo model consists of 2.7 billion parameters. In comparison, the GPT-3 API offers 4 models, ranging from 2.7 billion parameters to 175 billion parameters.
model size
Caption: GPT-3 parameter sizes as estimated here, and GPT-Neo as reported by EleutherAI.

As you can see, GPT-Neo is bigger than GPT-2 and comparable to the smallest GPT-3 model.

Performance benchmark metrics. EleutherAI reports that GPT-Neo outperformed the closest comparable GPT-3 model (GPT-3 Ada) on all NLP reasoning benchmarks.

GPT-Neo outperformed GPT-3 Ada on Hellaswag and Piqa. Hellaswag is an intelligent multi-choice sentence completion benchmark that has a context paragraph and four endings. Piqa measures common sense reasoning where the machine has to pick one out of two sentences that make the most sense. GPT-Neo also outperformed GPT-3 Ada on Winogrande, a benchmark that uses common sense to resolve ambiguous pronouns in a sentence.

However GPT-3 Davinci, the largest version of GPT-3, with about 65 times as many parameters, comfortably beats GPT-Neo in all the benchmarks, as you would expect.

benchmarks

Caption: Model metrics as reported by EleutherAI, except GPT-3 175B, which is from Open AI’s GPT-3 paper.

Examples. Let’s look at a few side-by-side examples of generated text from the largest GPT-3 model (from various GPT-3 Davinci examples found online) and GPT-Neo (that I generated using HuggingFace’s GPT-Neo 2.7B Transformers implementation).

The first example we will look at is completion of ELI-5 format sentences, where the text in italics was the prompt given to the model.

eli5

I would say both GPT-Neo and GPT-3 worked equally well in this example.

The next experiment is to check if GPT-Neo can correct grammatically incorrect sentences like GPT-3 can. Again the italicized parts are the prompt provided to the model.

grammar correction

Again, GPT-Neo worked great in correcting “eat” to “ate,” especially considering the model was not specifically trained to do this.

Finally, the famous “discovery of English-speaking unicorns” piece (GPT-2 version here and GPT-3 version here) reimagined and written by GPT-Neo. The prompt given to the model is in italics and bold.

In a shocking finding, scientists discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.

The only evidence was a single word: “hello.”

“The unicorns are very good listeners, so they learned to speak English from watching humans,” said Dr. David Orme, chief executive of the Zoological Society of London, in a press release. “In fact, they also learned to speak Spanish from watching humans. They said hello to us in both languages.”

While the discovery is stunning in and of itself, the news gets even more mind-blowing when you consider that the unicorns were there for thousands of years. They were “living fossils,” Orme said, which is a fancy way of saying that their origins have yet to be determined.

“It is extremely exciting that we now have evidence of a population of unicorns, that we know to be around and that is living,” Dr. John Fitzpatrick, a conservation and anthropology professor at the University of California, said in a press release. “This shows that they are not just a myth and that humans are not the only species on Earth to have developed language.”

The findings have been published in a new study in the journal Proceedings of the Royal Society B: Biological Sciences.

A ‘unicorn’ from the Andean mountains in Colombia. (Image: The Royal Society)

The discovery was made this summer in a remote but stunning valley in the Andean Mountains in Colombia called Bureta. It’s believed the unicorns were in their 20s. “It’s a very unusual place to find these animals and at the moment there is no evidence that humans have been there before,” Orme said.

The scientists said the unicorns had been living in that valley as long as their species has, which is estimated at at least 200,000 years.

This means the area’s rich history of megafauna, including dinosaurs, pterosaurs and saber-toothed cats, is still far from over.

“If it is true in a relatively isolated valley near Bureta Colombia that is more than 200,000 years old and now also having a population of these animals, then Bureta is truly a unique and special place,” Fitzpatrick said.

Once again, GPT-Neo was able to generate a coherent, almost-believable article without missing out on the central themes — unicorn discovery, the English-speaking aspect, the Andes, etc.

All in all, the performance metrics of GPT-Neo 2.7B in NLP benchmarks is better than GPT-3 2.7B (Ada), but much worse than the GPT-3 175B (Davinci). But qualitatively, GPT-Neo 2.7B’s completions and writing were as good as even GPT-3 175B (Davinci), the largest GPT-3 model.

The bottom line here is: GPT-Neo is a great open source alternative to GPT-3, especially given OpenAI’s closed access policy.

Abhishek Iyer is the founder of FreeText AI, a company specializing in text mining and review analysis.

VentureBeat

VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative technology and transact.

Our site delivers essential information on data technologies and strategies to guide you as you lead your organizations. We invite you to become a member of our community, to access:

  • up-to-date information on the subjects of interest to you
  • our newsletters
  • gated thought-leader content and discounted access to our prized events, such as Transform 2021: Learn More
  • networking features, and more

Become a member

Continue Reading

Trending