Connect with us


Machine learning security needs new perspectives and incentives



early-exit architecture examples

Elevate your enterprise data technology and strategy at Transform 2021.

At this year’s International Conference on Learning Representations (ICLR), a team of researchers from the University of Maryland presented an attack technique meant to slow down deep learning models that have been optimized for fast and sensitive operations. The attack, aptly named DeepSloth, targets “adaptive deep neural networks,” a range of deep learning architectures that cut down computations to speed up processing.

Recent years have seen growing interest in the security of machine learning and deep learning, and there are numerous papers and techniques on hacking and defending neural networks. But one thing made DeepSloth particularly interesting: The researchers at the University of Maryland were presenting a vulnerability in a technique they themselves had developed two years earlier.

In some ways, the story of DeepSloth illustrates the challenges that the machine learning community faces. On the one hand, many researchers and developers are racing to make deep learning available to different applications. On the other hand, their innovations cause new challenges of their own. And they need to actively seek out and address those challenges before they cause irreparable damage.

Shallow-deep networks

One of the biggest hurdles of deep learning is the computational costs of training and running deep neural networks. Many deep learning models require huge amounts of memory and processing power, and therefore they can only run on servers that have abundant resources. This makes them unusable for applications that require all computations and data to remain on edge devices or need real-time inference and can’t afford the delay caused by sending their data to a cloud server.

In the past few years, machine learning researchers have developed several techniques to make neural networks less costly. One range of optimization techniques called “multi-exit architecture” stops computations when a neural network reaches acceptable accuracy. Experiments show that for many inputs, you don’t need to go through every layer of the neural network to reach a conclusive decision. Multi-exit neural networks save computation resources and bypass the calculations of the remaining layers when they become confident about their results.

Above: Experiments show that for many inputs, neural networks can reach conclusive results without processing all layers.

In 2019, Yigitcan Kaya, a Ph.D. student in Computer Science at the University of Maryland, developed a multi-exit technique called “shallow-deep network,” which could reduce the average inference cost of deep neural networks by up to 50 percent. Shallow-deep networks address the problem of “overthinking,” where deep neural networks start to perform unneeded computations that result in wasteful energy consumption and degrade the model’s performance. The shallow-deep network was accepted at the 2019 International Conference on Machine Learning (ICML).

“Early-exit models are a relatively new concept, but there is a growing interest,” Tudor Dumitras, Kaya’s research advisor and associate professor at the University of Maryland, told TechTalks. “This is because deep learning models are getting more and more expensive computationally, and researchers look for ways to make them more efficient.”

shallow-deep network

Above: Shallow-deep networks bypass the computations of neural networks and make early exits when they reach an acceptability threshold.

Dumitras has a background in cybersecurity and is also a member of the Maryland Cybersecurity Center. In the past few years, he has been engaged in research on security threats to machine learning systems. But while a lot of the work in the field focuses on adversarial attacks, Dumitras and his colleagues were interested in finding all possible attack vectors that an adversary might use against machine learning systems. Their work has spanned various fields including hardware faults, cache side-channel attacks, software bugs, and other types of attacks on neural networks.

While working on the shallow-deep network with Kaya, Dumitras and his colleagues started thinking about the harmful ways the technique might be exploited.

“We then wondered if an adversary could force the system to overthink; in other words, we wanted to see if the latency and energy savings provided by early exit models like SDN are robust against attacks,” he said.

Slowdown attacks on neural networks

tudor dumitras

Above: Tudor Dumitras, assistant professor at the University of Maryland, College Park.

Dumitras started exploring slowdown attacks on shallow-deep networks with Ionut Modoranu, then a cybersecurity research intern at the University of Maryland. When the initial work showed promising results, Kaya and Sanghyun Hong, another Ph.D. student at the University of Maryland, joined the effort. Their research eventually culminated into the DeepSloth attack.

Like adversarial attacks, DeepSloth relies on carefully crafted input that manipulates the behavior of machine learning systems. However, while classic adversarial examples force the target model to make wrong predictions, DeepSloth disrupts computations. The DeepSloth attack slows down shallow-deep networks by preventing them from making early exits and forcing them to carry out the full computations of all layers.

“Slowdown attacks have the potential of negating the benefits of multi-exit architectures,” Dumitras said. “These architectures can halve the energy consumption of a deep neural network model at inference time, and we showed that for any input we can craft a perturbation that wipes out those savings completely.”

The researchers’ findings show that the DeepSloth attack can reduce the efficacy of the multi-exit neural networks by 90-100 percent. In the simplest scenario, this can cause a deep learning system to bleed memory and compute resources and become inefficient at serving users.

But in some cases, it can cause more serious harm. For example, one use of multi-exit architectures involves splitting a deep learning model between two endpoints. The first few layers of the neural network can be installed on an edge location, such as a wearable or IoT device. The deeper layers of the network are deployed on a cloud server. The edge side of the deep learning model takes care of the simple inputs that can be confidently computed in the first few layers. In cases where the edge side of the model does not reach a conclusive result, it defers further computations to the cloud.

In such a setting, the DeepSloth attack would force the deep learning model to send all inferences to the cloud. Aside from the extra energy and server resources wasted, the attack could have much more destructive impact.

“In a scenario typical for IoT deployments, where the model is partitioned between edge devices and the cloud, DeepSloth amplifies the latency by 1.5–5X, negating the benefits of model partitioning,” Dumitras said. “This could cause the edge device to miss critical deadlines, for instance in an elderly monitoring program that uses AI to quickly detect accidents and call for help if necessary.”

While the researchers made most of their tests on shallow-deep networks, they later found that the same technique would be effective on other types of early-exit models.

Attacks in real-world settings

yigitcan kaya

Above: Yigitcan Kaya, Ph.D. student in computer science at University of Maryland, College Park.

As with most works on machine learning security, the researchers first assumed that an attacker has full knowledge of the target model and has unlimited computing resources to craft DeepSloth attacks. But the criticality of an attack also depends on whether it can be staged in practical settings, where the adversary has partial knowledge of the target and limited resources.

“In most adversarial attacks, the attacker needs to have full access to the model itself; basically, they have an exact copy of the victim model,” Kaya told TechTalks. “This, of course, is not practical in many settings where the victim model is protected from outside, for example with an API like Google Vision AI.”

To develop a realistic evaluation of the attacker, the researchers simulated an adversary who doesn’t have full knowledge of the target deep learning model. Instead, the attacker has a surrogate model on which he tests and tunes the attack. The attacker then transfers the attack to the actual target. The researchers trained surrogate models that have different neural network architectures, different training sets, and even different early-exit mechanisms.

“We find that the attacker that uses a surrogate can still cause slowdowns (between 20-50%) in the victim model,” Kaya said.

Such transfer attacks are much more realistic than full-knowledge attacks, Kaya said. And as long as the adversary has a reasonable surrogate model, he will be able to attack a black-box model, such as a machine learning system served through a web API.

“Attacking a surrogate is effective because neural networks that perform similar tasks (e.g., object classification) tend to learn similar features (e.g., shapes, edges, colors),” Kaya said.

Dumitras says DeepSloth is just the first attack that works in this threat model, and he believes more devastating slowdown attacks will be discovered. He also pointed out that, aside from multi-exit architectures, other speed optimization mechanisms are vulnerable to slowdown attacks. His research team tested DeepSloth on SkipNet, a special optimization technique for convolutional neural networks (CNN). Their findings showed that DeepSloth examples crafted for multi-exit architecture also caused slowdowns in SkipNet models.

“This suggests that the two different mechanisms might share a deeper vulnerability, yet to be characterized rigorously,” Dumitras said. “I believe that slowdown attacks may become an important threat in the future.”

Security culture in machine learning research

Adversarial example time bomb

“I don’t think any researcher today who is doing work on machine learning is ignorant of the basic security problems. Nowadays even introductory deep learning courses include recent threat models like adversarial examples,” Kaya said.

The problem, Kaya believes, has to do with adjusting incentives. “Progress is measured on standardized benchmarks and whoever develops a new technique uses these benchmarks and standard metrics to evaluate their method,” he said, adding that reviewers who decide on the fate of a paper also look at whether the method is evaluated according to their claims on suitable benchmarks.

“Of course, when a measure becomes a target, it ceases to be a good measure,” he said.

Kaya believes there should be a shift in the incentives of publications and academia. “Right now, academics have a luxury or burden to make perhaps unrealistic claims about the nature of their work,” he says. If machine learning researchers acknowledge that their solution will never see the light of day, their paper might be rejected. But their research might serve other purposes.

For example, adversarial training causes large utility drops, has poor scalability, and is difficult to get right, limitations that are unacceptable for many machine learning applications. But Kaya points out that adversarial training can have benefits that have been overlooked, such as steering models toward becoming more interpretable.

One of the implications of too much focus on benchmarks is that most machine learning researchers don’t examine the implications of their work when applied to real-world settings and realistic settings.

“Our biggest problem is that we treat machine learning security as an academic problem right now. So the problems we study and the solutions we design are also academic,” Kaya says. “We don’t know if any real-world attacker is interested in using adversarial examples or any real-world practitioner in defending against them.”

Kaya believes the machine learning community should promote and encourage research in understanding the actual adversaries of machine learning systems rather than “dreaming up our own adversaries.”

And finally, he says that authors of machine learning papers should be encouraged to do their homework and find ways to break their own solutions, as he and his colleagues did with the shallow-deep networks. And researchers should be explicit and clear about the limits and potential threats of their machine learning models and techniques.

“If we look at the papers proposing early-exit architectures, we see there’s no effort to understand security risks although they claim that these solutions are of practical value,” he says. “If an industry practitioner finds these papers and implements these solutions, they are not warned about what can go wrong. Although groups like ours try to expose potential problems, we are less visible to a practitioner who wants to use an early-exit model. Even including a paragraph about the potential risks involved in a solution goes a long way.”

Ben Dickson is a software engineer and the founder of TechTalks, a blog that explores the ways technology is solving and creating problems.

This story originally appeared on Copyright 2021


VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative technology and transact.

Our site delivers essential information on data technologies and strategies to guide you as you lead your organizations. We invite you to become a member of our community, to access:

  • up-to-date information on the subjects of interest to you
  • our newsletters
  • gated thought-leader content and discounted access to our prized events, such as Transform 2021: Learn More
  • networking features, and more

Become a member

Continue Reading
Click to comment

Leave a Reply

Your email address will not be published. Required fields are marked *


Lucidworks: Chatbots and recommendations boost online brand loyalty



Who is loyal

Elevate your enterprise data technology and strategy at Transform 2021.

Pandemic-related shutdowns led consumers to divert the bulk of their shopping to online — and many of those shoppers are now hesitant about returning to stores as businesses begin to open back up. A recent survey of 800 consumers conducted by cloud company Lucidworks found that 59% of shoppers plan to either avoid in-person shopping as much as possible,  or visit in-person stores less often than before the pandemic.

Above: Shoppers across the U.S. and U.K. agree that high-quality products, personalized recommendations, and excellent customer service are the top three reasons they’re brand-loyal.

Image Credit: Lucidworks

As the world stabilizes, shoppers want brands to provide a multi-faceted shopping experience — expanded chatbot capabilities, diverse recommendations, and personalized experiences that take into account personal preferences and history, Lucidworks found in its study. More than half of shoppers in the survey, 55%, said they use a site’s chatbot on every visit. American shoppers use chatbots more than their counterparts in the United Kingdom, at 70%.

The majority of shoppers, 70%, use chatbots for customer service, and 53% said they want a chatbot to help them find specific products or check product compatibility. A little less than half, or 48%, said they use chatbots to find more information about a product, and 42% use chatbots to find policies such as shipping information and how to get refunds.

A quarter of shoppers will leave the website to seek information elsewhere if the chatbot doesn’t give them the answer. Brands that deploy chatbots capable of going beyond basic FAQs and can perform product and content discovery will provide the well-rounded chatbot experience shoppers expect, Lucidworks said.

Respondents also pointed to the importance of content recommendations. The survey found that almost a third of shoppers said they find recommendations for “suggested content” useful, and 61% of shoppers like to do research via reviews on the brand’s website where they’ll be purchasing from. A little over a third — 37% — of shoppers use marketplaces such as Amazon, Google Shopping, and eBay for their research.

Brands should try to offer something for every step in the shopping journey, from research to purchase to support, to keep shoppers on their sites longer. How online shopping will look in coming years is being defined at this very moment as the world reopens. Brands that are able to understand a shopper’s goal in the moment and deliver a connected experience that understands who shoppers are and what they like are well-positioned for the future, Lucidworks said.

Lucidworks used a self-serve survey tool, Pollfish, in late May 2021 to survey 800 consumers over the age of 18—400 in the U.K. and 400 in the U.S.—to understand how shoppers interact with chatbots, product and content recommendations, where they prefer to do research, and plans for future in-store shopping.

Read the full U.S./U.K. Consumer Survey Report from Lucidworks.


VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative technology and transact.

Our site delivers essential information on data technologies and strategies to guide you as you lead your organizations. We invite you to become a member of our community, to access:

  • up-to-date information on the subjects of interest to you
  • our newsletters
  • gated thought-leader content and discounted access to our prized events, such as Transform 2021: Learn More
  • networking features, and more

Become a member

Continue Reading


Breakroom teams up with High Fidelity to bring 3D audio to online meetings



Breakroom teams up with High Fidelity to bring 3D audio to online meetings

Elevate your enterprise data technology and strategy at Transform 2021.

Social meeting space Breakroom has integrated High Fidelity‘s 3D audio into its 3D virtual world for social and business events.

The deal is a convergence of some virtual world pioneers who have made their mark on the development of virtual life. Philip Rosedale is the CEO of High Fidelity, and he also launched Second Life in 2003. And Sine Wave Entertainment, the creator of Breakroom, got its start as a content brand in Second Life before it spun out to create its own virtual meeting spaces for real world events.

Adam Frisby, chief product officer and cofounder of Sine Wave, said in our interview conducted inside Breakroom that the High Fidelity spatial audio will help Breakroom create a triple-A quality experience in a virtual world.

“The real benefit of having 3D audio in a virtual world like this is you can have lots of conversations going on simultaneously,” said Frisby. “3D audio is the only way to replicate the real-world experience in an online environment. You can have a 150-person conference and end up with 10 groups of people talking at the same time. That has helped us with engagement.”

Above: Breakroom lets an event have dozens of simultaneous conversations where people don’t talk over each other, thanks to High Fidelity.

Image Credit: Sine Wave

Most online events get engagement times of 20 or 30 minutes. But Breakroom’s average events, ranging from 600 to 1,000 attendees, have engagement times of an hour and 40 minutes, Frisby said.

Sine Wave’s Breakroom draws heavily on lessons learned in Second Life to create a frictionless, mass market, user-friendly virtual world.

“You can hear everything better with High Fidelity,” said Rosedale, in our interview in Breakroom. “Breakroom combines low-latency server-side video and spatial audio in a way that lets you hold an event like it’s in the real world.”

High Fidelity is a real-time communications company. Its mission is to build technologies that power more human experiences in today’s digital world. The company’s patented spatial audio technology, originally developed for its VR software platform, adds immersive, high-quality voice chat to any application — for groups of any size. You can really tell how close you are to someone in a High Fidelity space when they talk to you, as voices become fainter the farther away they are.

“We are super excited about this general direction and we wound up building the audio subsystem and extracting that first,” Rosedale said. “It works well where there is no possibility of face-to-face meetings.”

breakroom 3

Above: I could hear Philip Rosedale’s voice clearly in this conversation in Breakroom.

Image Credit: Sine Wave

Spatial audio in a 3D virtual world helps encourage spontaneous conversations into a fun, productive setting, in a way that flatscreen video calls and webinars simply can’t match, Frisby said. It’s easy to tell in Breakroom who is speaking to you, and from what direction.

It took me a little while to figure out how to unmute my voice. Rosedale was jumping up and down while we were talking.

“It’s all remote rendered. And that means that we can bring people in on a variety of platforms,” Frisby said. “No matter what your target hardware is, you can actually get in here and still get good high fidelity. It’s a good quality 3D rendering experience here regardless of what device you’re on.”

I asked Rosedale if he could hear me chewing lettuce, which sounded very loud on my headsets. But he said no. It definitely helps if you have good headsets with 3D audio.

Breakroom is being used by organizations like Stanford University, the United Nations, and The Economist. Breakroom runs on any device with a Chrome browser, offering good 3D graphics and audio quality, with no installation required.

Frisby said that Breakroom is also a way for companies to enable remote workers to gather and meet each other in more relaxed environments as if it were an intermediate space between online-only environments and going back to work in offices.

breakroom 4

Above: Breakroom and High Fidelity are enabling conferences with spatial audio.

Image Credit: Sine Wave

Its full suite of communication tools includes voice chat, instant messenger, and in-world email. It has video conferencing, media sharing, and desktop sharing tools. It has a diverse range of fully customizable avatars and scenes. You can get around just by pointing and clicking on the environment.

It also has event management tools to facilitate conversation and agenda flow, branded interactive exhibition stands, and private meeting rooms, available for rent by sponsors. It has environments including dance clubs, beach and mountain retreats, casual games, quiz shows, and live music/comedy shows. It has an integrated shop where brands can upload and sell their content to customers for real cash.

It gives you the ability to seamlessly license and import any item from the Unity Asset Store (Sine Wave is a verified partner of Unity). The iOS and Android version of Breakroom is in closed beta and Breakroom for consoles and the Oculus Quest 2 coming soon. It has LinkedIn and Eventbrite integration, including ticket sales. It also has a self-serve portal for customers to quickly customize and configure their organizations’ Breakroom, as well as sub-licensing agreements which enable Breakroom customers to host and monetize events and experiences to their own customer base.

Frisby said it has been a technical challenge so that people don’t get kicked out of the room, but his team has managed to refine the technology during the pandemic. He thinks conferences are great use cases for the technology because so many people come together simultaneously and push the tech to the limit.

As for High Fidelity, Rosedale believes that the education market will come around, and the whole world will eventually move to better spatial experiences.


GamesBeat’s creed when covering the game industry is “where passion meets business.” What does this mean? We want to tell you how the news matters to you — not just as a decision-maker at a game studio, but also as a fan of games. Whether you read our articles, listen to our podcasts, or watch our videos, GamesBeat will help you learn about the industry and enjoy engaging with it.

How will you do that? Membership includes access to:

  • Newsletters, such as DeanBeat
  • The wonderful, educational, and fun speakers at our events
  • Networking opportunities
  • Special members-only interviews, chats, and “open office” events with GamesBeat staff
  • Chatting with community members, GamesBeat staff, and other guests in our Discord
  • And maybe even a fun prize or two
  • Introductions to like-minded parties

Become a member

Continue Reading


Moderne helps companies automate their code migration and fixes



Elevate your enterprise data technology and strategy at Transform 2021.

While every company may well be a software company these days, the software development sphere has evolved greatly over the past decade to get to this stage, with developer operations (DevOps), agile, and cloud-native considerations at the forefront.

Moreover, with APIs and open source software now serving as critical components of most modern software stacks, tracking code changes and vulnerabilities introduced by external developers can be a major challenge. This is something fledgling startup Moderne is setting out to solve with a platform that promises to automatically “fix, upgrade, and secure code” in minutes, including offering support for framework or API migrations and applying CVE (common vulnerabilities and exposures) patches.

The Seattle-based company, which will remain in private beta for the foreseeable future, today announced a $4.7 million seed round of funding to bring its SaaS product to market. The investment was led by True Ventures, with participation from a slew of angel and VC backers, including GitHub CTO Jason Warner; Datadog cofounder and CEO Olivier Pomel; Coverity cofounder Andy Chou; Mango Capital; and

Version control

If a third-party API provider or open source framework is updated, with the older version no longer actively supported, companies need to ensure their software remains secure and compliant. “It requires revving dependencies [updating version numbers in configuration files] and changing all the call sites for the APIs that have changed — it’s tedious, repetitive, but hasn’t been automated,” Moderne CEO and cofounder Jonathan Schneider told VentureBeat.

Moderne is built on top of OpenRewrite, an open source automated code refactoring tool for Java that Schneider developed at Netflix several years ago. While developers can already use the built-in refactoring and semantic search features included in integrated development environments (IDEs), if they need to perform a migration or apply a CVE patch, they have to follow multiple manual steps. Moreover, they can only work on a single repository at a time.

“So if an organization has hundreds of microservices — which is not uncommon for even very small organizations, and larger ones have thousands — each repository needs to be loaded into [the] IDE and operated one by one,” Schneider said. “A developer can spend weeks or months doing this across the codebase.”

OpenRewrite, on the other hand, provides “building blocks” — individual search and refactoring operations — that can be composed into an automated sequence called recipes anyone can use. Moderne’s offering complements OpenRewrite and allows companies to apply these recipes in bulk to their codebases.

Above: Moderne screenshot

Enterprises, specifically, can accumulate vast amounts of code. One of Moderne’s early product design partners is a “large financial institution” that incorporates some 250 million lines of Java code — or “one-eighth of all GitHub Java code,” Schneider noted, adding that this is actually on the “low to medium” side for what a typical enterprise might have.

“Some of this code is obsolete (e.g. accrued through historical acquisitions), some is under rapid development (e.g. mobile apps) — but the majority represents super valuable business assets, such as ATM software and branch management software,” Schneider said.

And let’s say a company decides to redeploy developers internally to work on rapid development projects — it still needs to consider the core software components that underpin the business and need to be maintained. Moderne automates the code migration and CVE patching process, freeing developers to work on other mission-critical projects.

When Moderne eventually goes to market, it will adopt an open core business model, with a free plan for the open source community and individual users, while the premium SaaS plan will support larger codebases and teams with additional features for collaboration.

The company said it will use its fresh cash injection to grow a “vibrant open source community for OpenRewrite,” expand its internal engineering team, and bolster its SaaS product ahead of launch.


VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative technology and transact.

Our site delivers essential information on data technologies and strategies to guide you as you lead your organizations. We invite you to become a member of our community, to access:

  • up-to-date information on the subjects of interest to you
  • our newsletters
  • gated thought-leader content and discounted access to our prized events, such as Transform 2021: Learn More
  • networking features, and more

Become a member

Continue Reading