ScienceDaily
Your source for the latest research news
Follow Facebook Twitter LinkedIn Subscribe RSS Feeds Newsletters
New:
  • Planet Nine-Like Exoplanet Around Distant Star
  • Rapid Genomics Strategy to Trace Coronavirus
  • New Superhighway System in the Solar System
  • Sifting Out the First Gravitational Waves
  • Neanderthals Buried Their Dead: New Evidence
  • Spiders in Space: Making Webs Without Gravity
  • Pterosaur Precursors: Evolutionary Gap
  • Obesity Impairs Immune Cell Function
  • Science of Sandcastles Is Clarified, Finally
  • Risk of Advanced Cancers: Evolution to Blame?
advertisement
Follow all of ScienceDaily's latest research news and top science headlines!
Science News
from research organizations

1

2

Accurate neural network computer vision without the 'black box'

Date:
December 15, 2020
Source:
Duke University
Summary:
New research offers clues to what goes on inside the minds of machines as they learn to see. Instead of attempting to account for a neural network's decision-making on a post hoc basis, their method shows how the network learns along the way, by revealing how much the network calls to mind different concepts to help decipher what it sees as the image travels through successive layers.
Share:
FULL STORY

The artificial intelligence behind self-driving cars, medical image analysis and other computer vision applications relies on what's called deep neural networks.

advertisement

Loosely modeled on the brain, these consist of layers of interconnected "neurons" -- mathematical functions that send and receive information -- that "fire" in response to features of the input data. The first layer processes a raw data input -- such as pixels in an image -- and passes that information to the next layer above, triggering some of those neurons, which then pass a signal to even higher layers until eventually it arrives at a determination of what is in the input image.

But here's the problem, says Duke computer science professor Cynthia Rudin. "We can input, say, a medical image, and observe what comes out the other end ('this is a picture of a malignant lesion', but it's hard to know what happened in between."

It's what's known as the "black box" problem. What happens in the mind of the machine -- the network's hidden layers -- is often inscrutable, even to the people who built it.

"The problem with deep learning models is they're so complex that we don't actually know what they're learning," said Zhi Chen, a Ph.D. student in Rudin's lab at Duke. "They can often leverage information we don't want them to. Their reasoning processes can be completely wrong."

Rudin, Chen and Duke undergraduate Yijie Bei have come up with a way to address this issue. By modifying the reasoning process behind the predictions, it is possible that researchers can better troubleshoot the networks or understand whether they are trustworthy.

advertisement

Most approaches attempt to uncover what led a computer vision system to the right answer after the fact, by pointing to the key features or pixels that identified an image: "The growth in this chest X-ray was classified as malignant because, to the model, these areas are critical in the classification of lung cancer." Such approaches don't reveal the network's reasoning, just where it was looking.

The Duke team tried a different tack. Instead of attempting to account for a network's decision-making on a post hoc basis, their method trains the network to show its work by expressing its understanding about concepts along the way. Their method works by revealing how much the network calls to mind different concepts to help decipher what it sees. "It disentangles how different concepts are represented within the layers of the network," Rudin said.

Given an image of a library, for example, the approach makes it possible to determine whether and how much the different layers of the neural network rely on their mental representation of "books" to identify the scene.

The researchers found that, with a small adjustment to a neural network, it is possible to identify objects and scenes in images just as accurately as the original network, and yet gain substantial interpretability in the network's reasoning process. "The technique is very simple to apply," Rudin said.

The method controls the way information flows through the network. It involves replacing one standard part of a neural network with a new part. The new part constrains only a single neuron in the network to fire in response to a particular concept that humans understand. The concepts could be categories of everyday objects, such as "book" or "bike." But they could also be general characteristics, such as such as "metal," "wood," "cold" or "warm." By having only one neuron control the information about one concept at a time, it is much easier to understand how the network "thinks."

The researchers tried their approach on a neural network trained by millions of labeled images to recognize various kinds of indoor and outdoor scenes, from classrooms and food courts to playgrounds and patios. Then they turned it on images it hadn't seen before. They also looked to see which concepts the network layers drew on the most as they processed the data.

advertisement

Chen pulls up a plot showing what happened when they fed a picture of an orange sunset into the network. Their trained neural network says that warm colors in the sunset image, like orange, tend to be associated with the concept "bed" in earlier layers of the network. In short, the network activates the "bed neuron" highly in early layers. As the image travels through successive layers, the network gradually relies on a more sophisticated mental representation of each concept, and the "airplane" concept becomes more activated than the notion of beds, perhaps because "airplanes" are more often associated with skies and clouds.

It's only a small part of what's going on, to be sure. But from this trajectory the researchers are able to capture important aspects of the network's train of thought.

The researchers say their module can be wired into any neural network that recognizes images. In one experiment, they connected it to a neural network trained to detect skin cancer in photos.

Before an AI can learn to spot melanoma, it must learn what makes melanomas look different from normal moles and other benign spots on your skin, by sifting through thousands of training images labeled and marked up by skin cancer experts.

But the network appeared to be summoning up a concept of "irregular border" that it formed on its own, without help from the training labels. The people annotating the images for use in artificial intelligence applications hadn't made note of that feature, but the machine did.

"Our method revealed a shortcoming in the dataset," Rudin said. Perhaps if they had included this information in the data, it would have made it clearer whether the model was reasoning correctly. "This example just illustrates why we shouldn't put blind faith in "black box" models with no clue of what goes on inside them, especially for tricky medical diagnoses," Rudin said.

make a difference: sponsored opportunity

Story Source:

Materials provided by Duke University. Note: Content may be edited for style and length.


Journal Reference:

  1. Zhi Chen, Yijie Bei, Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2020; 2 (12): 772 DOI: 10.1038/s42256-020-00265-z

Cite This Page:

  • MLA
  • APA
  • Chicago
Duke University. "Accurate neural network computer vision without the 'black box'." ScienceDaily. ScienceDaily, 15 December 2020. <www.sciencedaily.com/releases/2020/12/201215140827.htm>.
Duke University. (2020, December 15). Accurate neural network computer vision without the 'black box'. ScienceDaily. Retrieved December 15, 2020 from www.sciencedaily.com/releases/2020/12/201215140827.htm
Duke University. "Accurate neural network computer vision without the 'black box'." ScienceDaily. www.sciencedaily.com/releases/2020/12/201215140827.htm (accessed December 15, 2020).

  • RELATED TOPICS
    • Matter & Energy
      • Telecommunications
      • Medical Technology
      • Engineering
      • Sports Science
    • Computers & Math
      • Computers and Internet
      • Neural Interfaces
      • Computer Modeling
      • Information Technology
advertisement

  • RELATED TERMS
    • Neural network
    • Artificial neural network
    • Computer worm
    • Mobile phone
    • Local area network
    • Voice over IP
    • Artificial intelligence
    • Instant messaging

1

2

3

4

5
RELATED STORIES

New Method of Analyzing Networks Reveals Hidden Patterns in Data
Sep. 10, 2019 — A new way of measuring how relationships in a network change over time can reveal important details about the network, according to researchers. For example, when applied to the world economy, the ...
All-Optical Neural Network for Deep Learning
Aug. 29, 2019 — In a key step toward making large-scale optical neural networks practical, researchers have demonstrated a first-of-its-kind multilayer all-optical artificial neural network. Researchers detail their ...
Worm 'Uploaded' to Computer and Taught Amazing Tricks
Feb. 6, 2018 — The tiny worm C. elegans is the only living being whose neural network has been analyzed completely. It can therefore be transferred to a computer, creating a virtual copy of the worm which behaves ...
Memristors Power Quick-Learning Neural Network
Dec. 22, 2017 — A new type of neural network made with memristors can dramatically improve the efficiency of teaching machines to think like humans. The network, called a reservoir computing system, could predict ...
FROM AROUND THE WEB

ScienceDaily shares links with sites in the TrendMD network and earns revenue from third-party advertisers, where indicated.
  Print   Email   Share

advertisement

1

2

3

4

5
Most Popular
this week

SPACE & TIME
New Superhighway System Discovered in the Solar System
Spiders in Space: Without Gravity, Light Becomes Key to Orientation
Exoplanet Around Distant Star Resembles Reputed 'Planet Nine' in Our Solar System
MATTER & ENERGY
New Hubble Data Explains Missing Dark Matter
Two Distinctly Different Liquid States of Water
Science of Sandcastles Is Clarified, Finally
COMPUTERS & MATH
Three Reasons Why COVID-19 Can Cause Silent Hypoxia
'Electronic Amoeba' Finds Approximate Solution to Traveling Salesman Problem in Linear Time
Video Games Can Change Your Brain
advertisement

Strange & Offbeat
 

SPACE & TIME
Researchers Identify Where Giant Jets from Black Holes Discharge Their Energy
The Farthest Galaxy in the Universe
Powerful Electrical Events Quickly Alter Surface Chemistry on Mars, Other Planetary Bodies
MATTER & ENERGY
Salt-Tolerant Bacteria With an Appetite for Sludge Make Biodegradable Plastics
Create a Realistic VR Experience Using a Normal 360-Degree Camera
When Chemistry With Green Light Mimics What Happens in Life
COMPUTERS & MATH
'Earable' Computing: A New Research Area in the Making
Challenges of Fusing Robotics and Neuroscience
'The Robot Made Me Do It': Robots Encourage Risk-Taking Behavior in People
SD
  • SD
    • Home Page
    • Top Science News
    • Latest News
  • Home
    • Home Page
    • Top Science News
    • Latest News
  • Health
    • View all the latest top news in the health sciences,
      or browse the topics below:
      Health & Medicine
      • Allergy
      • Alternative Medicine
      • Birth Control
      • Cancer
      • Diabetes
      • Diseases
      • Heart Disease
      • HIV and AIDS
      • Obesity
      • Stem Cells
      • ... more topics
      Mind & Brain
      • ADD and ADHD
      • Addiction
      • Alzheimer's
      • Autism
      • Depression
      • Headaches
      • Intelligence
      • Psychology
      • Relationships
      • Schizophrenia
      • ... more topics
      Living Well
      • Parenting
      • Pregnancy
      • Sexual Health
      • Skin Care
      • Men's Health
      • Women's Health
      • Nutrition
      • Diet and Weight Loss
      • Fitness
      • Healthy Aging
      • ... more topics
  • Tech
    • View all the latest top news in the physical sciences & technology,
      or browse the topics below:
      Matter & Energy
      • Aviation
      • Chemistry
      • Electronics
      • Fossil Fuels
      • Nanotechnology
      • Physics
      • Quantum Physics
      • Solar Energy
      • Technology
      • Wind Energy
      • ... more topics
      Space & Time
      • Astronomy
      • Black Holes
      • Dark Matter
      • Extrasolar Planets
      • Mars
      • Moon
      • Solar System
      • Space Telescopes
      • Stars
      • Sun
      • ... more topics
      Computers & Math
      • Artificial Intelligence
      • Communications
      • Computer Science
      • Hacking
      • Mathematics
      • Quantum Computers
      • Robotics
      • Software
      • Video Games
      • Virtual Reality
      • ... more topics
  • Enviro
    • View all the latest top news in the environmental sciences,
      or browse the topics below:
      Plants & Animals
      • Agriculture and Food
      • Animals
      • Biology
      • Biotechnology
      • Endangered Animals
      • Extinction
      • Genetically Modified
      • Microbes and More
      • New Species
      • Zoology
      • ... more topics
      Earth & Climate
      • Climate
      • Earthquakes
      • Environment
      • Geography
      • Geology
      • Global Warming
      • Hurricanes
      • Ozone Holes
      • Pollution
      • Weather
      • ... more topics
      Fossils & Ruins
      • Ancient Civilizations
      • Anthropology
      • Archaeology
      • Dinosaurs
      • Early Humans
      • Early Mammals
      • Evolution
      • Lost Treasures
      • Origin of Life
      • Paleontology
      • ... more topics
  • Society
    • View all the latest top news in the social sciences & education,
      or browse the topics below:
      Science & Society
      • Arts & Culture
      • Consumerism
      • Economics
      • Political Science
      • Privacy Issues
      • Public Health
      • Racial Disparity
      • Religion
      • Sports
      • World Development
      • ... more topics
      Business & Industry
      • Biotechnology & Bioengineering
      • Computers & Internet
      • Energy & Resources
      • Engineering
      • Medical Technology
      • Pharmaceuticals
      • Transportation
      • ... more topics
      Education & Learning
      • Animal Learning & Intelligence
      • Creativity
      • Educational Psychology
      • Educational Technology
      • Infant & Preschool Learning
      • Learning Disorders
      • STEM Education
      • ... more topics
  • Quirky
    • Top News
    • Human Quirks
    • Odd Creatures
    • Bizarre Things
    • Weird World
Free Subscriptions

Get the latest science news with ScienceDaily's free email newsletters, updated daily and weekly. Or view hourly updated newsfeeds in your RSS reader:

  • Email Newsletters
  • RSS Feeds
Follow Us

Keep up to date with the latest news from ScienceDaily via social networks:

  • Facebook
  • Twitter
  • LinkedIn
Have Feedback?

Tell us what you think of ScienceDaily -- we welcome both positive and negative comments. Have any problems using the site? Questions?

  • Leave Feedback
  • Contact Us
About This Site  |  Staff  |  Reviews  |  Contribute  |  Advertise  |  Privacy Policy  |  Editorial Policy  |  Terms of Use
Copyright 2020 ScienceDaily or by other parties, where indicated. All rights controlled by their respective owners.
Content on this website is for information only. It is not intended to provide medical or other professional advice.
Views expressed here do not necessarily reflect those of ScienceDaily, its staff, its contributors, or its partners.
Financial support for ScienceDaily comes from advertisements and referral programs, where indicated.
— CCPA: Do Not Sell My Information — — GDPR: Privacy Settings —