Deep Residual Shrinkage Network: O se Metotia Artificial Intelligence mo Data e tele Noise

O le Deep Residual Shrinkage Network o se improved variant o le Deep Residual Network. O le mea moni, o le Deep Residual Shrinkage Network e tu’ufa’atasia ai le Deep Residual Network, attention mechanisms, ma soft thresholding functions.

E mafai ona tatou malamalama i le working principle o le Deep Residual Shrinkage Network i le auala lenei. Muamua, e fa’aaoga e le network ia attention mechanisms e iloa ai unimportant features. Ona fa’aaoga lea e le network ia soft thresholding functions e seti ai nei unimportant features i le zero. I se isi itu, e iloa e le network ia important features ma taofi nei important features. O lenei fa’agasologa e fa’aleleia ai le ability o le deep neural network. O lenei fa’agasologa e fesoasoani i le network e extract ai useful features mai signals o lo’o iai noise.

1. Research Motivation (Mafua’aga o le Su’esu’ega)

Muamua, o le noise e le mafai ona ‘alo’ese mai ai pe a fa’avasega (classify) e le algorithm ia samples. O fa’ata’ita’iga o lenei noise e aofia ai Gaussian noise, pink noise, ma Laplacian noise. I se fa’aupuga lautele, o samples e masani ona iai information e le feso’ota’i (irrelevant) ma le current classification task. E mafai ona tatou fa’auigaina lenei irrelevant information o se noise. O lenei noise e ono fa’aitiitia ai le classification performance. (O le Soft thresholding o se la’asaga autu (key step) i le tele o signal denoising algorithms.)

Fa’ata’ita’iga, mafaufau i se talanoaga i tafatafa o le auala. O le audio e ono aofia ai leo o pu o ta’avale ma uili. Atonu tatou te faia se speech recognition i luga o nei signals. O leo i tua (background sounds) o le a mautinoa le a’afia ai o results. Mai le va’aiga o le deep learning, e tatau i le deep neural network ona aveese (eliminate) ia features o lo’o feso’ota’i ma pu ma uili. O lenei elimination e taofia ai features mai le a’afia ai o speech recognition results.

Lona lua, o le tele o le noise e masani ona fesuisuia’i i le va o samples. O lenei suiga e tupu tusa lava pe i totonu o le dataset e tasi. (O lenei suiga e iai uiga tutusa ma attention mechanisms. Ave se image dataset e fai ma fa’ata’ita’iga. O le nofoaga o le target object e ono ‘ese’ese i images. E mafai e attention mechanisms ona taula’i (focus) i le nofoaga fa’apitoa o le target object i image ta’itasi.)

Mo se fa’ata’ita’iga, mafaufau i le a’oa’oina (training) o se cat-and-dog classifier fa’atasi ai ma images e lima ua fa’aigoaina o le “maile” (dog).

• Image 1 e ono iai se maile ma se ‘imoa (mouse).
• Image 2 e ono iai se maile ma se kusi (goose).
• Image 3 e ono iai se maile ma se moa (chicken).
• Image 4 e ono iai se maile ma se asini (donkey).
• Image 5 e ono iai se maile ma se pato (duck).

I le taimi o le training, o irrelevant objects o le a fa’alavelave i le classifier. O nei objects e aofia ai ‘imoa, kusi, moa, asini, ma pato. O lenei fa’alavelave e i’u ai i le fa’aitiitia o le classification accuracy. Fa’apea e mafai ona tatou iloa nei irrelevant objects. Ona mafai lea ona tatou aveese (eliminate) ia features e feso’ota’i ma nei objects. I lenei auala, e mafai ai ona tatou fa’aleleia le accuracy o le cat-and-dog classifier.

2. Soft Thresholding

O le Soft thresholding o se la’asaga autu (core step) i le tele o signal denoising algorithms. E aveese (eliminate) e le algorithm ia features pe afai o absolute values o features e maualalo ifo nai lo se threshold fa’apitoa. E fa’aitiitia (shrinks) e le algorithm ia features agai i le zero pe afai o absolute values o features e maualuga atu nai lo lenei threshold. E mafai e researchers ona fa’atino (implement) le soft thresholding e fa’aaoga ai le formula lenei:

O le derivative o le soft thresholding output e tusa ai ma le input o le:

O le formula i luga o lo’o fa’aalia ai o le derivative o le soft thresholding e tasi (1) po’o le leai (0). O lenei uiga e tutusa lelei ma le uiga o le ReLU activation function. O le mea lea, e mafai ai e le soft thresholding ona fa’aitiitia le risk o le gradient vanishing ma le gradient exploding i totonu o deep learning algorithms.

I totonu o le soft thresholding function, o le setiina o le threshold e tatau ona fa’amalieina tulaga e lua. Muamua, o le threshold e tatau ona avea ma numera positive. Lona lua, o le threshold e le tatau ona sili atu i le maximum value o le input signal. A leai, o le output o le a avea atoa ma zero.

E le gata i lea, e sili pe a fa’amalieina e le threshold se tulaga lona tolu. E tatau i sample ta’itasi ona iai lana lava threshold tuto’atasi (independent threshold) e fa’avae i luga o le noise content o le sample.

O le mafua’aga ona o le noise content e masani ona ‘ese’ese i le va o samples. Fa’ata’ita’iga, atonu e la’ititi le noise i le Sample A ae tele le noise i le Sample B i totonu o le dataset lava e tasi. I lenei tulaga, e tatau i le Sample A ona fa’aaoga se threshold la’ititi i le taimi o le soft thresholding. E tatau i le Sample B ona fa’aaoga se threshold tele. I totonu o deep neural networks, e ui ina leiloa explicit physical definitions a nei features ma thresholds, ae tumau pea le basic logic. O lona uiga, e tatau i sample ta’itasi ona iai se independent threshold. O le specific noise content e fa’atonuina lenei threshold.

3. Attention Mechanism

E faigofie ona malamalama researchers i attention mechanisms i le matata o computer vision. O le visual systems a manu e mafai ona fa’avasega targets e ala i le vave va’ai (scanning) o le eria atoa. Mulimuli ane, e taula’i le attention a visual systems i le target object. O lenei gaioiga e fa’atagaina ai systems e extract nisi fa’amatalaga au’ili’ili (details). I le taimi lava e tasi, e taofia (suppress) e systems ia irrelevant information. Mo fa’amatalaga au’ili’ili, fa’amolemole va’ai i literature e uiga i attention mechanisms.

O le Squeeze-and-Excitation Network (SENet) o lo’o fa’atusalia ai se deep learning method fou e fa’aaoga ai attention mechanisms. I samples ‘ese’ese, e ‘ese’ese fo’i le sao o feature channels i le classification task. E fa’aaoga e le SENet se sub-network la’ititi e maua ai se seti o weights (Learn a set of weights). Ona fa’atele lea e le SENet o nei weights i features o channels ta’itasi. O lenei gaioiga e fetu’una’i ai le tele (magnitude) o features i channel ta’itasi. E mafai ona tatou va’ai i lenei fa’agasologa o le Apply weighting to each feature channel (fa’aogaina o le weighting i feature channel ta’itasi).

I lenei auala, o sample ta’itasi e iai lana lava seti tuto’atasi o weights. O lona uiga, o weights mo so’o se samples e lua e ‘ese’ese. I le SENet, o le specific path mo le mauaina o weights o le “Global Pooling → Fully Connected Layer → ReLU Function → Fully Connected Layer → Sigmoid Function.”

4. Soft Thresholding with Deep Attention Mechanism

O le Deep Residual Shrinkage Network e fa’aaoga le fausaga o le SENet sub-network. E fa’aaoga e le network lenei fausaga e fa’atino ai le soft thresholding i lalo o le deep attention mechanism. O le sub-network (o lo’o fa’ailoa i totonu o le pusa mumu) e a’oa’oina se seti o thresholds (Learn a set of thresholds). Ona fa’aaoga lea e le network le soft thresholding i feature channel ta’itasi e fa’aaoga ai nei thresholds.

I totonu o lenei sub-network:

1. Muamua, e fa’atusatusa e le system ia absolute values o features uma i totonu o le input feature map.
2. Ona faia lea e le system le global average pooling ma le averaging e maua ai se feature, fa’ailogaina o le A.
3. I le isi ala (path), e tu’uina atu e le system le feature map i totonu o se fully connected network la’ititi pe a uma le global average pooling.
4. O lenei fully connected network e fa’aaoga le Sigmoid function o se layer mulimuli.
5. O lenei function e fa’atonutonu (normalizes) le output i le va o le 0 ma le 1.
6. O lenei fa’agasologa e maua ai se coefficient, fa’ailogaina o le α.

E mafai ona tatou fa’amatala le final threshold o le α × A. O le mea lea, o le threshold o le fua (product) o numera e lua. O le tasi numera o lo’o i le va o le 0 ma le 1. O le isi numera o le average o absolute values o le feature map. O lenei auala e fa’amautinoa ai o le threshold e positive. O lenei auala e fa’amautinoa ai fo’i o le threshold e le tetele naua.

E le gata i lea, o samples ‘ese’ese e maua ai thresholds ‘ese’ese. O le i’uga, e mafai ona tatou malamalama i lenei auala o se specialized attention mechanism. O le mechanism e iloa ai features e le feso’ota’i (irrelevant) ma le current task. O le mechanism e liliu ai nei features i values e latalata i le zero e ala i convolutional layers e lua. Ona seti lea e le mechanism nei features i le zero e fa’aaoga ai le soft thresholding. Pe, o le mechanism e iloa ai features e feso’ota’i (relevant) ma le current task. O le mechanism e liliu ai nei features i values e mamao mai le zero e ala i convolutional layers e lua. Mulimuli ane, e fa’asaoina (preserves) e le mechanism nei features.

I le fa’ai’uga, matou te fa’aputuina (stack) se numera o basic modules (Stack many basic modules). Matou te fa’aaofia fo’i convolutional layers, batch normalization, activation functions, global average pooling, ma fully connected output layers. E tatau foi ona iai le Identity path mo le sologa lelei o le a’oa’oina. O lenei fa’agasologa e fausia ai le Deep Residual Shrinkage Network atoa.

5. Generalization Capability (Mafai ona Fa’aaoga Lautele)

O le Deep Residual Shrinkage Network o se general method mo feature learning. O le mafua’aga ona o samples e masani ona iai noise i le tele o feature learning tasks. E iai fo’i i samples ia irrelevant information. O lenei noise ma irrelevant information e ono a’afia ai le performance o feature learning. Fa’ata’ita’iga:

Mafaufau i le image classification. O se image e ono iai le tele o isi objects i le taimi e tasi. E mafai ona tatou malamalama i nei objects o se “noise”. O le Deep Residual Shrinkage Network atonu e mafai ona fa’aaoga le attention mechanism. E matauina e le network lenei “noise”. Ona fa’aaoga lea e le network le soft thresholding e seti ai features e feso’ota’i ma lenei “noise” i le zero. O lenei gaioiga e ono fa’aleleia ai le accuracy o le image classification.

Mafaufau i le speech recognition. Aemaise lava, mafaufau i environments e pisa e pei o nofoaga o talanoaga i tafatafa o le auala po’o totonu o se falegaosimea (factory workshop). O le Deep Residual Shrinkage Network e ono fa’aleleia le accuracy o le speech recognition. Po’o le mea sili, o le network e ofoina atu se auala (methodology). O lenei methodology e mafai ona fa’aleleia le speech recognition accuracy.

Reference

Minghang Zhao, Shisheng Zhong, Xuyun Fu, Baoping Tang, Michael Pecht, Deep residual shrinkage networks for fault diagnosis, IEEE Transactions on Industrial Informatics, 2020, 16(7): 4681-4690.

https://ieeexplore.ieee.org/document/8850096

BibTeX

@article{Zhao2020,
  author    = {Minghang Zhao and Shisheng Zhong and Xuyun Fu and Baoping Tang and Michael Pecht},
  title     = {Deep Residual Shrinkage Networks for Fault Diagnosis},
  journal   = {IEEE Transactions on Industrial Informatics},
  year      = {2020},
  volume    = {16},
  number    = {7},
  pages     = {4681-4690},
  doi       = {10.1109/TII.2019.2943898}
}

Academic Impact (A’afiaga Fa’aleaoaoga)

O lenei paper ua maua le silia ma le 1,400 citations i luga o le Google Scholar.

E tusa ai ma statistics e le’i mae’a, ua fa’aaoga e researchers le Deep Residual Shrinkage Network (DRSN) i le silia ma le 1,000 publications/studies. O nei applications e aofia ai le tele o matata. O nei matata e aofia ai mechanical engineering, electrical power, vision, healthcare, speech, text, radar, ma remote sensing.