Deep Residual Shrinkage Network ayeke mbeni version so aga nzoni mingi ahon Deep Residual Network. Ti tene ni polele, Deep Residual Shrinkage Network abungbi Deep Residual Network, Attention mechanisms, na Soft thresholding functions.
E lingbi ti mä yâ ti Deep Residual Shrinkage Network tongana so. Kozoni, network ni ayeke sara kua na Attention mechanisms ti hinga a-unimportant features. Na pekoni, network ni ayeke sara kua na Soft thresholding functions ti sara si a-features so aga zero. Me, ti a-important features, network ni ayeke bata ala. Ye so ayeke sara si deep neural network ni awara ngangu mingi. Ye so ayeke mû maboko na network ni ti wara a-useful features na yâ ti signals so ayeke na noise.
1. Research Motivation
Kozoni, noise ayeke ye so e lingbi ti kpe ni pëpe tongana algorithm ayeke classer a-samples. A-exemples ti noise so ayeke Gaussian noise, pink noise, na Laplacian noise. Ti tene ni na lege ni, a-samples mingi ayeke na information so aboussoin ni ayeke pëpe. E lingbi ti bâ information so tongana noise. Noise so apeut ti sara si classification performance ni akiri na gbe ni. (Soft thresholding ayeke kota ye mingi na yâ ti a-signal denoising algorithms.)
Na exemple, zia e gbu li na ndo ti lisoro na bord ti lege. Audio ni apeut ti duti na bruit ti trompette ti oto na a-pneus ni. Peut-être e yeke sara speech recognition na ndo ti a-signals so. Bruit so ayeke na yâ ti background ayeke bousculer résultat ni. Na lege ti deep learning, deep neural network adoit ti lungula a-features ti trompette na a-pneus ni. Ye so ayeke empêcher a-features so ti bousculer résultat ti speech recognition ni.
Use ni, wungo ti noise ni ayeke changé ka mingi na popo ti a-samples. Ye so ayeke si même na yâ ti dataset oko. (Ye so akpa terê mingi na Attention mechanisms. Zia e bâ mbeni image dataset. Place ti objet ni apeut ti duti nde nde na yâ ti a-images ni. Attention mechanisms apeut ti focus na place ti objet ni na yâ ti image oko oko.)
Na exemple, zia e formé mbeni cat-and-dog classifier na a-images 5 ti “chien”. Image 1 apeut ti duti na chien na sourit. Image 2 apeut ti duti na chien na oie. Image 3 apeut ti duti na chien na kondo. Image 4 apeut ti duti na chien na lele. Image 5 apeut ti duti na chien na canard. Tongana e yeke formé classifier ni, a-objets so aboussoin ni ayeke pëpe ayeke bousculer ni. A-objets so ayeke sourit, oie, kondo, lele, na canard. Ye so ayeke sara si classification accuracy ni akiri na gbe ni. Tongana e lingbi ti hinga a-objets so. Si, e lingbi ti lungula a-features ti a-objets so. Na lege so, e lingbi ti monter accuracy ti cat-and-dog classifier ni.
2. Soft Thresholding
Soft thresholding ayeke kota ye na yâ ti a-signal denoising algorithms mingi. Algorithm ni ayeke lungula a-features tongana a-absolute values ti a-features ni ayeke kete ahon mbeni threshold. Algorithm ni ayeke shrink a-features ni ti gue na mbage ti zero tongana a-absolute values ni ayeke kota ahon threshold so. A-researchers apeut ti sara Soft thresholding na lege ti formula so:
\[y = \begin{cases} x - \tau & x > \tau \\ 0 & -\tau \le x \le \tau \\ x + \tau & x < -\tau \end{cases}\]Derivative ti Soft thresholding output na lege ti input ayeke so:
\[\frac{\partial y}{\partial x} = \begin{cases} 1 & x > \tau \\ 0 & -\tau \le x \le \tau \\ 1 & x < -\tau \end{cases}\]Formula so afa so derivative ti Soft thresholding ayeke 1 wala 0. Ye so akpa terê na ReLU activation function. Tongaso, Soft thresholding apeut ti réduire risque ti gradient vanishing na gradient exploding na yâ ti deep learning algorithms.
Na yâ ti Soft thresholding function, a-conditions so ayeke important mingi. Kozoni, threshold ni apeut ti duti negative pëpe. Use ni, threshold ni apeut ti hon maximum value ti input signal pëpe. Tongana pëpe, output ni ayeke ga kue zero.
Na ndo ni, a yeke nzoni threshold ni arespecter mbeni condition so. Sample oko oko adoit ti wara threshold ti lo mveni so adépend na noise content ti sample ni.
Ngbanga ti nyen? Ngbanga ti so noise content ayeke changé ka mingi na popo ti a-samples. Na exemple, na yâ ti dataset oko, Sample A apeut ti duti na noise mingi pëpe, me Sample B apeut ti duti na noise mingi. Tongaso, Sample A adoit ti sara kua na kete threshold. Sample B adoit ti sara kua na kota threshold. Na yâ ti deep neural networks, atâa so e lingbi ti expliquée physique ti a-features na a-thresholds so pëpe, logic ni angbâ oko. Ye so aye ti tene so, sample oko oko adoit ti wara independent threshold. Noise content ni la ayeke déterminer threshold so.
3. Attention Mechanism
A-researchers alingbi ti mä yâ ti Attention mechanisms na yâ ti computer vision hio. A-système visuel ti a-nyama apeut ti hinga a-cibles na lege ti scan ti ndo ni hio. Na pekoni, système visuel ni ayeke focus Attention na ndo ti objet ni. Ye so ayeke mû lege na système ni ti wara a-détails mingi. Na même temps, système ni ayeke ignoré information so aboussoin ni ayeke pëpe. Ti wara a-détails mingi, a yeke nzoni e bâ a-documents na ndo ti Attention mechanisms.
Squeeze-and-Excitation Network (SENet) ayeke mbeni fini deep learning method so asara kua na Attention mechanisms. Na popo ti a-samples nde nde, contribution ti a-feature channels nde nde ayeke changé. SENet ayeke sara kua na mbeni kete sub-network ti Learn a set of weights. Na pekoni, SENet ayeke multiplié a-weights so na a-features ti a-channels ni. Ye so ayeke changé grandeur ti a-features na yâ ti channel oko oko. E lingbi ti bâ process so tongana Apply weighting to each feature channel.
Na lege so, sample oko oko ayeke na a-weights ti lo mveni. Ye so aye ti tene so a-weights ti a-samples use ayeke nde nde. Na yâ ti SENet, lege ti wara a-weights ni ayeke “Global Pooling → Fully Connected Layer → ReLU Function → Fully Connected Layer → Sigmoid Function.”
4. Soft Thresholding with Deep Attention Mechanism
Deep Residual Shrinkage Network ayeke sara kua na structure ti SENet sub-network. Network ni ayeke sara kua na structure so ti implémenter Soft thresholding na gbe ti deep Attention mechanism. Sub-network ni (so ayeke na yâ ti boite so ayeke rouge) ayeke Learn a set of thresholds. Na pekoni, network ni ayeke sara Soft thresholding na a-feature channels na lege ti a-thresholds so.
Na yâ ti sub-network so, kozoni kue, système ni ayeke calculer a-absolute values ti a-features kue na yâ ti input feature map. Na pekoni, système ni ayeke sara global average pooling na moyenne ni ti wara mbeni feature, so e iri ni A. Na mbeni mbage ni, na peko ti global average pooling, système ni ayeke envoyer feature map ni na yâ ti mbeni kete fully connected network. Fully connected network so ayeke sara kua na Sigmoid function na fin ni. Function so ayeke sara si output ni ayeke na popo ti 0 na 1. Ye so ayeke mû na e mbeni coefficient, so e iri ni α. E lingbi ti tene so threshold ni ayeke α × A. Tongaso, threshold ni ayeke produit ti a-chiffres use. Mbeni chiffre ni ayeke na popo ti 0 na 1. Mbeni chiffre ni ayeke moyenne ti a-absolute values ti feature map ni. Lege so ayeke sara si threshold ni ayeke positive. Lege so ayeke sara si threshold ni aga kota mingi pëpe.
Na ndo ni, a-samples nde nde ayeke wara a-thresholds nde nde. Tongaso, e lingbi ti mä yâ ti ye so tongana mbeni spécial Attention mechanism. Mechanism ni ayeke hinga a-features so ayeke irrelevant. Mechanism ni ayeke sara si a-features so aga nduru na 0 na lege ti a-convolutional layers use. Na pekoni, mechanism ni ayeke sara si a-features so aga zero na lege ti Soft thresholding. Wala, mechanism ni ayeke hinga a-features so ayeke relevant. Mechanism ni ayeke sara si a-features so ayeke yongoro na 0 na lege ti a-convolutional layers use. Ti hunzi ni, mechanism ni ayeke bata a-features so.
Ti hunzi ni, e ayeke Stack many basic modules. E ayeke zia a-convolutional layers, Batch Normalization, activation functions, global average pooling, na fully connected output layers. Ye so ayeke formé Deep Residual Shrinkage Network so ayeke complet. E lingbi ti bâ Identity path na yâ ti diagramme ni nga.
5. Generalization Capability
Deep Residual Shrinkage Network ayeke mbeni general method ti feature learning. Ngbanga ti nyen? Ngbanga ti so na yâ ti a-kua ti feature learning mingi, a-samples ayeke na noise. A-samples ayeke nga na information so aboussoin ni ayeke pëpe. Noise na information so apeut ti sara si feature learning ague nzoni pëpe. Na exemple:
Zia e bâ image classification. Image oko apeut ti duti na a-objets mingi. E lingbi ti bâ a-objets so tongana “noise”. Deep Residual Shrinkage Network apeut ti sara kua na Attention mechanism. Network ni ayeke remarquer “noise” so. Na pekoni, network ni ayeke sara kua na Soft thresholding ti sara si a-features ti “noise” so aga zero. Ye so apeut ti sara si image classification accuracy amonter.
Zia e bâ speech recognition. Spécialement na yâ ti a-environnements so bruit ayeke mingi, tongana lisoro na bord ti lege wala na yâ ti usine. Deep Residual Shrinkage Network apeut ti sara si speech recognition accuracy amonter. Wala, network ni ayeke mû mbeni lege. Lege so apeut ti sara si speech recognition accuracy amonter.
6. Academic Impact
Paper so awara a-citations ahon 1400 na ndo ti Google Scholar.
A-researchers asara kua na Deep Residual Shrinkage Network (DRSN) na yâ ti a-publications/études ahon 1000. A-applications so ayeke na yâ ti a-domaines mingi. A-domaines so abungbi mechanical engineering, electrical power, vision, healthcare, speech, text, radar, na remote sensing.
Reference
Minghang Zhao, Shisheng Zhong, Xuyun Fu, Baoping Tang, Michael Pecht, Deep residual shrinkage networks for fault diagnosis, IEEE Transactions on Industrial Informatics, 2020, 16(7): 4681-4690.
https://ieeexplore.ieee.org/document/8850096
BibTeX
@article{Zhao2020,
author = {Minghang Zhao and Shisheng Zhong and Xuyun Fu and Baoping Tang and Michael Pecht},
title = {Deep Residual Shrinkage Networks for Fault Diagnosis},
journal = {IEEE Transactions on Industrial Informatics},
year = {2020},
volume = {16},
number = {7},
pages = {4681-4690},
doi = {10.1109/TII.2019.2943898}
}