Deep Residual Shrinkage Network ab pexe la buñu gëna defar te mu jóge ci Deep Residual Network. Ci lu gatt, dafa boole Deep Residual Network, attention mechanisms, ak soft thresholding functions.
Ci lu yomb, ni Deep Residual Shrinkage Network di doxee mooy: dafay jëfandikoo attention mechanisms ngir xamme features yi amul solo, te jëfandikoo soft thresholding functions ngir def leen ñu nekk zero; waaye ci beneen wàll, dafay xamme features yi am solo te denc leen. Process bii dafay gëna dëgëral kàttanu deep neural network bi ngir mu mën a génne features yu am njariñ ci signals yi am noise.
1. Research Motivation (Li tax ñu def gëstu bi)
Bu njëkk, suñuy def classification ci samples yi, noise yi—naka Gaussian noise, pink noise, ak Laplacian noise—dañuy faral di am, te mënul ñàkk. Ci lu gëna yaatu, samples yi dañuy faral di am xibaar (information) yo xamne amul solo ci classification task bi ñu nekk. Xibaar yooyu itam, mën nañu leen jàppe ni noise. Noise yooyu mën nañu yàq classification performance bi. (Soft thresholding nekk na mbir mu am solo lool ci signal denoising algorithms yu bari.)
Misaal, su nit ñi di waxtaan ci boru mbedd, audio bi mën na ëmb coowu liir-liiru oto wala coowu ruku oto yi. Su ñuy def speech recognition ci signals yooyu, results yi dinañu soppeeku ndax coow yooyu nekk ci background bi. Ci gijaayu deep learning, features yi nga xamne ñoo méngóo ak liir-liiru oto yi walla ruku yi, war nañu leen dindi ci biir deep neural network bi ngir bañ ñu yàq speech recognition results yi.
Bakaat bi ci top (Secondly), donte ci biir benn dataset la, noise bi nekk ci sample bu nekk dafay faral di wuute. (Li dafa nuru attention mechanisms; su ñu jëlee ab image dataset def ko misaal, bërëb bi target object bi nekk mën na wuute ci images yi, te attention mechanisms mën nañu jox cëf (focus) ci bërëb bi target object bi nekk ci image bu nekk.)
Misaal, su ñuy train ab cat-and-dog classifier, jëlal juroomi images yo xamne dañu leen label “dog” (xaj). Image bu njëkk bi mën na am xaj ak jinax, ñaareelu image bi am xaj ak géner, ñetteelu image bi am xaj ak ginaar, ñeenteelu image bi am xaj ak mbaam, juroomeelu image bi am xaj ak kanaara. Ci biir training bi, classifier bi dina am jaxa-jaxa ndax objects yi ci nekk te amul solo, naka jinax yi, géner yi, ginaar yi, mbaam yi, ak kanaara yi. Loolu dina tax classification accuracy bi wàcc. Su ñu mënee xamme objects yooyu yépp te dindi features yi ci aju, mën nañu gëna yékkati accuracy bu cat-and-dog classifier bi.
2. Soft Thresholding
Soft thresholding nekk na mbir mu am solo lool ci signal denoising algorithms yu bari. Dafay dindi features yi nga xamne seen absolute values dafa gëna tuuti ci ab threshold, te dafay “shrink” (wàññi) features yi nga xamne seen absolute values dafa ëpp threshold bi, jëme leen ci zero. Mën nañu ko def ci formula bii:
\[y = \begin{cases} x - \tau & x > \tau \\ 0 & -\tau \le x \le \tau \\ x + \tau & x < -\tau \end{cases}\]Derivative bu soft thresholding output bu ñu ko nattale ak input bi mooy:
\[\frac{\partial y}{\partial x} = \begin{cases} 1 & x > \tau \\ 0 & -\tau \le x \le \tau \\ 1 & x < -\tau \end{cases}\]Ni ñu ko gisee ci kaw, derivative bu soft thresholding benn la (1) wala zero (0). Jikko bii dafa niroo lool ak ReLU activation function. Li ko waral, soft thresholding mën na dimbali deep learning algorithms yi bañ gradient vanishing ak gradient exploding.
Ci biir soft thresholding function, threshold bi dafa wara méngóo ak ñaari conditions: Bu njëkk, threshold bi dafa wara nekk positive number; ñaareel bi, threshold bi warul ëpp maximum value bu input signal bi, su ko deful output bi yépp day nekk zero.
Te itam, threshold bi dafa wara méngóo ak ñetteelu condition: sample bu nekk dafa wara am threshold bu moom boppam (independent threshold) aju ci noise content bi mu ëmb.
Li ko waral mooy, noise content bi dafay faral di wuute ci diggante samples yi. Misaal, dafa yomb ci biir benn dataset, Sample A am noise bu tuuti, waaye Sample B am noise bu bari. Ci situation bii, su ñuy def soft thresholding ci denoising algorithm, Sample A dafa wara jëfandikoo threshold bu tuuti, waaye Sample B dafa wara jëfandikoo threshold bu mag. Donte ci biir deep neural networks, features yii ak thresholds yii dañuy ñàkk seen maanaa dëgg-dëgg ci wàllu physique, waaye logic bi ci biir mooy beneen bi. Maanaam, sample bu nekk dafa wara am threshold bu moom boppam te aju ci noise bi mu ëmb.
3. Attention Mechanism
Attention mechanisms yomb na xam ci wàllu computer vision. Gëtu mala yi (visual systems of animals) mën nañu xamme targets yi: dañuy gaaw seet zone bi yépp, ci saasi ñu jox attention ci target object bi ngir gis details yu bari, te bayyi xibaar yi amul solo. Ngir xam lu ci gëna leer, mën ngeen seet literature bi aju ci attention mechanisms.
Squeeze-and-Excitation Network (SENet) ab pexe deep learning bu bees la bu yor attention mechanisms. Ci samples yu wuute, feature channels yu wuute dañuy am solo yu wuute ci classification task bi. SENet dafay jëfandikoo ab sub-network bu tuuti ngir am Learn a set of weights (am ab setu weights), te dina multiplier weights yooyu ak features yu channel bu nekk ngir soppi maggaayu features yi. Process bii, mën nañu ko jàppe ni Apply weighting to each feature channel.
Ci pexe bii, sample bu nekk dafa am set of weights bu moom boppam. Maanaam, weights yu ñaari samples yu wuute dañuy wuute. Ci biir SENet, yoon bi ñuy jaar ngir am weights yi mooy “Global Pooling → Fully Connected Layer → ReLU Function → Fully Connected Layer → Sigmoid Function.”
4. Soft Thresholding ak Deep Attention Mechanism
Deep Residual Shrinkage Network dafa jël xalaat bi ci sub-network structure bu SENet bi ñu wax ci kaw, ngir def soft thresholding ci biir deep attention mechanism. Ci biir sub-network bii (bi nekk ci biir red box bi), mën nañu Learn a set of thresholds ngir def soft thresholding ci feature channel bu nekk.
Ci biir sub-network bii, dañuy njëkk calculé absolute values bu mbooleem features yi nekk ci input feature map bi. Ci biir global average pooling ak averaging, dañuy am ab feature, bu ñu tudde A. Ci beneen Identity path bi, feature map bi gannaaw global average pooling dañu koy dugal ci fully connected network bu tuuti. Fully connected network bii dafay jëfandikoo Sigmoid function ci layer bu mujj bi ngir output bi nekk ci diggante 0 ak 1, loolu jox ñu ab coefficient bu ñu tudde α. Threshold bi mujj mooy nekk α × A. Kon, threshold bi mooy ab lim ci diggante 0 ak 1 bu ñu multiplier ak average bu absolute values yu feature map bi. Pexe bii dafay tax threshold bi nekk positive, te du rëy torop.
Te itam, samples yu wuute dañuy am thresholds yu wuute. Kon, ci lu gatt, mën nañu ko jàppe ni ab attention mechanism bu special: dafay xamme features yi amul solo ci task bi, soppi leen ñu jege zero jaarale ko ci ñaari convolutional layers, te def soft thresholding ngir ñu nekk zero; walla, dafay xamme features yi am solo, soppi leen ñu soré zero, te denc leen.
Ci mujj gi, dañuy Stack many basic modules (teg modules yu bari) ak convolutional layers, batch normalization, activation functions, global average pooling, ak fully connected output layers, ngir am Deep Residual Shrinkage Network bi mat sëkk.
5. Generalization Capability (Mën a dox ci leneen)
Deep Residual Shrinkage Network, ci dëgg-dëgg, ab pexe feature learning la bu méngóo ak lu bari (general). Li ko waral mooy, ci feature learning tasks yu bari, samples yi dañuy faral di am noise ak xibaar (information) yo xamne amul solo. Noise yii ak xibaar yu amul solo mën nañu yàq performance bu feature learning bi. Misaal:
Ci image classification, su ab image amee objects yu bari, objects yooyu mën nañu leen jàppe ni “noise.” Deep Residual Shrinkage Network mën na jëfandikoo attention mechanism ngir xamme “noise” yooyu, te jëfandikoo soft thresholding ngir features yi méngóo ak “noise” yooyu nekk zero. Loolu mën na yékkati accuracy bu image classification bi.
Ci speech recognition, rawatina ci bërëb yu am coow lu bari naka boru mbedd wala ci biir usine, Deep Residual Shrinkage Network mën na yékkati accuracy bu speech recognition bi, wala mu joxe ab pexe ngir yékkati ko.
Reference (Téere yi ñu jëfandikoo)
Minghang Zhao, Shisheng Zhong, Xuyun Fu, Baoping Tang, Michael Pecht, Deep residual shrinkage networks for fault diagnosis, IEEE Transactions on Industrial Informatics, 2020, 16(7): 4681-4690.
https://ieeexplore.ieee.org/document/8850096
BibTeX
@article{Zhao2020,
author = {Minghang Zhao and Shisheng Zhong and Xuyun Fu and Baoping Tang and Michael Pecht},
title = {Deep Residual Shrinkage Networks for Fault Diagnosis},
journal = {IEEE Transactions on Industrial Informatics},
year = {2020},
volume = {16},
number = {7},
pages = {4681-4690},
doi = {10.1109/TII.2019.2943898}
}
Academic Impact (Solo si ci wàllu xam-xam)
Paper bii, Google Scholar woné na ne ñu bari jëfandikoo nañu ko (lu ëpp 1400 citations).
Su ñu xoolee statistics yi (donte matuñu), Deep Residual Shrinkage Network (DRSN) jëfandikoo nañu ko walla soppi nañu ko ngir jëfandikoo ko ci lu ëpp 1000 publications/studies ci wàll yu bari, naka mechanical engineering, electrical power, vision, healthcare, speech, text, radar, ak remote sensing.