Deep Residual Shrinkage Network ɛ version mi cï rialikä kɛ Deep Residual Network. Rɛy lät dɛ, Deep Residual Shrinkage Network ɛ mat Deep Residual Network, Attention mechanisms, kɛnɛ Soft thresholding functions kɛl.
Kɔn bɛ ŋa̱c lät kɛ Deep Residual Shrinkage Network rɛy dhöl ɛmɛ. Kɛ nhiam, network ɛ la luak kɛ Attention mechanisms kɛ kui ŋa̱c features tin thil raar (unimportant features). Kä, network ɛ la luak kɛ Soft thresholding functions bi features tin thil raar tiit a ben pɔl a thil (set to zero). Kɛ dhöl mɔ, network ɛ la nɛn features tin kɔn kɔr (important features) kä gɔa kɛ ni kap. Process ɛmɛ ɛ jak Deep Neural Network a bum. Process ɛmɛ ɛ luak network kɛ ŋa̱c features tin gɔa rɛy Signals tin tɔ kɛ Noise.
1. Research Motivation
Kɛ nhiam, të ɣöö algorithm la lät kɛ classify samples, Noise ɛ tɔ thïn a thil rɔ. Noise ɛmɛ ɛ la mat Gaussian noise, Pink noise, kɛnɛ Laplacian noise. Rɛy dhöl mi diit, samples ti ŋuan cikɛ tɔ kɛ information mi thil raar kɛ task in. Kɔn bɛ information ɛmɔ cɔl ɛ Noise. Noise ɛmɛ ɛ jak classification performance a bɛ wä piny. ( Soft thresholding ɛ step mi diit rɛy signal denoising algorithms ti ŋuan.)
Cet kɛ, bi kɔn nɛn conversation rɛy dhöl. Audio ɛmɔ bɛ tɔ kɛ thol car kɛnɛ thol wheels. Kɔn bɛ Speech Recognition lät kɛ signals ti. Thol background ti bɛ results jak a jiäk. Kɛ nɛn Deep Learning, Deep Neural Network bɛ features ti thol car kɛnɛ wheels kwan kɔ. Lät ɛmɛ ɛ jak features ti a bɛ thil te kɛ Speech Recognition results.
Kɛ rɛw, nɔŋ dɛ Noise ɛ la gɔ̱g rɛy samples. Gɔ̱g ɛmɛ ɛ la tɔ thïn rɛy dataset kɛl. (Gɔ̱g ɛmɛ ɛ ce̱tkɛ Attention mechanisms. Cet kɛ Image dataset. Të nɔŋ target object rɛy images bɛ gɔ̱g. Attention mechanisms bɛ nɛn të nɔŋ target object rɛy image kɛl.)
Cet kɛ, bi kɔn cat-and-dog classifier lät kɛ images da̱ŋ dhieec tin la “dog.” Image 1 bɛ tɔ kɛ dog kɛnɛ mouse. Image 2 bɛ tɔ kɛ dog kɛnɛ goose. Image 3 bɛ tɔ kɛ dog kɛnɛ chicken. Image 4 bɛ tɔ kɛ dog kɛnɛ donkey. Image 5 bɛ tɔ kɛ dog kɛnɛ duck. Rɛy training, objects ti thil raar bɛ classifier yöŋ. Objects ti ɛ mouse, goose, chicken, donkey, kɛnɛ duck. Yöŋ ɛmɛ ɛ jak classification accuracy a wä piny. Mi kɔn bɛ objects ti ŋa̱c. Kä, kɔn bɛ features tin cɔl objects ti kwan kɔ. Kɛ dhöl ɛmɛ, kɔn bɛ cat-and-dog classifier jak a bɛ lät a gɔa.
2. Soft Thresholding
Soft thresholding ɛ step mi diit rɛy signal denoising algorithms ti ŋuan. Algorithm ɛ bɛ features kwan kɔ mi absolute values dɛ features ti thiin kɛ threshold. Algorithm ɛ bɛ features riet kɛ zero mi absolute values dɛ features ti diit kɛ threshold. Ji Research cikɛ Soft thresholding lät kɛ formula ɛmɛ:
\[y = \begin{cases} x - \tau & x > \tau \\ 0 & -\tau \le x \le \tau \\ x + \tau & x < -\tau \end{cases}\]Derivative kɛ Soft thresholding output kɛ kui input ɛ:
\[\frac{\partial y}{\partial x} = \begin{cases} 1 & x > \tau \\ 0 & -\tau \le x \le \tau \\ 1 & x < -\tau \end{cases}\]Formula ɛmɔ ɛ nyoth ɛn derivative kɛ Soft thresholding ɛ 1 wala 0. Lät ɛmɛ ɛ ce̱tkɛ lät ReLU activation function. Kɛ kui ɛmɔ, Soft thresholding ɛ bɛ risk kɛ Gradient vanishing kɛnɛ Gradient exploding jak a wä piny rɛy Deep Learning algorithms.
Rɛy Soft thresholding function, threshold ɛmɔ bɛ luak kɛ conditions da̱ŋ rɛw. Kɛ nhiam, threshold bɛ tɔ positive number. Kɛ rɛw, threshold ɛ cï bɛ diit kɛ maximum value kɛ input signal. Mi cɛ diit, output bɛ tɔ zero kɛr.
Kä, threshold ɛ bɛ gɔa mi cɛ luak kɛ condition mɔk diɔk. Sample kɛl bɛ tɔ kɛ threshold dɛ kɛ rɔ kɛ kui Noise content kɛ sample ɛmɔ.
Kɛ ɣöö Noise content ɛ la gɔ̱g rɛy samples. Cet kɛ, Sample A bɛ tɔ kɛ Noise mi thiin kä Sample B bɛ tɔ kɛ Noise mi diit rɛy dataset kɛl. Të ɛmɔ, Sample A bɛ luak kɛ threshold mi thiin rɛy Soft thresholding. Sample B bɛ luak kɛ threshold mi diit. Features ti kɛnɛ thresholds ti thil physical definitions ti cï ŋa̱c rɛy Deep Neural Networks. Duundɛ, logic ɛmɔ ɛ tɔ thïn. Kɛ kui ɛmɔ, sample kɛl bɛ tɔ kɛ independent threshold. Noise content ɛmɔ ɛ jen bɛ threshold ŋa̱c.
3. Attention Mechanism
Ji Research bɛ Attention mechanisms ŋa̱c a gɔa rɛy Computer Vision. Visual systems kɛ lei cikɛ targets ŋa̱c kɛ scanning area dial a pio̱l. Kä, visual systems cikɛ focus Attention kɛ target object. Lät ɛmɛ ɛ jak systems a bɛ details ti ŋuan kap. Kɛ thaar kɛl, systems cikɛ information mi thil raar kwan kɔ. Kɛ kui details, nɛn literature kɛ kui Attention mechanisms.
Squeeze-and-Excitation Network (SENet) ɛ Deep Learning method mi thil mi luak kɛ Attention mechanisms. Rɛy samples ti gɔ̱g, Feature channels ti gɔ̱g cikɛ luak a gɔ̱g kɛ kui classification task. SENet ɛ luak kɛ sub-network mi thiin bi Learn a set of weights (weights ti ŋuan). Kä, SENet ɛ multiplies weights ti kɛ features kɛ channels ti. Operation ɛmɛ ɛ feature magnitude adjust rɛy channel kɛl. Kɔn bɛ lät ɛmɛ nɛn cet kɛ Apply weighting to each feature channel (levels ti gɔ̱g kɛ Attention kɛ kui feature channels ti gɔ̱g).
Kɛ dhöl ɛmɛ, sample kɛl ɛ tɔ kɛ independent set of weights. Kɛ kui ɛmɔ, weights kɛ kui arbitrary samples da̱ŋ rɛw cikɛ gɔ̱g. Rɛy SENet, dhöl mi kɔn weights yuh thïn ɛ “Global Pooling → Fully Connected Layer → ReLU Function → Fully Connected Layer → Sigmoid Function.”
4. Soft Thresholding with Deep Attention Mechanism
Deep Residual Shrinkage Network ɛ luak kɛ structure kɛ SENet sub-network. Network ɛ luak kɛ structure ɛmɛ bi Soft thresholding lät rɛy Deep Attention Mechanism. Sub-network (mi tɔ rɛy box mi bany) ɛ Learn a set of thresholds. Kä, network ɛ luak kɛ thresholds ti bi Soft thresholding lät kɛ feature channel kɛl.
Rɛy sub-network ɛmɛ, system ɛ calculate absolute values kɛ features dial rɛy input feature map. Kä, system ɛ lät kɛ Global Average Pooling kɛnɛ averaging bi feature kɛl yuh, mi cɔali A. Rɛy path mɔ, system ɛ feature map na̱ŋ rɛy small fully connected network kɛ ɣöö Global Average Pooling. Fully connected network ɛmɛ ɛ luak kɛ Sigmoid function cet kɛ layer mɔk thuk. Function ɛmɛ ɛ output normalize rɛy 0 kɛnɛ 1. Process ɛmɛ ɛ coefficient yuh, mi cɔali α. Kɔn bɛ threshold mɔk thuk nɛn cet kɛ α×A. Kɛ kui ɛmɔ, threshold ɛ product kɛ numbers da̱ŋ rɛw. Number kɛl ɛ tɔ rɛy 0 kɛnɛ 1. Number mɔ ɛ average kɛ absolute values kɛ feature map. Method ɛmɛ ɛ jak threshold a bɛ tɔ positive. Method ɛmɛ ɛ jak threshold a cï bɛ diit a lay.
Kä, samples ti gɔ̱g cikɛ thresholds ti gɔ̱g yuh. Kɛ kui ɛmɔ, kɔn bɛ method ɛmɛ nɛn cet kɛ specialized Attention mechanism. Mechanism ɛmɛ ɛ ŋa̱c features tin thil raar kɛ task in. Mechanism ɛmɛ ɛ features ti transform kɛ values tin thiɛk kɛ 0 rɛy Convolutional layers da̱ŋ rɛw. Kä, mechanism ɛ features ti set kɛ zero kɛ luak Soft thresholding. Wala, mechanism ɛ features tin kɔn kɔr ŋa̱c. Mechanism ɛ features ti transform kɛ values tin wä me̱e̱ kɛ 0 rɛy Convolutional layers da̱ŋ rɛw. Kɛ thuk, mechanism ɛ features ti kap (preserve).
Kɛ thuk, kɔn bɛ Stack many basic modules. Kɔn la mat Convolutional layers, Batch Normalization, Activation functions, Global Average Pooling, kɛnɛ Fully Connected output layers. Process ɛmɛ ɛ Deep Residual Shrinkage Network construct a gɔa.
5. Generalization Capability
Deep Residual Shrinkage Network ɛ general method kɛ kui Feature Learning. Kɛ ɣöö, rɛy Feature Learning tasks ti ŋuan, samples cikɛ tɔ kɛ Noise. Samples cikɛ tɔ kɛ information mi thil raar. Noise kɛnɛ information ɛmɔ bɛ Feature Learning performance yöŋ. Cet kɛ:
Bi kɔn nɛn Image Classification. Image kɛl bɛ tɔ kɛ objects ti ŋuan. Kɔn bɛ objects ti nɛn cet kɛ “Noise.” Deep Residual Shrinkage Network bɛ luak kɛ Attention mechanism. Network bɛ “Noise” ɛmɔ nɛn. Kä, network bɛ luak kɛ Soft thresholding bi features kɛ “Noise” ɛmɔ set kɛ zero. Lät ɛmɛ bɛ Image Classification accuracy jak a bɛ wä nhiam.
Bi kɔn nɛn Speech Recognition. Kɛ kui environments tin tɔ kɛ Noise mi diit cet kɛ conversation rɛy dhöl wala rɛy factory workshop. Deep Residual Shrinkage Network bɛ Speech Recognition accuracy jak a bɛ wä nhiam. Wala, network ɛ method mi gɔa ka̱m kɔ. Method ɛmɛ bɛ Speech Recognition accuracy jak a bɛ wä nhiam.
Reference
Minghang Zhao, Shisheng Zhong, Xuyun Fu, Baoping Tang, Michael Pecht, Deep residual shrinkage networks for fault diagnosis, IEEE Transactions on Industrial Informatics, 2020, 16(7): 4681-4690.
https://ieeexplore.ieee.org/document/8850096
BibTeX
@article{Zhao2020,
author = {Minghang Zhao and Shisheng Zhong and Xuyun Fu and Baoping Tang and Michael Pecht},
title = {Deep Residual Shrinkage Networks for Fault Diagnosis},
journal = {IEEE Transactions on Industrial Informatics},
year = {2020},
volume = {16},
number = {7},
pages = {4681-4690},
doi = {10.1109/TII.2019.2943898}
}
Academic Impact
Paper ɛmɛ cɛ citations ti diit kɛ 1,400 yuh rɛy Google Scholar.
Kɛ statistics tin cï thuk, ji Research cikɛ Deep Residual Shrinkage Network (DRSN) luak rɛy publications/studies ti diit kɛ 1,000. Lät ɛmɛ ɛ tɔ rɛy fields ti ŋuan. Fields ti ɛ mechanical engineering, electrical power, vision, healthcare, speech, text, radar, kɛnɛ remote sensing.