Deep Residual Shrinkage Network, is Deep Residual Network akke tayse version kinni. Asli wannah, Deep Residual Shrinkage Network is Deep Residual Network, Attention mechanisms, ke Soft thresholding functions itta fanah hayta.
Nan Deep Residual Shrinkage Network manna abba inta way, ta gural fahmitna. Bartillo, network is Attention mechanisms turaamah unimportant features yaaxige. Sarra, network is Soft thresholding functions turaamah ta unimportant features Zero fanah hayta. Amaymih, network is important features yableeh, ken diirisa. Ta process is Deep Neural Network qangoosa. Ta process is Signal noise luk temeete waqdi, useful features yableenim yandoowe.
1. Research Motivation
Bartillo, algorithm Samples classify abba waqdi, Noise (xexxar) kuraamah mannu. Ta Noise misal Gaussian noise, Pink noise, ke Laplacian noise kinni. Naba gural wagneek, Samples task fanah irrelevant information le yakken. Nan ta irrelevant information Noise kinni inna. Ta Noise classification performance kuraamah duddah. (Soft thresholding is signal denoising algorithms waqdi naba step kinni.)
Masalan, gita caxah nammay garayta. Audio is makiina xexxar ke makiina biil le yakke. Nan ta signals Speech Recognition abna. Background sounds is results yexxeere. Deep Learning gural wagneek, Deep Neural Network is makiina xexxar features kuraale. Ta kuraamah, Speech Recognition results meqe yakke.
Nammay haytoh, kullu Sample noise amount itta maicce. Ta Dataset inki teek, variation yane. (Ta variation is Attention mechanisms luk sissik kinni. Image dataset masalan beyna. Target object location kullu Image waqdi itta maicce. Attention mechanisms is kullu Image target object location fanah focus abba.)
Masalan, nan Cat-and-Dog classifier train abna, 5 Images “Dog” label le.
- Image 1 is Dog ke Mouse le.
- Image 2 is Dog ke Goose le.
- Image 3 is Dog ke Chicken le.
- Image 4 is Dog ke Donkey le.
- Image 5 is Dog ke Duck le.
Train abna waqdi, irrelevant objects is classifier yexxeere. Ta objects Mouse, Goose, Chicken, Donkey, ke Duck kinni. Ta interference is classification accuracy kura. Hamiin, nan ta irrelevant objects naaxige. Sarra, nan ta objects features kura. Ta gural, nan Cat-and-Dog classifier accuracy tayse yakke.
2. Soft Thresholding
Soft thresholding is signal denoising algorithms waqdi core step kinni. Features absolute values is Threshold akke qunxa teek, algorithm is ta features kura (eliminates). Features absolute values is Threshold akke naba teek, algorithm is ta features Zero fanah shrink abba. Researchers is Soft thresholding ta formula turaamah implement abba:
\[y = \begin{cases} x - \tau & x > \tau \\ 0 & -\tau \le x \le \tau \\ x + \tau & x < -\tau \end{cases}\]Soft thresholding output derivative input fanah taham kinni:
\[\frac{\partial y}{\partial x} = \begin{cases} 1 & x > \tau \\ 0 & -\tau \le x \le \tau \\ 1 & x < -\tau \end{cases}\]Daga yani formula is Soft thresholding derivative 1 hinnim 0 kinni yuybulle. Ta property is ReLU activation function property luk inki kinni. Amaymih, Soft thresholding is Deep Learning algorithms waqdi gradient vanishing ke gradient exploding risk kura.
Soft thresholding function waqdi, Threshold setting nammay shuruut le. Bartillo, Threshold is positive number yakke le. Nammay haytoh, Threshold is Input Signal maximum value akke naba yakke wuwa. Hinnim teek, output kullum Zero yakke.
Kaadu, Threshold sidoc haytoh shuruut le teek meqe. Kullu Sample isi independent Threshold yakke le, Sample noise content turaamah.
Sabab is, noise content kullu Samples waqdi itta maicce. Masalan, Sample A noise qunxa le, lakin Sample B noise naba le, inki Dataset waqdi. Ta condition waqdi, Sample A is Soft thresholding gexxa waqdi qunxa Threshold beyta. Sample B is naba Threshold beyta. Deep Neural Networks waqdi, ta features ke thresholds physical definitions kura. Lakin, basic logic inki kinni. Yaanam, kullu Sample isi independent Threshold le. Noise content ta Threshold determination abba.
3. Attention Mechanism
Researchers is Computer Vision field waqdi Attention mechanisms fahmitam duddan. Xaylo visual systems is target yaxageenim area sissik scan abba. Sarra, visual systems is target object fanah Attention focus abba. Ta action is systems details yableenim yandoowe. Waqdi inki, systems irrelevant information kura (suppress).
Squeeze-and-Excitation Network (SENet) is Deep Learning method cusub, Attention mechanisms use abba. Samples itta maicce waqdi, Feature channels classification task fanah contribution itta maicce. SENet is small sub-network use abba, Learn a set of weights abam fanah. Sarra, SENet ta weights is features channels luk multiply abba. Ta operation is kullu channel features magnitude adjust abba. Nan ta process is Apply weighting to each feature channel inna.
Ta gural, kullu Sample isi independent Weighting le. Yaanam, nammay Sample weights itta maicce. SENet waqdi, path weights geeyam fanah is: “Global Pooling → Fully Connected Layer → ReLU Function → Fully Connected Layer → Sigmoid Function.”
4. Soft Thresholding with Deep Attention Mechanism
Deep Residual Shrinkage Network is SENet sub-network structure use abba. Network is ta structure use abba Soft thresholding deep attention mechanism gural abam fanah. Sub-network (red box waqdi yuybulle) is Learn a set of thresholds abba. Sarra, network is Soft thresholding kullu feature channel fanah ta thresholds turaamah abba.
Ta sub-network waqdi, system bartillo Input feature map features absolute values hisaaba. Sarra, system Global Average Pooling abba, feature geeyam fanah, A inna. Akke path, system Feature map small Fully Connected Network fanah hayta, Global Average Pooling sarra. Ta Fully Connected Network is Sigmoid function final layer inna use abba. Ta function output 0 ke 1 fanah normalizes abba. Ta process coefficient yacee, α inna. Nan final threshold α × A inna nable. Amaymih, Threshold nammay number product kinni. Inki number 0 ke 1 fanah. Akke number Feature map absolute values average kinni. Ta method is Threshold positive yakke yandoowe. Ta method kaadu Threshold naba yakke wuwa.
Kaadu, different Samples is different Thresholds yacee. Amaymih, nan ta method specialized Attention mechanism inna fahmitna. Mechanism is unimportant features task fanah yaaxige. Mechanism ta features values 0 gubi fanah transform abba (two convolutional layers turaamah). Sarra, Mechanism ta features Zero fanah Soft thresholding turaamah hayta. Hinnim, Mechanism important features task fanah yaaxige. Mechanism ta features values 0 akke der fanah transform abba. Final, Mechanism ta features diirisa.
Final, nan Stack many basic modules abna. Nan Convolutional layers, Batch Normalization, Activation functions, Global Average Pooling, ke Fully Connected output layers osisne. Ta process is Deep Residual Shrinkage Network complete abba.
5. Generalization Capability
Deep Residual Shrinkage Network is feature learning fanah general method kinni. Sabab is, feature learning tasks mangigo waqdi, Samples noise le. Samples kaadu irrelevant information le. Ta noise ke irrelevant information is feature learning performance yexxeere. Masalan:
Image classification wagay. Image is mango objects le. Nan ta objects “Noise” inna fahmitna. Deep Residual Shrinkage Network is Attention mechanism use abam duddah. Network ta “Noise” yaaxige. Sarra, Network Soft thresholding use abba, ta “Noise” features Zero fanah hayta. Ta action is Image classification accuracy tayse abba.
Speech recognition wagay. Masalan, noisy environments, gita caxah hinnim factory workshop waqdi. Deep Residual Shrinkage Network is Speech recognition accuracy tayse abam duddah. Hinnim, Network methodology yacee. Ta methodology accuracy tayse abam duddah.
Reference
Minghang Zhao, Shisheng Zhong, Xuyun Fu, Baoping Tang, Michael Pecht, Deep residual shrinkage networks for fault diagnosis, IEEE Transactions on Industrial Informatics, 2020, 16(7): 4681-4690.
https://ieeexplore.ieee.org/document/8850096
BibTeX
@article{Zhao2020,
author = {Minghang Zhao and Shisheng Zhong and Xuyun Fu and Baoping Tang and Michael Pecht},
title = {Deep Residual Shrinkage Networks for Fault Diagnosis},
journal = {IEEE Transactions on Industrial Informatics},
year = {2020},
volume = {16},
number = {7},
pages = {4681-4690},
doi = {10.1109/TII.2019.2943898}
}
Academic Impact
Ta paper is Google Scholar waqdi 1,400 citations akke mango geyte.
Statistics wagneek, Researchers is Deep Residual Shrinkage Network (DRSN) 1,000 publications/studies akke mango waqdi use aben. Ta applications mango fields le. Ta fields Mechanical engineering, Electrical power, Vision, Healthcare, Speech, Text, Radar, ke Remote sensing kinni.