Deep Residual Shrinkage Network kun fooyya’iinsa Deep Residual Network ti. Bu’uura irratti, Deep Residual Shrinkage Network inni kun Deep Residual Network, Attention mechanisms, fi Soft thresholding functions walitti makiinsaudha.
Mee maaliif akka Deep Residual Shrinkage Network kun hojjetu haa ilaallu. Tokkoffaa, network-ichi Attention mechanisms fayyadamuun features hin barbaachisne ykn “unimportant features” adda baasa. Itti aansuudhaan, network-ichi Soft thresholding functions fayyadamuun features hin barbaachisne kanniin gara zeerootti jijjiira. Faallaa kanaatiin, network-ichi features barbaachisoo ta’an adda baasuun isaan tiksa. Adeemsi kun dandeettii Deep Neural Network cimsuudha. Akkasumas, adeemsi kun network-ichi signals noise qaban keessaa features faayidaa qaban akka inni baasu gargaara.
1. Research Motivation
Tokkoffaa, yeroo algorithm-n tokko samples gita adda addaatti qoodu (classify godhu), noise ykn jeeqamsi hin oolu. Fakkeenyaaf noise akkanaa keessaa Gaussian noise, Pink noise, fi Laplacian noise kanneen jedhaman eeruun ni danda’ama. Bal’inaan yoo ilaalle, samples yeroo baay’ee odeeffannoo hojii classification ammaatiif hin barbaachisne of keessaa qabu. Odeeffannoo hin barbaachisne kana akka noise-tti fudhachuu dandeenya. Noise kun immoo gahumsa classification gadi buusuu danda’a. (Soft thresholding kun tarkaanfii murteessaa algorithms signal denoising hedduu keessattiudha).
Mee fakkeenyaaf, haasaa karaa cinaatti taasifamu haa fudhannu. Sagaleen audio sanaa sagalee konkolaataa fi sagalee gommaa of keessaa qabaachuu danda’a. Nuti signals kana irratti speech recognition raawwachuu barbaanna ta’a. Sagaleewwan duubaan dhufan (background sounds) kunniin bu’aa isaa irratti dhiibbaa geessisuun isaanii hin oolu. Ilaalcha Deep Learning irraa yoo ilaalle, Deep Neural Network-ichi features sagalee konkolaataa fi gommaa sanaan walqabatan Identity path irraa balleessuu ykn eliminate gochuu qaba. Kana gochuun features sunniin bu’aa speech recognition akka hin miine ittisa.
Lammaffaa, hamma noise samples gidduutti argamu yeroo baay’ee garaagarummaa qaba. Garaagarummaan kun dataset tokkicha keessattis ni uumama. (Garaagarummaan kun waan Attention mechanisms waliin wal fakkaatu qaba. Mee fakkeenyaaf dataset image tokko haa fudhannu. Bakki target object-n sun itti argamu image adda addaa keessatti adda adda ta’uu danda’a. Attention mechanisms sun bakka target object-n sun image tokkoon tokkoon isaa keessatti argamu irratti xiyyeeffachuu danda’u).
Akka fakkeenyaatti, mee classifier pussaa fi saree adda baasu (cat-and-dog classifier) images “saree” jedhamanii moggaafaman shan fayyadamuun train goona haa jennu. Image 1ffaan saree fi hantuuta qabaachuu danda’a. Image 2ffaan saree fi daakkiyyee (goose) qabaachuu danda’a. Image 3ffaan saree fi handaaqqoo qabaachuu danda’a. Image 4ffaan saree fi harree qabaachuu danda’a. Image 5ffaan immoo saree fi daakiyyee bishaan irraa (duck) qabaachuu danda’a. Yeroo training, wantoonni hin barbaachisne kunniin classifier sana jeeqan. Wantoonni kunniin hantuuta, daakkiyyee, handaaqqoo, harree fi daakiyyee bishaan irraa fa’a. Jeeqamsi kun immoo sirrummaa classification (classification accuracy) gadi buusa. Osoo wantoota hin barbaachisne kana adda baasuu dandeenye, features wantoota kanaan walqabatan sana balleessuu dandeenya. Karaa kanaan, sirrummaa cat-and-dog classifier sanaa fooyyessuu dandeenya.
2. Soft Thresholding
Soft thresholding kun tarkaanfii ijoo algorithms signal denoising hedduu keessattiudha. Algorithm-n sun yoo gatiin absolute value features sanaa threshold murtaa’e tokkoo gadi ta’e, features sana ni balleessa. Yoo gatiin absolute value features sanaa threshold sana caale immoo, algorithm-n sun features sana gara zeerootti akka siqu (shirink) taasisa. Researchers formula armaan gadii fayyadamuun Soft thresholding hojiirra oolchuu danda’u:
\[y = \begin{cases} x - \tau & x > \tau \\ 0 & -\tau \le x \le \tau \\ x + \tau & x < -\tau \end{cases}\]Derivative output Soft thresholding input isaa wajjin wal bira qabamee yoo ilaalamu:
\[\frac{\partial y}{\partial x} = \begin{cases} 1 & x > \tau \\ 0 & -\tau \le x \le \tau \\ 1 & x < -\tau \end{cases}\]Formula-n armaan olitti kenname kun kan agarsiisu, derivative Soft thresholding yeroo hunda 1 ykn 0 ta’uu isaati. Amalli kun amala activation function ReLU jedhamu waliin tokko. Kanaafuu, Soft thresholding algorithms Deep Learning keessatti rakkoolee akka Gradient vanishing fi Gradient exploding hir’isuu danda’a.
Function Soft thresholding keessatti, haalli threshold itti murtaa’u ulaagaalee lama guutuu qaba. Tokkoffaa, threshold sun lakkoofsa positive ta’uu qaba. Lammaffaa, threshold sun gatii guddaa input signal sanaa caaluu hin qabu. Yoo akkas ta’e, output-n isaa guutummaatti zeeroo ta’a.
Dabalataan, threshold sun ulaagaa sadaffaas osoo guutee filatamaadha. Sample tokkoon tokkoon isaa hamma noise sample sana keessa jiru irratti hundaa’uun, threshold of danda’e (independent threshold) qabaachuu qaba.
Sababni isaas, hamma noise samples gidduutti argamu yeroo baay’ee garaagarummaa waan qabuufi. Fakkeenyaaf, Sample A noise xiqqaa qabaachuu danda’a, Sample B immoo dataset tokkicha keessatti noise baay’ee qabaachuu danda’a. Haala kana keessatti, yeroo Soft thresholding hojjetamu, Sample A threshold xiqqaa fayyadamuu qaba. Sample B immoo threshold guddaa fayyadamuu qaba. Deep Neural Networks keessatti features fi thresholds kunniin hiika fiizikaalaa ifa ta’e dhabuu danda’u. Haa ta’u malee, logic bu’uuraa isaa jijjiirama hin qabu. Kana jechuun, sample tokkoon tokkoon isaa threshold of danda’e qabaachuu qaba. Hammi noise sample sana keessa jiru threshold kana murteessa.
3. Attention Mechanism
Researchers damee Computer Vision keessa jiran Attention mechanisms kana salphumatti hubachuu danda’u. Sirni argaa bineensotaa (visual systems) naannoo hunda saffisaan sakatta’uun targets adda baasuu danda’a. Sana booda, visual systems sun xiyyeeffannoo ykn attention gara target object sanaatti godhu. Gochi kun system-ichi details dabalataa akka argatu gargaara. Yeroo wal fakkaatutti, system-ichi odeeffannoo hin barbaachisne akka ukkaamsu (suppress) taasisa. Waa’ee kanaa bal’inaan beekuuf, barruulee waa’ee Attention mechanisms dubbisuun ni danda’ama.
Squeeze-and-Excitation Network (SENet) mala Deep Learning haaraa kan Attention mechanisms fayyadamuudha. Samples adda addaa keessatti, feature channels garagaraa hojii classification sanaaf gumaacha adda addaa qabu. SENet sub-network xiqqoo tokko fayyadamuun Learn a set of weights (weights murtaa’an barata). Sana booda, SENet weights kanniin features channels sanneenii waliin walitti baay’isa. Operation kun hamma features channel tokkoon tokkoon isaa ni sirreessa. Nuti adeemsa kana akka Apply weighting to each feature channel (channel feature hundatti weighting fayyadamuu) jechuun hubachuu dandeenya.
Mala kana keessatti, sample hundi set of weights of danda’e qaba. Jechuunis, weights samples lama kamiyyuu adda adda jechuudha. SENet keessatti, daandiin weights ittiin argatan “Global Pooling → Fully Connected Layer → ReLU Function → Fully Connected Layer → Sigmoid Function” dha.
4. Soft Thresholding with Deep Attention Mechanism
Deep Residual Shrinkage Network caasaa sub-network SENet sana fayyadama. Network-ichi caasaa kana fayyadamuun Soft thresholding bifa Deep Attention Mechanism ta’een hojiirra oolcha. Sub-network kun (kan saanduqa diimaa keessatti agarsiifame) Learn a set of thresholds (thresholds murtaa’an barata). Sana booda, network-ichi thresholds kanniin fayyadamuun feature channel hunda irratti Soft thresholding raawwata.
Sub-network kana keessatti, jalqaba system-ichi absolute values features input feature map irra jiran hunda shallaga. Itti aansuun, system-ichi Global Average Pooling fi averaging hojjechuun feature tokko, kan A jedhamee bakka buufame argata. Identity path irraa kan hafe, daandii isa biraa irra, system-ichi erga Global Average Pooling taasisee booda feature map sana gara fully connected network xiqqaatti galcha. Fully connected network kun Sigmoid function akka layer dhumaatti fayyadama. Function kun output isaa gidduu 0 fi 1 tti normalize godha. Adeemsi kun coefficient tokko, kan α jedhamee bakka buufame kenna. Nuti threshold dhumaa akka α × A tti ibsuu dandeenya. Kanaafuu, threshold kun bu’aa baay’isuu lakkoofsota lamaati. Lakkoofsi tokko 0 fi 1 gidduutti argama. Lakkoofsi inni biraa immoo average absolute values feature map sanaati. Malli kun threshold sun positive ta’uu isaa mirkaneessa. Malli kun threshold sun garmalee guddaa akka hin taanes ni to’ata.
Dabalataan, samples adda addaa thresholds adda addaa kennu. Kanaafuu, nuti mala kana akka Attention Mechanism addaa ta’e tokkootti hubachuu dandeenya. Mechanism kun features hojii ammaatiif hin barbaachisne adda baasa. Mechanism kun features kanniin karaa Convolutional layers lamaatiin gara gatii zeerootti dhihaatuutti jijjiira. Sana booda, mechanism kun Soft thresholding fayyadamuun features kanniin gara zeerootti geessa. Yookiin immoo, mechanism kun features hojii ammaatiif barbaachisoo ta’an adda baasa. Mechanism kun features kanniin karaa Convolutional layers lamaatiin gara gatii zeeroo irraa fagaatutti jijjiira. Dhuma irratti, mechanism kun features kanniin ni tiksa.
Xumura irratti, nuti basic modules hamma ta’e tokko wal irra tuulla (Stack many basic modules). Akkasumas Convolutional layers, Batch Normalization, Activation functions, Global Average Pooling, fi Fully Connected output layers itti daballa. Adeemsi kun Deep Residual Shrinkage Network guutuu ta’e ijaara.
5. Generalization Capability
Deep Residual Shrinkage Network kun mala waliigalaa (general method) kan feature learning ti. Sababni isaas, hojii feature learning hedduu keessatti, samples yeroo baay’ee noise of keessaa qabu. Samples odeeffannoo hin barbaachisne of keessaa qabu. Noise fi odeeffannoon hin barbaachisne kunniin bu’aa feature learning irratti dhiibbaa geessisuu danda’u. Fakkeenyaaf:
Mee Image classification haa ilaallu. Image tokko yeroo wal fakkaatutti wantoota biraa hedduu of keessaa qabaachuu danda’a. Nuti wantoota kana akka “Noise” tti hubachuu dandeenya. Deep Residual Shrinkage Network Attention mechanism fayyadamuu danda’a ta’a. Network-ichi “Noise” kana hubata. Sana booda, network-ichi Soft thresholding fayyadamuun features “Noise” kanaan walqabatan gara zeerootti geessa. Tarkaanfiin kun sirrummaa image classification fooyyessuu danda’a.
Mee Speech recognition haa ilaallu. Keessattuu, naannoo sagaleen itti baay’atu (noisy environments) kan akka haasaa karaa cinaa ykn warshaa keessaa. Deep Residual Shrinkage Network sirrummaa speech recognition fooyyessuu danda’a. Yookiin yoo xiqqaate, network-ichi methodology tokko ni dhiyeessa. Methodology kun sirrummaa speech recognition fooyyessuu kan danda’uudha.
References
Minghang Zhao, Shisheng Zhong, Xuyun Fu, Baoping Tang, Michael Pecht, Deep residual shrinkage networks for fault diagnosis, IEEE Transactions on Industrial Informatics, 2020, 16(7): 4681-4690.
https://ieeexplore.ieee.org/document/8850096
BibTeX
@article{Zhao2020,
author = {Minghang Zhao and Shisheng Zhong and Xuyun Fu and Baoping Tang and Michael Pecht},
title = {Deep Residual Shrinkage Networks for Fault Diagnosis},
journal = {IEEE Transactions on Industrial Informatics},
year = {2020},
volume = {16},
number = {7},
pages = {4681-4690},
doi = {10.1109/TII.2019.2943898}
}
Academic Impact
Waraqaan qorannoo kun Google Scholar irratti citations 1400 ol argateera.
Akka ragaa guutuu hin taane (incomplete statistics) agarsiisutti, researchers Deep Residual Shrinkage Network (DRSN) kana barruulee/qorannoo 1000 ol keessatti hojiirra oolchaniiru. Applications kunniin dameewwan bal’aa hammata. Dameewwan kunniin Mechanical engineering, Electrical power, Vision, Healthcare, Speech, Text, Radar, fi Remote sensing fa’a.