Deep Residual Shrinkage Network do Deep Residual Network reak’ mit’tan improved variant kana. Asol re, noa do Deep Residual Network, attention mechanisms, ar soft thresholding functions reak’ integration kana.
Mit’ lekate lel gale, Deep Residual Shrinkage Network reak’ working principle do nonka bujhau ganoa: noa do attention mechanisms beohar kate’ unimportant features ko chihnai (identify) nam-a ar soft thresholding functions beohar kate’ onako zero re set-a; ar ulta sed khon, important features ko chihnai nam-a ar onako retain-a (dohoya). Noa process do noise menak’ signal khon useful features extract lagit’ deep neural network reak’ ability strong-a.
1. Research Motivation
Puhil re, jkhon samples classify huiyug-a, noise — jemon Gaussian noise, pink noise, ar Laplacian noise — tahin do inevitable gea (baang sah ganoa). Barti kate’, samples re onkan information menag-a jahata do current classification task lagi’t jarur banug-a, noa ho noise mente bujhau ganoa. Noa noise do classification performance re barij’ prabhav (negative effect) e ema. (Soft thresholding do signal denoising algorithms re mit’tan key step kana.)
Jemon, hor dhare re galmarao jokhj, car horns ar chaka (wheels) reak’ sade, aḍang (audio) saw mesa godog-a. Jkhon nonkan signals re speech recognition korao huiyug-a, un jokhj result do background sounds te effect nischit gea. Deep learning perspective khon nel lekh, horns ar wheels reak’ features do deep neural network bhitri re eliminate gidig jarur kana, jemon speech recognition results re prabhav alo purau ma.
Dosar katha, mit’tan dataset re ho, noise reak’ amount do sample khon sample re juda-juda ge tahina. (Noa do attention mechanisms saw milau menag-a; jemon image dataset re, target object reak’ location do chitra (image) khon chitra re juda ge hoyoa, ar attention mechanisms do target object reak’ specific location re focus daṛeag-a.)
Udihran leka te, cat-and-dog classifier train jokhj, “dog” label menak’ 5 gotang image hatao me. Puhil image re dog ar mouse tahin daṛeag-a, dosar re dog ar goose, tesar re dog ar chicken, pun-na re dog ar donkey, ar mone-a re dog ar duck tahin daṛeag-a. Training jokhj, classifier do irrelevant objects jemon mice, geese, chickens, donkeys, ar ducks hotete disturb hoyoa, jaha karon te classification accuracy komog-a. Judi abo noako irrelevant objects — mice, geese, chickens, donkeys, ar ducks — identify kate’ onako reak’ features eliminate daṛeag-a bon, tobe cat-and-dog classifier reak’ accuracy badhao (improve) ganoa.
2. Soft Thresholding
Soft thresholding do ayma signal denoising algorithms reak’ core step kana. Noa do onkan features eliminate-a jahata reak’ absolute values do threshold khon kom gea, ar jaha features reak’ absolute values threshold khon barti gea onako do zero sed shrink-a. Noa do lata re em akan formula te implement ganoa:
\[y = \begin{cases} x - \tau & x > \tau \\ 0 & -\tau \le x \le \tau \\ x + \tau & x < -\tau \end{cases}\]Input re soft thresholding output reak’ derivative do huiyug kana:
\[\frac{\partial y}{\partial x} = \begin{cases} 1 & x > \tau \\ 0 & -\tau \le x \le \tau \\ 1 & x < -\tau \end{cases}\]Chetan re leka, soft thresholding reak’ derivative do 1 bankhan 0 hoyoa. Noa property do ReLU activation function saw saman gea. Ona te, soft thresholding do deep learning algorithms re gradient vanishing ar gradient exploding reak’ risk kom daṛeag-a.
Soft thresholding function re, threshold set jokhj baria condition purau hoyoa: Puhil, threshold do positive number hoi jarur kana; Dosar, threshold do input signal reak’ maximum value khon barti bang hoi jarur kana, bankhan output do pura zero hoyoa.
Ina sawte, threshold do tesar condition purau lekh boge hoyoa: noise content hisab te man-mi’d (each) sample reak’ apon apon independent threshold tahin jarur.
Cheda’je, noise content do samples re juda-juda tahina. Jemon, mit’tan dataset re Sample A re kom noise menag-a ar Sample B re barti noise menag-a. Nokan situation re, denoising algorithm re soft thresholding korao jokhj, Sample A lagit’ katij threshold ar Sample B lagit’ latu threshold beohar jarur kana. Deep neural networks re, judihu noako features ar thresholds do physical definitions ko ad-a, menkhan asol logic do saman gea. Soja katha te, apon apon noise content hisab te man-mi’d sample reak’ independent threshold tahin jarur kana.
3. Attention Mechanism
Computer vision field re Attention mechanisms bujhau do alga gea. Jib-jontu koak’ visual systems do target object alga te chihnai (distinguish) daṛeag-a pura area scan kate’, ina tayom target object re attention focus kate’ barti details extract-a, ar irrelevant information do suppress-a. Barti baday lagit’, attention mechanisms reak’ literature padhao pe.
Squeeze-and-Excitation Network (SENet) do attention mechanisms beohar ed’ mit’tan nawa deep learning method kana. Juda-juda samples re, classification task lagit’ different feature channels reak’ contribution do juda-juda gea. SENet do weights reak’ set nam lagit’ mit’tan katij sub-network e beohar-a ar ina tayom noako weights do respective channels reak’ features saw multiply kate’ man-mi’d channel features reak’ magnitude adjust-a. Noa process do feature channels re juda-juda level reak’ attention apply leka bujhau ganoa.
Noa approach re, man-mi’d sample reak’ apon apon independent weights reak’ set menag-a. Orthat, jahangey baria samples reak’ weights do alag gea. SENet re, weights nam reak’ specific path do “Global Pooling → Fully Connected Layer → ReLU Function → Fully Connected Layer → Sigmoid Function” kana.
4. Soft Thresholding with Deep Attention Mechanism
Deep Residual Shrinkage Network do chetan re lai akan SENet sub-network structure khon inspiration hatao akada, jemon deep attention mechanism re soft thresholding implement daṛeag-a. Sub-network (araa box bhitri re) dwara, thresholds reak’ set learn ganoa man-mi’d feature channel re soft thresholding apply lagit’.
Noa sub-network re, input feature map reak’ jyoto features reak’ absolute values calculate hoyoa. Ina tayom, global average pooling ar averaging dwara, mit’tan feature namog-a, jaha do A mente denote akana. Eta’ path re, global average pooling tayom feature map do mit’tan katij fully connected network re input hoyoa. Noa fully connected network do Sigmoid function e beohar-a final layer lekate, jemon output do 0 ar 1 tala re normalize daṛeag-a, jaha do α mente denote akana. Final threshold do α×A lekate express ganoa. Ona te, threshold do 0 ar 1 tala reak’ number ar feature map reak’ absolute values reak’ average reak’ product kana. Noa method ensure-a je threshold do positive gea ar khub latu do bang-a.
Ina sawte, different samples reak’ different thresholds hoyoa. Onate, noa do mit’tan specialized attention mechanism leka bujhau ganoa: current task lagit’ irrelevant features chihnai nam-a, baria convolutional layers beohar kate’ zero sur values re badol-a, ar soft thresholding beohar kate’ onako zero re set-a; bankhan, current task lagit’ relevant (jarur) features chihnai nam-a, baria convolutional layers beohar kate’ zero khon sangiń values re badol-a, ar onako preserve (dohoya) ka-a.
Muchad re, kichu basic modules sawte convolutional layers, batch normalization, activation functions, global average pooling, ar fully connected output layers stack kate’, pura Deep Residual Shrinkage Network do benao akana.
5. Generalization Capability
Asol re, Deep Residual Shrinkage Network do mit’tan general feature learning method kana. Cheda’je, ayma feature learning tasks re, samples re noise ar irrelevant information do menak’ gea. Noa noise ar irrelevant information do feature learning reak’ performance re prabhav e ema. Jemon:
Image classification re, judi mit’tan image re ayma eta’ objects menag-a, noako objects do “noise” mente bujhau ganoa. Deep Residual Shrinkage Network do attention mechanism beohar kate’ noa “noise” notice daṛeag-a ar soft thresholding beohar kate’ noa “noise” reak’ corresponding features do zero re set daṛeag-a, jaha karon te image classification accuracy badhao daṛeag-a.
Speech recognition re, asol kayte noisy environments jemon hor dhare re bankhan factory workshop bhitri re galmarao jokhj, Deep Residual Shrinkage Network do speech recognition accuracy badhao daṛeag-a, bankhan kom se kom mit’tan methodology offer-a jaha do speech recognition accuracy badhao daṛeag-a.
Reference
Minghang Zhao, Shisheng Zhong, Xuyun Fu, Baoping Tang, Michael Pecht, Deep residual shrinkage networks for fault diagnosis, IEEE Transactions on Industrial Informatics, 2020, 16(7): 4681-4690.
https://ieeexplore.ieee.org/document/8850096
BibTeX
@article{Zhao2020,
author = {Minghang Zhao and Shisheng Zhong and Xuyun Fu and Baoping Tang and Michael Pecht},
title = {Deep Residual Shrinkage Networks for Fault Diagnosis},
journal = {IEEE Transactions on Industrial Informatics},
year = {2020},
volume = {16},
number = {7},
pages = {4681-4690},
doi = {10.1109/TII.2019.2943898}
}
Academic Impact
Noa paper do Google Scholar re 1,400 khon barti citations nam akada.
Incomplete statistics hisab te, Deep Residual Shrinkage Network (DRSN) do 1,000 khon barti publications/studies re directly apply bankhan modify kate’ apply hui akana, jemon mechanical engineering, electrical power, vision, healthcare, speech, text, radar, ar remote sensing fields re.