Deep Residual Shrinkage Network (DRSN) hi Deep Residual Network atanga tihchangtlung (improved) a ni. A tlangpuiin, Deep Residual Shrinkage Network hian Deep Residual Network te, attention mechanisms te, leh soft thresholding function te a integrate (fin) khawm a ni.
Deep Residual Shrinkage Network working principle hi hetiang hian kan hrethiam thei ang. A hmasain, network hian feature pawimawh lo te identify turin attention mechanisms a hmang a. Chumi hnuah, heng feature pawimawh lo te hi zero-a chantir turin soft thresholding functions a hmang leh a ni. A letling zawngin, network hian feature pawimawh te a identify a, heng feature pawimawh te hi a retain (vawng tha) thung a ni. He process hian deep neural network theihna (ability) a tichak a. Noise awmna signal atangin feature tangkai te extract turin network a pui a ni.
1. Zirchianna Chhan (Research Motivation)
A hmasain, algorithm-in sample a classify dawn hian, noise hi pumpelh theih a ni lo. Heng noise entirnate chu Gaussian noise, pink noise, leh Laplacian noise te hi an ni. A zau zawnga sawi chuan, sample te hian current classification task nena inzawmna nei lo (irrelevant) information an keng tel fo thin a. Heng irrelevant information te hi noise angin kan interpret thei a ni. He noise hian classification performance a tichhe thei a. (Soft thresholding hi signal denoising algorithm tam takah step pawimawh tak a ni.)
Entirnan, kawng sira inbiakna (conversation) lo ngaihtuah ta ila. Audio chuan car horn ri leh ke (wheel) ri te a keng tel thei a. Heng signal-ah te hian speech recognition kan ti maithei a. Background sound te chuan result a nghawng ngei ngei dawn a ni. Deep learning thlirna atang chuan, deep neural network chuan horn leh wheel ri te nena inmil feature te chu a eliminate (paih bo) tur a ni. He elimination hian speech recognition result a khawih pawi tur lakah feature te a veng a ni.
Pahnihnaah chuan, noise awm zat hi sample hrang hrangah a inang lo fo. He danglamna (variation) hi dataset pakhatah pawh a thleng thei. (He variation hian attention mechanisms nen inanna a nei a. Image dataset han entir ta ila. Target object awmna hmun chu image hrang hrangah a danglam thei a. Attention mechanisms chuan image tinah target object awmna bik lai tak chu a focus thei a ni.)
Entirnan, “dog” tia label “uicho” paruk awmna cat-and-dog classifier train tur lo ngaihtuah ila. Image 1-ah ui leh sazu (mouse) a awm maithei. Image 2-ah ui leh goose a awm maithei. Image 3-ah ui leh ar (chicken) a awm maithei. Image 4-ah ui leh sabengtung (donkey) a awm maithei. Image 5-ah ui leh varak (duck) a awm maithei. Training laiin, object tangkai lo te chuan classifier an tibuai (interfere) dawn a ni. Heng object te chu sazu, goose, ar, sabengtung, leh varak te an ni. He interference hian classification accuracy a tihniam a ni. Heng object tangkai lo te hi identify thei ta ila. Tichuan, heng object te nena inmil feature te hi kan eliminate thei ang. Hetiang hian, cat-and-dog classifier accuracy kan improve thei a ni.
2. Soft Thresholding
Soft thresholding hi signal denoising algorithm tam takah core step (step pawimawh ber) a ni. Feature te absolute value chu threshold engemaw zat aia a te zawk chuan, algorithm-in feature te a eliminate thin. Feature te absolute value chu he threshold aia a len zawk chuan, algorithm-in feature te chu zero lam hawiin a shrink (titem) thin a ni. Researcher ten soft thresholding chu he formula hmang hian an implement thei a:
\[y = \begin{cases} x - \tau & x > \tau \\ 0 & -\tau \le x \le \tau \\ x + \tau & x < -\tau \end{cases}\]Soft thresholding output derivative input laka mi chu:
\[\frac{\partial y}{\partial x} = \begin{cases} 1 & x > \tau \\ 0 & -\tau \le x \le \tau \\ 1 & x < -\tau \end{cases}\]Formula chunga mi khian a lantir chu soft thresholding derivative chu 1 emaw 0 emaw a ni tih hi a ni. He property hi ReLU activation function property nen a inang chiah a. Chuvangin, soft thresholding hian deep learning algorithm-a gradient vanishing leh gradient exploding risk a tlem thei a ni.
Soft thresholding function-ah hian, threshold set danin condition pahnih a satisfy a ngai a. Pakhatnaah, threshold chu positive number a ni tur a ni. Pahnihnaah, threshold chuan input signal value sang ber (maximum value) a khum tur a ni lo. Chutilo chu, output zawng zawng a zero vek ang.
Chung bakah chuan, threshold chuan condition pathumna satisfy thei se a duhawm hle. Sample tin hian sample-a noise awm dan a zirin threshold independent (hrang theuh) an nei tur a ni.
A chhan chu, noise awm zat hi sample hrang hrangah a inang lo fo thin. Entirnan, dataset pakhatah Sample A-in noise tlem te a nei maithei a, Sample B-in noise tam tak a nei thung maithei. Hetiang dinhmunah hian, soft thresholding tih laiin Sample A chuan threshold te zawk a hmang tur a ni a. Sample B chuan threshold lian zawk a hmang tur a ni. Deep neural network-ah chuan heng feature leh threshold te hian an physical definition chiang tak hloh mah se. Amaherawhchu, a basic logic chu a pangngai reng a ni. Sawi fiah nan, sample tinin threshold independent tak an nei tur a ni. Noise content bil tak chuan he threshold hi a determine thin a ni.
3. Attention Mechanism
Researcher ten computer vision field-a attention mechanisms hi awlsam takin an hrethiam thei ang. Rannung (animals) visual system te chuan area zawng zawng rang taka scan-in target te an distinguish thei a. Chumi hnuah, visual system te chuan target object chungah attention an focus thin a. He action hian system te chu detail tam zawk extract turin a pui a. Chutih rualin, system ten information tangkai lo te an suppress (ti tlem) bawk a ni. A chipchiar nan, khawngaihin attention mechanisms chungchang literature en rawh u.
Squeeze-and-Excitation Network (SENet) hi attention mechanisms hmang deep learning method thar lam tak a ni. Sample hrang hrangah, feature channel hrang hrangte chuan classification task atan contribution hrang hrang an nei a. SENet chuan weight set (weight rual) khat nei turin sub-network te tak te a hmang a. Tichuan, SENet chuan heng weight te hi channel tin feature te nen a multiply (pun) a. He operation hian channel tina feature te magnitude (len lam) a adjust a ni. He process hi feature channel hrang hrangah attention level hrang hrang apply angah kan ngai thei a ni.
He approach-ah hian, sample tinin weight set independent tak an nei a. Sawi fiah nan, sample eng pawh pahnih weight te chu an inang lo a ni. SENet-ah chuan, weight lakna path (kaltlangna) chu “Global Pooling → Fully Connected Layer → ReLU Function → Fully Connected Layer → Sigmoid Function” a ni.
4. Deep Attention Mechanism hmanga Soft Thresholding
Deep Residual Shrinkage Network hian SENet sub-network structure a hmang a. Network chuan deep attention mechanism hnuaia soft thresholding implement nan he structure hi a hmang a ni. Sub-network (box sen chhunga mi) chuan threshold set khat a learn (zir) a. Tichuan, network chuan heng threshold te hmang hian feature channel tinah soft thresholding a apply a ni.
He sub-network ah hian, system chuan input feature map-a feature zawng zawng absolute value a calculate hmasa phawt a. Tichuan, system chuan global average pooling leh averaging a ti a, feature pakhat, A tia chhinchhiah hi a nei a. Path dangah chuan, global average pooling hnuah system chuan feature map chu fully connected network te tak te ah a input a. He fully connected network hian final layer angin Sigmoid function a hmang a. He function hian output chu 0 leh 1 inkarah a normalize a. He process hian coefficient pakhat, α tia chhinchhiah, a siam chhuak a ni. Final threshold chu α × A tiin kan express thei ang. Chuvangin, threshold chu number pahnih puntir chhuah (product) a ni. Number pakhat chu 0 leh 1 inkar a ni a. Number dang chu feature map absolute value te average a ni. He method hian threshold chu positive a ni tih a ensure a. He method hian threshold chu a lian lutuk lo tih a ensure bawk a ni.
Bakah, sample hrang hrangin threshold hrang hrang an siam chhuak a. Chuvangin, he method hi attention mechanism special tak angin kan interpret thei a ni. Mechanism chuan current task nena inzawmna nei lo feature te a identify a. Mechanism chuan convolutional layer pahnih kaltlangin heng feature te hi 0 hnaih value-ah a transform a. Tichuan, mechanism chuan soft thresholding hmangin heng feature te hi zero-ah a dah a ni. A lehlamah chuan, mechanism chuan current task nena inzawmna nei feature te a identify a. Mechanism chuan convolutional layer pahnih kaltlangin heng feature te hi 0 hlat tak value-ah a transform a. A tawpah, mechanism chuan heng feature te hi a preserve (vawng tha) a ni.
A tawpah, basic module engemaw zat kan stack (chhuah chhawng) a. Convolutional layer te, batch normalization te, activation function te, global average pooling te, leh fully connected output layer te pawh kan telh bawk a. He process hian Deep Residual Shrinkage Network pum pui a construct a ni.
5. Hman Tlakna Huap Zau (Generalization Capability)
Deep Residual Shrinkage Network hi feature learning tana method general (huap zau) tak a ni. A chhan chu, feature learning task tam takah sample ten noise an keng tel fo vang a ni. Sample ten irrelevant information an keng tel bawk a. Heng noise leh irrelevant information te hian feature learning performance a nghawng thei a ni. Entirnan:
Image classification lo ngaihtuah ila. Image pakhat hian object dang tam tak a keng tel kawp maithei a. Heng object te hi “noise” angin kan hrethiam thei a. Deep Residual Shrinkage Network hian attention mechanism hi a utilize thei maithei a ni. Network chuan he “noise” hi a notice a. Tichuan, network chuan soft thresholding hmangin he “noise” nena inmil feature te hi zero-ah a set a. He action hian image classification accuracy a improve thei a ni.
Speech recognition lo ngaihtuah ila. A bik takin, kawng sir emaw factory workshop chhung ang chi environment bengchheng (noisy) takah. Deep Residual Shrinkage Network hian speech recognition accuracy a improve thei a. Emaw a tlem berah pawh, network hian methodology (tihdan) a offer a ni. He methodology hian speech recognition accuracy a improve theihna a nei a ni.
Lehkhabu Rawn (Reference)
Minghang Zhao, Shisheng Zhong, Xuyun Fu, Baoping Tang, Michael Pecht, Deep residual shrinkage networks for fault diagnosis, IEEE Transactions on Industrial Informatics, 2020, 16(7): 4681-4690.
https://ieeexplore.ieee.org/document/8850096
BibTeX
@article{Zhao2020,
author = {Minghang Zhao and Shisheng Zhong and Xuyun Fu and Baoping Tang and Michael Pecht},
title = {Deep Residual Shrinkage Networks for Fault Diagnosis},
journal = {IEEE Transactions on Industrial Informatics},
year = {2020},
volume = {16},
number = {7},
pages = {4681-4690},
doi = {10.1109/TII.2019.2943898}
}
Academic Impact
He paper hian Google Scholar-ah citation 1400 chuang a hmu tawh a.
Statistics kimchang lo mah se, researcher ten Deep Residual Shrinkage Network (DRSN) hi publication/zirchianna 1000 chuangah an apply tawh a. Heng application te hian field zau tak a huam a. Heng field te chu mechanical engineering, electrical power, vision, healthcare, speech, text, radar, leh remote sensing te an ni.