Deep Residual Shrinkage Network yɛle Deep Residual Network i ɲrun cɛcɛ kun. Sɛ e ka kan’n, Deep Residual Shrinkage Network’n, ɔ fa Deep Residual Network, Attention mechanisms, ɔ nin Soft thresholding functions be bo nun yɛ ɔ yo i junman’n niɔn.
Maan e nian kɛ Deep Residual Shrinkage Network’n ɔ yo di junman’n. Klicye, Network’n fa Attention mechanisms be sie i nzɔliɛ features nga be ti-man cinnjin’n. I sin’n, Network’n fa Soft thresholding functions be yo features nga be ti-man cinnjin’n be zero. Sanngɛ, kɛ ɔ ko yo features nga be ti cinnjin’n, ɔ sie be, ɔ fa-man be yo-man zero. Cɛcɛ nga ti, Deep neural network’n i wun miɛn. Cɛcɛ sɔ’n uka Network’n maan ɔ kpa Features nga be ti kpa’n, Signals nga be le Noise’n be nun.
1. Ninnge nga ti yɛ be suili su’n (Research Motivation)
Klicye, kɛ Algorithm’n w’a fa Samples’n be sie be bui’n, Noise’n o lɛ titi, e kwla wan-man. Noise sɔ’n i ɲrun wie yɛle Gaussian noise, Pink noise, ɔ nin Laplacian noise. I kpa bɔbɔ’n, Samples‘n be nun kpanngban le amaneɛ mɔ be ti-man Classification task nga e su di’n i liɛ. E kwla se kɛ amaneɛ sɔ mun be ti Noise. Noise sɔ’n ti, Classification‘n i bo kwla gui i ase. (Soft thresholding ti ninnge cinnjin kpa Signal denoising algorithms kpanngban be nun.)
Maan e fa ninnge kun e nian. Sɛ sran nɲɔn be su koko yalɛ atin nuan lɔ. Kɛ be su kan ndɛ’n, lɔtɔ’n i Horn ɔ nin i Wheels be di, maan Audio‘n fa be ngba. Sɛ e waan e yo Speech recognition Signals sɔ’n su. Waa-waa sɔ’n ɔ́ yó maan junman’n i bo su gua-man kpa. Sɛ e nian Deep learning i atin’n su’n, ɔ fata kɛ Deep neural network‘n nunnun Features nga be ti Horn ɔ nin Wheels be liɛ’n. Sɛ ɔ nunnun be’n, be su kwla yo-man Speech recognition‘n i tɛ.
Nɲɔn su’n, Noise nga ɔ o Samples’n be nun’n, be nuan cɛ-man. Kɛ e nian Dataset kunngba’n bɔbɔ’n, ɔ ti sɔ. (I sɔ’n ti kɛ Attention mechanisms sa. Maan e fa Image dataset e nian. Target object‘n, i ɲrun kwla kaci Images‘n be nun. Attention mechanisms kwla nian Target object‘n i ɲrun zɛɛ Image kun nun.)
Maan e fa e ɲin e sie i Classifier kun mɔ be fa kpa awo nin alua be nun’n su. Maan e se kɛ e le Images nnun (5) mɔ be klɛli be su “Alua” (dog). Image 1 kwla le alua nin Mouse (mɔɔ). Image 2 kwla le alua nin Goose. Image 3 kwla le alua nin Chicken (ako). Image 4 kwla le alua nin Donkey (sofietɛ). Image 5 kwla le alua nin Duck (dabu). Kɛ bé kile Classifier‘n i like’n, ninnge sɔ mun, mɔ be ti-man alua’n, bé yó Interference. Ninnge sɔ mun yɛle Mice, Geese, Chickens, Donkeys, ɔ nin Ducks. Interference sɔ’n ti, Classification accuracy‘n kɔ i bo. Sɛ e kwla si ninnge sɔ mun. Yɛ e nunnun Features nga be ti ninnge sɔ mun be liɛ’n. I sɔ’n ti, Classifier‘n, kɛ ɔ́ kpá awo nin alua nun’n, ɔ́ yó kpa tra laa’n.
2. Soft Thresholding
Soft thresholding ti like cinnjin kpa Signal denoising algorithms kpanngban be nun. Sɛ Features’n be absolute values’n be ka Threshold’n i bo’n, Algorithm’n nunnun Features sɔ mun. Sɛ Features’n be absolute values’n be tra Threshold’n, Algorithm’n “Shrinks” (ɔ ka be bo) Features sɔ mun maan be ko mian zero’n wun. Sran nga be su ninnge sɔ’n su’n, be kwla fa ninnge nga be klɛli i yɛ be di Soft thresholding junman’n:
\[y = \begin{cases} x - \tau & x > \tau \\ 0 & -\tau \le x \le \tau \\ x + \tau & x < -\tau \end{cases}\]Soft thresholding i Derivative‘n yɛ:
\[\frac{\partial y}{\partial x} = \begin{cases} 1 & x > \tau \\ 0 & -\tau \le x \le \tau \\ 1 & x < -\tau \end{cases}\]Ninnge nga be klɛli i ɲrun lɛ’n kile kɛ Soft thresholding i Derivative‘n ti 1 annzɛ 0. I sɔ’n ti kɛ ReLU activation function sa. I sɔ’n ti, Soft thresholding‘n uka Deep learning algorithms‘n, maan Gradient vanishing nin Gradient exploding be su tɔ-man be su.
Kɛ e fa Soft thresholding function’n, kɛ é síe Threshold’n, ɔ fata kɛ e nian ninnge nɲɔn su. Klicye, Threshold’n, ɔ fata kɛ ɔ yo nɔnbu m’ɔ ti Positive. Nɲɔn su’n, Threshold’n su kwla tra-man Input signal’n i dandan’n. Sɛ amun a yo-man sɔ’n, Output’n kwlaa yó Zero.
Asa ekun, ɔ ti kpa kɛ Threshold’n nian ninnge nsan su’n su. Sample kun bɔ Sample kun, ɔ fata kɛ ɔ nya i liɛ Threshold, kɛ ɔ ko nian Noise nga o Sample sɔ’n nun’n su.
Ninnge nga ti yɛ ɔ ti sɔ’n, yɛle kɛ Noise‘n i nuan cɛ-man Samples‘n be nun. Maan e se kɛ, Dataset kunngba’n nun, Sample A kwla le Noise kaan sa, yɛ Sample B le Noise kpanngban. I sɔ’n ti, kɛ é yó Soft thresholding‘n, ɔ fata kɛ Sample A fa Threshold kaan, yɛ Sample B fa Threshold dandan. I ndɛ’n i ɲrun lɔ’n, Deep neural networks‘n nun’n, Features ɔ nin Thresholds sɔ mun be wun weinwein-man kpa. Sanngɛ, i bo ninnge’n te yo kunngba. Yɛle kɛ, Sample kun bɔ Sample kun, ɔ fata kɛ ɔ nya i liɛ Threshold. Noise nga ɔ o nun’n yɛ ɔ kile Threshold sɔ’n niɔn.
3. Attention Mechanism
Sran nga be su ninnge sɔ’n su’n, be wun Computer vision i Attention mechanisms‘n i wlɛ ndɛndɛ. Ninnge nga e fa nian like’n, ɔ nian lika’n wunmuan’n nun ndɛndɛ fa kpa like nga e kunndɛ’n. I sin’n, ɔ fa i ɲin’n sie i Target object‘n su. I sɔ’n ti, e wun ninnge sɔ’n i kpa. Kɛ ɔ yo sɔ’n, ninnge nga be ti-man cinnjin’n, ɔ bu-man be akunndan.
Squeeze-and-Excitation Network (SENet) ti Deep learning i cɛcɛ uflɛ m’ɔ fa Attention mechanisms di junman. Samples‘n be nun’n, Feature channels‘n be ngba be di-man Classification task‘n i junman kunngba. SENet fa Sub-network kaan sa fa Learn a set of weights. I sin’n, SENet fa Weights sɔ mun be bu Features nga be o Channels‘n be su’n. Junman sɔ’n kaci Features‘n be dandan’n Channel kun bɔ Channel kun su. E kwla se kɛ i sɔ’n yɛle Apply weighting to each feature channel.
Atin sɔ’n su’n, Sample kun bɔ Sample kun le i liɛ Weights. Yɛle kɛ, Sample nɲɔn be Weights‘n ti-man kun. SENet nun’n, atin nga be fa ɲan Weights‘n yɛ: “Global Pooling → Fully Connected Layer → ReLU Function → Fully Connected Layer → Sigmoid Function”.
4. Soft Thresholding with Deep Attention Mechanism
Deep Residual Shrinkage Network‘n fa SENet sub-network i Structure‘n. Network‘n fa Structure sɔ’n di Soft thresholding junman Deep attention mechanism i bo. Sub-network‘n (mɔ be fa kplɛ bo i nzɔliɛ’n) Learn a set of thresholds. I sin’n, Network‘n fa Thresholds sɔ mun di Soft thresholding junman Feature channel kun bɔ Feature channel kun su.
Sub-network sɔ’n nun’n, System‘n bu Input feature map‘n i Features‘n kwlaa be Absolute values. I sin’n, System‘n yo Global average pooling yɛ ɔ bu i nuan, maan ɔ ɲan Feature kun, mɔ e flɛ i A. Atin uflɛ’n su’n, System‘n fa Feature map‘n sie i Fully connected network kaan kun nun, kɛ ɔ ko wie Global average pooling‘n. Fully connected network sɔ’n fa Sigmoid function‘n yo i Layer kasiɛn’n. Function sɔ’n yo maan Output‘n ka 0 nin 1 be afiɛn. I sɔ’n man e nɔnbu kun, mɔ e flɛ i α. E kwla se kɛ Threshold kasiɛn’n yɛle α × A. I sɔ’n ti, Threshold‘n yɛle nɔnbu nɲɔn be ba. Nɔnbu kun o 0 nin 1 be afiɛn. Nɔnbu kun’n yɛle Feature map‘n i Absolute values‘n be nuan nɔnbu’n. Cɛcɛ sɔ’n ti, Threshold’n ti Positive titi. Asa, cɛcɛ sɔ’n ti, Threshold’n su tra-man nuan.
Asa ekun, Samples’n be nuan cɛ-man, i sɔ’n ti Thresholds’n be nuan cɛ-man. I sɔ’n ti, e kwla se kɛ cɛcɛ sɔ’n ti Attention mechanism kɛ ɔ ti i liɛ ngunmin sa. Mechanism sɔ’n wun Features nga be ti-man cinnjin Task’n i liɛ’n. Mechanism sɔ’n fa Convolutional layers nɲɔn fa kaci Features sɔ mun maan be mian 0 wun. I sin’n, Mechanism sɔ’n fa Soft thresholding fa yo Features sɔ mun 0. Annzɛ, Mechanism sɔ’n wun Features nga be ti cinnjin Task’n i liɛ’n. Mechanism sɔ’n fa Convolutional layers nɲɔn fa kaci Features sɔ mun maan be mian-man 0 wun. Kɛ ɔ yo sɔ’n, Mechanism sɔ’n sie Features sɔ mun, ɔ nunnun-man be.
Kɛ é kɔ́ i bue’n, e Stack many basic modules. E fa Convolutional layers, Batch normalization, Activation functions, Global average pooling, ɔ nin Fully connected output layers be wla nun. Ninnge sɔ mun yɛ be kaci Deep Residual Shrinkage Network‘n niɔn.
I wun desin’n (Diagram) nun’n, amun kwla wun atin’n: E le Identity path mɔ e si i laa. Yɛ e le Weighting mɔ e fa Soft thresholding di junman’n.
5. Kɛ ɔ kwla di junman lika kwlaa nun’n (Generalization Capability)
Deep Residual Shrinkage Network ti Method mɔ e kwla fa Learn Features lika kwlaa nun. Ninnge nga ti yɛ ɔ ti sɔ’n, yɛle kɛ Samples‘n le Noise titi Feature learning tasks kpanngban nun. Samples‘n le amaneɛ mɔ be ti-man cinnjin. Noise sɔ’n nin amaneɛ sɔ’n be kwla yo maan Feature learning‘n i bo gua-man kpa. Maan e nian:
Maan e fa Image classification e nian. Image kun kwla le ninnge kpanngban uflɛ i nun. E kwla se kɛ ninnge sɔ mun ti “Noise”. Deep Residual Shrinkage Network‘n kwla fa Attention mechanism‘n di junman. Network‘n wun “Noise” sɔ’n. I sin’n, Network‘n fa Soft thresholding fa yo Features nga be ti “Noise” sɔ’n be liɛ’n, maan be kaci zero. I sɔ’n ti, Image classification accuracy‘n kwla yo kpa tra laa’n.
Maan e fa Speech recognition e nian. I kpa bɔbɔ’n, lika nga wɛlɛ o lɔ’n, kɛ lɔtɔ’n atin nuan lɔ annzɛ izini nun lɔ sa. Deep Residual Shrinkage Network‘n kwla yo maan Speech recognition accuracy‘n yo kpa. Annzɛ kɛ, Network‘n kile e atin. Atin sɔ’n kwla yo maan Speech recognition accuracy‘n yo kpa.
Ninnge nga be niannin su be klɛli’n (Reference)
Minghang Zhao, Shisheng Zhong, Xuyun Fu, Baoping Tang, Michael Pecht, Deep residual shrinkage networks for fault diagnosis, IEEE Transactions on Industrial Informatics, 2020, 16(7): 4681-4690.
https://ieeexplore.ieee.org/document/8850096
BibTeX
@article{Zhao2020,
author = {Minghang Zhao and Shisheng Zhong and Xuyun Fu and Baoping Tang and Michael Pecht},
title = {Deep Residual Shrinkage Networks for Fault Diagnosis},
journal = {IEEE Transactions on Industrial Informatics},
year = {2020},
volume = {16},
number = {7},
pages = {4681-4690},
doi = {10.1109/TII.2019.2943898}
}
Kɛ ɔ yoli sran mun i kpa’n (Academic Impact)
Be fa Paper nga di junman Google Scholar su tra kpɛ 1400.
Sɛ e nian’n, sran nga be su ninnge sɔ’n su’n, be fa Deep Residual Shrinkage Network (DRSN)‘n di junman Publications/Studies mɔ be tra 1000 be nun. Be fa di junman Fields kpanngban nun. Fields sɔ mun yɛle Mechanical engineering, Electrical power, Vision, Healthcare, Speech, Text, Radar, ɔ nin Remote sensing.