Deep Residual Shrinkage Network: Mutindu ya Artificial Intelligence sambua na Highly Noisy Data

Deep Residual Shrinkage Network kele mutindu ya mpa mpe ya kuluta mbote ya Deep Residual Network. Na kutuba ya mbote, Deep Residual Shrinkage Network ke vukisa Deep Residual Network, Attention mechanisms, mpe Soft thresholding functions.

Beto lenda bakisa mutindu Deep Residual Shrinkage Network ke salaka na mutindu yai. Ya ntete, Network ke sadilaka Attention mechanisms sambua na kuzaba Features yina kele na mfunu ve. Na nima, Network ke sadilaka Soft thresholding functions sambua na kutula Features yina kele na mfunu ve na zero. Na ndambu ya nkaka, Network ke zabaka Features ya mfunu mpe yau ke bumbaka (retains) Features yina ya mfunu. Diambu yai ke kumisaka Deep Neural Network ngolo. Yau ke sadisaka Network na kubaka Features ya mbote na kati ya ba Signals yina kele na Noise.

1. Research Motivation

Ya ntete, Noise ke vandaka kaka ntangu Algorithm ke sala Classify na ba Samples. Ba mbandu ya Noise yai kele Gaussian noise, Pink noise, mpe Laplacian noise. Na kutuba ya nene, ba Samples mbala mingi ke vandaka na bansangu yina me swaswana na kisalu ya Classification. Beto lenda binga bansangu yai Noise. Noise yai lenda kitisa ngolo ya Classification. (Soft thresholding kele step ya mfunu na ba Algorithms mingi ya Signal denoising.)

Mu mbandu, yindula disolo na lweka ya nzila. Audio lenda vanda na makelele ya ba klaxon mpe ba pine ya kamio. Beto lenda sala Speech recognition na ba Signals yai. Makelele ya nima ta bebisa ba resultats. Na mutindu ya Deep Learning, Deep Neural Network fwete katula Features ya ba klaxon mpe ba pine. Diambu yai ke kangisa Features yina na kubebisa Speech recognition.

Ya zole, kiteso ya Noise ke swaswanaka na kati ya ba Samples. Kuswaswana yai ke salamaka ata na kati ya Dataset mosi. (Kuswaswana yai kele bonso Attention mechanisms. Baka Dataset ya bafoto bonso mbandu. Kisika yina kima ya beto ke sosa kele, lenda swaswana na bafoto. Attention mechanisms lenda tala kaka kisika yina kima kele na konso foto.)

Mu mbandu, beto ke longa Classifier ya pusu (cat) ti mbwa (dog) ti bafoto tanu yina beto me tula zina “mbwa”. Image 1 lenda vanda na mbwa ti mpuku. Image 2 lenda vanda na mbwa ti ngansia (goose). Image 3 lenda vanda na mbwa ti nsusu. Image 4 lenda vanda na mbwa ti mpunda (donkey). Image 5 lenda vanda na mbwa ti canard (duck). Na ntangu ya Training, bima yina kele na mfunu ve ta yangisa Classifier. Bima yai kele mpuku, ngansia, nsusu, mpunda, mpe ba canard. Mavwanga yai ke salaka nde Classification accuracy kukita. Kana beto lenda zaba bima yai ya mfunu ve. Ebuna, beto lenda katula Features na yau. Na mutindu yai, beto lenda tomisa Accuracy ya Classifier ya pusu ti mbwa.

2. Soft Thresholding

Soft thresholding kele step ya mfunu na kati ya ba Algorithms mingi ya Signal denoising. Algorithm ke katulaka Features kana absolute values ya Features kele na nsi ya Threshold mosi. Algorithm ke “shrink” (kufimpa) ba Features pene-pene na zero kana absolute values ya Features kele na zulu ya Threshold yina. Ba Researchers lenda sadila Soft thresholding na formula yai:

Derivative ya Output ya Soft thresholding na kutadila Input kele:

Formula yina kele na zulu ke monisa nde Derivative ya Soft thresholding kele 1 to 0. Kikadulu yai kele mutindu mosi na ReLU activation function. Yau yina, Soft thresholding lenda kitisa kigonsa ya Gradient vanishing mpe Gradient exploding na kati ya ba Algorithms ya Deep Learning.

Na kati ya Soft thresholding function, mutindu ya kutula Threshold fwete lungisa mambu zole. Ya ntete, Threshold fwete vanda numero ya positive. Ya zole, Threshold fwete luta ve valere ya nene ya Input signal. Kana ve, Output ta vanda kaka zero.

Diaka, yau ta vanda mbote kana Threshold me lungisa diambu ya tatu. Konso Sample fwete vanda na Threshold na yau mosi na kutadila kiteso ya Noise yina kele na kati.

Kikuma kele nde, kiteso ya Noise ke swaswanaka mbala mingi na kati ya ba Samples. Mu mbandu, Sample A lenda vanda na Noise fioti, kasi Sample B kele na Noise mingi na kati ya Dataset mosi. Na diambu yai, Sample A fwete sadila Threshold ya fioti ntangu ya Soft thresholding. Sample B fwete sadila Threshold ya nene. Ba Features yai ti ba Thresholds ke vidisaka ndimbu na yau ya kieleka na Deep Neural Networks. Kasi, mayele (logic) ya kisina ke bikalaka mutindu mosi. Na kutuba ya nkaka, konso Sample fwete vanda na Threshold na yau mosi. Kiteso ya Noise muntu ke ponaka Threshold yina.

3. Attention Mechanism

Ba Researchers lenda bakisa Attention mechanisms na Computer Vision kukonda mpasi. Meso ya bambisi lenda zaba bima na kutala nswalu bisika nionso. Na nima, meso ke tulaka Attention na kima yina yau ke sosa. Diambu yai ke pesaka nzila na ba systeme na kumona mambu mingi (details). Na ntangu mosi, ba systeme ke buyaka bansangu yina kele na mfunu ve. Sambua na kuzaba mambu mingi, beno tanga mikanda ya Attention mechanisms.

Squeeze-and-Excitation Network (SENet) kele mutindu ya mpa ya Deep Learning yina ke sadilaka Attention mechanisms. Na kati ya ba Samples ya kuswaswana, ba Feature channels ya kuswaswana ke salaka kisalu ya Classification na mutindu ya kuswaswana. SENet ke sadilaka Sub-network ya fioti sambua na kubaka Learn a set of weights. Na nima, SENet ke multiply ba Weights yai na ba Features ya ba Channels yina. Kisalu yai ke soba nene ya ba Features na konso Channel. Beto lenda mona diambu yai bonso Apply weighting to each feature channel.

Na mutindu yai, konso Sample kele na Set of weights na yau mosi. Na kutuba ya nkaka, ba Weights ya ba Sample zole ke vandaka ya kuswaswana. Na SENet, nzila ya kubaka ba Weights kele: “Global Pooling → Fully Connected Layer → ReLU Function → Fully Connected Layer → Sigmoid Function.”

4. Soft Thresholding with Deep Attention Mechanism

Deep Residual Shrinkage Network ke sadilaka structure ya SENet sub-network. Network ke sadilaka structure yai sambua na kusala Soft thresholding na nsi ya Deep attention mechanism. Sub-network (yina kele na kati ya box ya mbwaki) ke sala Learn a set of thresholds. Na nima, Network ke sala Soft thresholding na konso Feature channel na kusadilaka ba Thresholds yina.

Na kati ya Sub-network yai, systeme ke salaka ntete calcul ya Absolute values ya ba Features nionso na Input feature map. Na nima, systeme ke salaka Global average pooling mpe Average sambua na kubaka Feature mosi, yina beto ke binga A. Na nzila ya nkaka (Identity path), systeme ke tula Feature map na kati ya Fully connected network ya fioti na nima ya Global average pooling. Fully connected network yai ke sadilaka Sigmoid function bonso Layer ya nsuka. Function yai ke tulaka Output na kati ya 0 mpe 1. Diambu yai ke pesaka Coefficient mosi, yina beto ke binga α. Beto lenda tuba nde Threshold ya nsuka kele α × A. Yau yina, Threshold kele product ya ba numero zole. Numero mosi kele na kati ya 0 na 1. Numero ya nkaka kele Average ya Absolute values ya Feature map. Mutindu yai ke ndimisa nde Threshold kele positive. Mutindu yai ke ndimisa mpe nde Threshold kele ve nene kuluta.

Diaka, ba Samples ya kuswaswana ke pesaka ba Thresholds ya kuswaswana. Yau yina, beto lenda bakisa mutindu yai bonso Attention mechanism ya kieleka. Mechanism yai ke zabaka ba Features yina kele na mfunu ve na kisalu ya ntangu yai. Mechanism ke sobaka ba Features yai na ba valere yina kele pene-pene ya 0 na nzila ya ba Convolutional layers zole. Na nima, Mechanism ke tulaka ba Features yai na zero na nzila ya Soft thresholding. To, Mechanism ke zabaka ba Features yina kele na mfunu na kisalu ya ntangu yai. Mechanism ke sobaka ba Features yai na ba valere yina kele ntama na 0 na nzila ya ba Convolutional layers zole. Na nsuka, Mechanism ke bumbaka ba Features yai.

Na nsuka, beto ke sala Stack many basic modules. Beto ke tula mpe ba Convolutional layers, Batch normalization, Activation functions, Global average pooling, mpe Fully connected output layers. Mutindu yai ke tungaka Deep Residual Shrinkage Network ya mvimba.

5. Generalization Capability

Deep Residual Shrinkage Network kele mutindu ya nene (general method) sambua na Feature learning. Kikuma kele nde, na bisalu mingi ya Feature learning, ba Samples ke vandaka na Noise. Ba Samples ke vandaka mpe na bansangu yina kele na mfunu ve. Noise yai ti bansangu ya mfunu ve lenda bebisa kisalu ya Feature learning. Mu mbandu:

Yindula Image classification. Foto mosi lenda vanda na bima ya nkaka mingi. Beto lenda bakisa bima yai bonso “Noise”. Deep Residual Shrinkage Network lenda sadila Attention mechanism. Network ke monaka “Noise” yai. Na nima, Network ke sadilaka Soft thresholding sambua na kutula ba Features ya “Noise” yai na zero. Diambu yai lenda tomisa Accuracy ya Image classification.

Yindula Speech recognition. Mingi-mingi na bisika ya makelele bonso masolo na lweka ya nzila to na kati ya usine. Deep Residual Shrinkage Network lenda tomisa Accuracy ya Speech recognition. To na ndambu ya nkaka, Network ke pesaka mutindu (methodology). Mutindu yai lenda tomisa Accuracy ya Speech recognition.

Reference

Minghang Zhao, Shisheng Zhong, Xuyun Fu, Baoping Tang, Michael Pecht, Deep residual shrinkage networks for fault diagnosis, IEEE Transactions on Industrial Informatics, 2020, 16(7): 4681-4690.

https://ieeexplore.ieee.org/document/8850096

BibTeX

@article{Zhao2020,
  author    = {Minghang Zhao and Shisheng Zhong and Xuyun Fu and Baoping Tang and Michael Pecht},
  title     = {Deep Residual Shrinkage Networks for Fault Diagnosis},
  journal   = {IEEE Transactions on Industrial Informatics},
  year      = {2020},
  volume    = {16},
  number    = {7},
  pages     = {4681-4690},
  doi       = {10.1109/TII.2019.2943898}
}

Academic Impact

Mukanda (Paper) yai me baka ba citations kuluta 1,400 na Google Scholar.

Na kutadila ba statistique, ba Researchers me sadila Deep Residual Shrinkage Network (DRSN) na mikanda/malongi kuluta 1,000. Bisalu yai kele na ba domain mingi. Ba domain yai kele Mechanical engineering, Electrical power, Vision, Healthcare, Speech, Text, Radar, mpe Remote sensing.