Deep Residual Shrinkage Network: An Artificial Intelligence Method for Highly Noisy Data

Deep Residual Shrinkage Network udi mushindu muakaja anyi “improved variant” wa Deep Residual Network. Mu bulelela, Deep Residual Shrinkage Network udi u-integrater bintu bisatu: Deep Residual Network, Attention mechanisms, ne Soft thresholding functions.

Tudi mua kumanya mushindu udi Deep Residual Shrinkage Network wenza mudimu mu njila eu: Tshia kumpala, network udi wenza mudimu ne Attention mechanisms bua kumanya “unimportant features” (features idi kayiyi ne mushinga). Pashishe, network udi wenza mudimu ne Soft thresholding functions bua kuteka features eyi ya patupu ku zero. Kadi, network udi umanya “important features” (features ya mushinga) ne udi u-retain features eyi. Mudimu eu udi ukolesha bukole bua deep neural network. Bualu ebu budi buambuluisha network bua kupatula “useful features” mu signals idi ne Noise.

1. Research Motivation

Tshia kumpala, Noise katu wapungila to patudi tu-classifieur samples. Bilejilu bia noise eu bidi bu mudi Gaussian noise, pink noise, ne Laplacian noise. Tuambe ne, samples itu misangu mivule ne mamanyisha adi kaayi ne mushinga bua mudimu wa classification utudi tuenza. Tudi mua kuangata mamanyisha aa a patupu bu Noise. Noise eu udi mua kukepesha bukole bua classification. (Soft thresholding udi tshitupa tshinene mu algorithms ya bungi ya signal denoising.)

Tuangate tshilejilu tshia muyuki ku luseke lua njila. Audio udi mua kuikala ne mitoyi ya mela ya mashinyi ne nkanu ya mashinyi. Tudi mua kujinga kuenza speech recognition pa signals eyi. Mitoyi ya panyima (background sounds) neyinyange tshipeta. Mu dimumona dia Deep Learning, deep neural network udi ne bua kumbusha features idi yumvuangana ne mela ya mashinyi ne nkanu. Kumbusha uku kudi ku-preventer features eyi bua kanyangi tshipeta tshia speech recognition.

Tshia ibidi, bungi bua Noise butu bushilangana pankatshi pa samples. Dishilangana edi didi dienzeka nansha munda mua dataset umue. (Dishilangana edi didi difuanangana ne Attention mechanisms. Tuangate image dataset bu tshilejilu. Muaba udi tshintu tshitudi tukeba (target object) udi mua kushilangana mu image yonso. Attention mechanisms idi mua ku-focus pa muaba udi tshintu atshi mu image yonso.)

Bua tshilejilu, elayi meji ne tudi tu-train cat-and-dog classifier ne images itanu idi ne tshimanyinu tshia “dog” (mbwa).

• Image 1 udi mua kuikala ne mbwa ne mpuku.
• Image 2 udi mua kuikala ne mbwa ne dibuwe.
• Image 3 udi mua kuikala ne mbwa ne nzolo.
• Image 4 udi mua kuikala ne mbwa ne mpunda.
• Image 5 udi mua kuikala ne mbwa ne dibata.

Pautudi tu-train, bintu ebi bia patupu (irrelevant objects) nebi-interfere ne classifier. Bintu ebi bidi: mpuku, dibuwe, nzolo, mpunda, ne dibata. Divulangana edi didi dikebesha dikepela dia classification accuracy. Biobi ne tudi mua kumanya bintu ebi bia patupu, pashishe tudi mua kumbusha features idi ipetangana ne bintu ebi. Mu mushindu eu, tudi mua kuvudija accuracy wa cat-and-dog classifier.

2. Soft Thresholding

Soft thresholding udi tshitupa tshia nshindamenu mu algorithms ya bungi ya signal denoising. Algorithm udi umbusha features biobi bikala ne absolute values ya features eyi mikese kupita threshold kampanda. Algorithm udi u-shrink features ku zero biobi bikala ne absolute values ya features eyi minene kupita threshold. Ba researchers badi mua ku-implementer Soft thresholding ne formule eu:

Derivative wa Soft thresholding output kudi input udi:

Formule wa hulu udi uleja ne derivative wa Soft thresholding udi 1 anyi 0. Ngikadilu eu (property) udi mumue ne wa ReLU activation function. Nunku, Soft thresholding udi mua kukepesha risk wa gradient vanishing ne gradient exploding munda mua algorithms ya Deep Learning.

Mu Soft thresholding function, disungula dia threshold didi ne bua kukumbaja conditions ibidi. Tshia kumpala, threshold udi ne bua kuikala positive number. Tshia ibidi, threshold kena ne bua kupita maximum value wa input signal. Biobi nanku, output neikale yonso zero.

Kabidi, bidi bimpe bua threshold akumbaje condition waisatu. Sample yonso udi ne bua kuikala ne threshold wawu wa pa nkayawu (independent threshold) bilondeshile bungi bua Noise udi mu sample au.

Bualu, bungi bua Noise butu bushilangana munkatshi mua samples. Tshilejilu, Sample A udi mua kuikala ne Noise mukese, kadi Sample B udi ne Noise wa bungi munda mua dataset umue. Mu bualu ebu, Sample A udi ne bua kuenza mudimu ne threshold mukese padiye wenza Soft thresholding. Sample B udi ne bua kuenza mudimu ne threshold munene. Mu deep neural networks, nansha mudi features eyi ne thresholds bijimija diumvuija diabi dia physics (physical definitions), kadi logic wa nshindamenu udi ushala mumue. Mu tshikoso, sample yonso udi ne bua kuikala ne threshold wa pa nkayawu. Bungi bua Noise ke budi bu-determiner threshold eu.

3. Attention Mechanism

Ba researchers badi mua kumanya bimpe Attention mechanisms mu tshitupa tshia computer vision. Mimuena ya nyama (visual systems of animals) idi mua kusunguluja targets padiyi i-scanner lukasa muaba wonso. Pashishe, mimuena idi i-focus attention pa target object. Tshienzedi etshi tshidi tshiambuluisha systems bua kupatula details ya bungi. Pa tshikondo tshimue, systems idi i-suppresser mamanyisha adi kaayi ne mushinga. Bua kumanya malu makuabu, bala mikanda idi yakula bua Attention mechanisms.

Squeeze-and-Excitation Network (SENet) udi uleja mushindu mupiamupia wa Deep Learning udi wenza mudimu ne Attention mechanisms. Mu samples mishilangane, feature channels bishilangane bidi bifila bintu bishilangane ku mudimu wa classification. SENet udi wenza mudimu ne sub-network mukese bua kupeta a set of weights (bisumbu bia weights). Pashishe, SENet udi u-multiplier weights eyi ne features ya channels yoyi. Mushindu eu udi u-adjuster bunene bua features mu channel yonso. Tudi mua kumona bualu ebu bu: Apply weighting to each feature channel (kuteka attention mishilangane pa channel yonso).

Mu njila eu, sample yonso udi ne independent set of weights. Tuambe ne, weights ya samples ibidi yonso idi mishilangane. Mu SENet, njila wa kupeta weights udi: “Global Pooling → Fully Connected Layer → ReLU Function → Fully Connected Layer → Sigmoid Function.”

4. Soft Thresholding with Deep Attention Mechanism

Deep Residual Shrinkage Network udi wenza mudimu ne structure wa SENet sub-network. Network udi wenza mudimu ne structure eu bua ku-implementer Soft thresholding munda mua deep attention mechanism. Sub-network (udi muleja mu tshibuta tshikunze mu tshimfuanyi) udi u- Learn a set of thresholds. Pashishe, network udi u-apply Soft thresholding kudi feature channel yonso ne thresholds eyi.

Munda mua sub-network eu, system udi tshia kumpala u-calculer absolute values ya features yonso idi mu input feature map. Pashishe, system udi wenza global average pooling ne averaging bua kupeta feature, utudi tubikila ne A. Mu njila mukuabu, system udi u-input feature map munda mua fully connected network mukese panyima pa global average pooling. Fully connected network eu udi wenza mudimu ne Sigmoid function bu layer wa ndekelu. Function eu udi u-normalizer output pankatshi pa 0 ne 1. Bualu ebu budi bupatula coefficient, utudi tubikila ne α. Tudi mua kufunda threshold wa ndekelu bu α × A. Nunku, threshold udi tshipeta tshia numero ibidi (product of two numbers). Numero umue udi pankatshi pa 0 ne 1. Numero mukuabu udi average wa absolute values ya feature map. Mushindu eu udi ujadika ne threshold udi positive. Mushindu eu udi kabidi ujadika ne threshold ki mmunene bikole to.

Kabidi, samples mishilangane idi ipatula thresholds mishilangane. Nunku, tudi mua kumvua mushindu eu bu special attention mechanism. Mechanism eu udi umanya features idi kayiyi ne mushinga ne task wa mpindieu. Mechanism udi u-transform features eyi bua ikale values idi pabuipi ne zero ku diambuluisha dia convolutional layers ibidi. Pashishe, mechanism udi uteka features eyi ku zero ne Soft thresholding. Anyi tuambe ne, mechanism udi umanya features idi ne mushinga ne task wa mpindieu. Mechanism udi u-transform features eyi bua ikale values idi kule ne zero ku diambuluisha dia convolutional layers ibidi. Ku ndekelu, mechanism udi u-preserve (ulama) features eyi.

Bua kujikija, tudi tu- Stack many basic modules bungi kampanda. Tudi tusangisha kabidi convolutional layers, batch normalization, activation functions, global average pooling, ne fully connected output layers. Bualu ebu budi buibaka Deep Residual Shrinkage Network mujima.

5. Generalization Capability

Deep Residual Shrinkage Network udi mushindu wa bonso (general method) wa feature learning. Bualu, mu misangu mivule ya feature learning tasks, samples itu ne Noise. Samples itu kabidi ne mamanyisha a patupu. Noise eu ne mamanyisha a patupu bidi mua kunyanga tshipeta tshia feature learning. Tshilejilu:

Tuangate image classification. Image umue udi mua kuikala ne bintu bikuabu bia bungi pa tshikondo tshimue. Tudi mua kumvua bintu ebi bu “Noise”. Deep Residual Shrinkage Network udi mua kuenza mudimu ne Attention mechanism. Network udi umona “Noise” eu. Pashishe, network udi wenza mudimu ne Soft thresholding bua kuteka features idi ipetangana ne “Noise” eu ku zero. Tshienzedi etshi tshidi mua kuvudija image classification accuracy.

Tuangate speech recognition. Nangananga mu miaba idi ne mitoyi ya bungi, bu mudi ku luseke lua njila anyi munda mua usine (factory workshop). Deep Residual Shrinkage Network udi mua kuvudija speech recognition accuracy. Anyi ku ndekelu, network udi ufila methodology. Methodology eu udi mukumbane bua kuvudija speech recognition accuracy.

Reference

Minghang Zhao, Shisheng Zhong, Xuyun Fu, Baoping Tang, Michael Pecht, Deep residual shrinkage networks for fault diagnosis, IEEE Transactions on Industrial Informatics, 2020, 16(7): 4681-4690.

https://ieeexplore.ieee.org/document/8850096

BibTeX

@article{Zhao2020,
  author    = {Minghang Zhao and Shisheng Zhong and Xuyun Fu and Baoping Tang and Michael Pecht},
  title     = {Deep Residual Shrinkage Networks for Fault Diagnosis},
  journal   = {IEEE Transactions on Industrial Informatics},
  year      = {2020},
  volume    = {16},
  number    = {7},
  pages     = {4681-4690},
  doi       = {10.1109/TII.2019.2943898}
}

Academic Impact

Paper eu ukadi mupeta citations bipite pa 1,400 pa Google Scholar.

Bilondeshile statistics idi kayiyi mijima, ba researchers bakadi benze mudimu ne Deep Residual Shrinkage Network (DRSN) mu publications/studies bipite pa 1,000. Miaba idi mudimu eu muenzeka idi ya bungi. Miaba eyi idi: mechanical engineering, electrical power, vision, healthcare, speech, text, radar, ne remote sensing.