Deep Residual Shrinkage Network: Inshila ya Artificial Intelligence iya ma Data aya kwata High Noise

Deep Residual Shrinkage Network musango uwawaminako uwa Deep Residual Network. Mu kwipifya, Deep Residual Shrinkage Network isata Deep Residual Network, attention mechanisms, na soft thresholding functions pamo.

Kuti twaumfwikisha ifyo Deep Residual Shrinkage Network ibomba muli iyi nshila. Ica kubalilapo, network ibomfya attention mechanisms ukusanga ama features ayashicindeme. Lyena, network ibomfya soft thresholding functions ukucita set aya ma features ayashicindeme kuli zero. Icapusanako, network ilasanga ama features ayacindama no kuyasunga. Uyu mulimo ulakosha amaka ya deep neural network. Uyu mulimo ulalenga network ukusanga ama features ayasuma ukufuma ku signals ishikwete noise.

1. Umulandu wa Kufwailisha (Research Motivation)

Ica kubalilapo, noise yaba fye ponse nangu algorithm ilecita classify ama samples. Ama example ya iyi noise yasanshamo Gaussian noise, pink noise, na Laplacian noise. Mu kusalanganya, ama samples ilingi yalakwata ifyebo ifishikumine ku mulimo wa classification uulebombwa. Kuti twatila ifi fyebo ifishikumine e noise. Iyi noise kuti yacefya amaka ya classification. (Soft thresholding e nshila ikalamba mu ma algorithms ayengi aya signal denoising.)

Ku ca kumwenako, tontonkanyeni pa kulanshanya mu mbali ya musebo. Mu mashiwi kuti mwaba icongo ca ma horn ya myotoka na ma wheel. Limbi kuti tulefwaya ukucita speech recognition pali aya mashiwi. Icongo ca ku numa cileta ubwafya ku fifumamo. Ukulingana na deep learning, deep neural network ifwile ukufumyamo ama features aya ma horn na ma wheel. Uku kufumyamo kulalesha ama features ukonaula ififuma mu speech recognition.

Ica bubili, ubwingi bwa noise ilingi bulapusanapusana pa kati ka ma samples. Uku kupusana kubako nangu fye ni mu dataset imo ine. (Uku kupusana kwaba kwati ni attention mechanisms. Tulebula image dataset nge ca kumwenako. Apa kuli icintu ico tulefwaya (target object) mu image kuti papusana ukulingana ne fikope. Attention mechanisms kuti yabika amano pa cifulo cene apo icintu cili mu image imo na imo.)

Ku ca kumwenako, tontonkanyeni pa kusambilisha classifier ya paka na imbwa ukubomfya ifikope fisano ifyalembwapo ati “dog.” Image 1 limbi kuti mwaba imbwa na kwingalika. Image 2 limbi mwaba imbwa na coco. Image 3 limbi mwaba imbwa na nkoko. Image 4 limbi mwaba imbwa na punda. Image 5 limbi mwaba imbwa na bata. Ilyo classifier ilesambilila (training), ifintu ifishikumineko filaleta icongo kuli classifier. Ifi fintu fyasanshamo bakwingalika, bacoco, inkoko, bapunda, na babata. Ici icongo cilenga classification accuracy ukuya panshi. Nga kuti twasanga ifi fintu ifishikumineko. Lyena, kuti twafumyamo ama features ayeminenako ifi fintu. Muli iyi nshila, kuti twawamyako accuracy ya classifier ya paka na imbwa.

2. Soft Thresholding

Soft thresholding e nshila ikalamba mu ma algorithms ayengi aya signal denoising. Algorithm ilafumyamo ama features nga ca kutila absolute value ya aya ma features yali panshi ya threshold imo. Algorithm ile-shrink-a ama features ukuya kuli zero nga ca kutila absolute value ya aya ma features yali pa muulu wa iyi threshold. Abafwailisha (Researchers) kuti babomfya soft thresholding ukulingana na iyi formula:

Derivative ya soft thresholding output ukulingana na input ni:

Iyi formula ya pa muulu ilelanga ukuti derivative ya soft thresholding ni 1 nangu 0. Uyu musango waba fye cimo na ReLU activation function. Eico, soft thresholding kuti yacefyako ubwafya bwa gradient vanishing na gradient exploding mu ma algorithms ya deep learning.

Muli soft thresholding function, uku-setting-a threshold kufwile ukukonka ama conditions yabili. Ica kubalilapo, threshold ifwile ukuba positive number. Ica bubili, threshold taifwile ukucila pali maximum value ya input signal. Nga ca kutila yacila, output kuti yaba fye zero yonse.

Na kabili, threshold ifwile no kukonka condition ya butatu. Sample imo na imo ifwile ukukwata threshold ya iko yeka ukulingana na noise content ya iyo sample.

Umulandu waba wa kutila noise content ilingi ilapusana mu ma samples. Ku ca kumwenako, Sample A limbi kuti yakwata noise inono lelo Sample B yakwata noise iingi muli dataset imo ine. Muli ubu busanso, Sample A ifwile ukubomfya threshold inono ilyo tulecita soft thresholding. Sample B ifwile ukubomfya threshold ikalamba. Muli deep neural networks, nangu ca kutila aya ma features na ma thresholds tafyakwata physical definition iya-angalila. Lelo, amano yakalamba (logic) yaba fye cimo. Mu mashiwi yambi, sample imo na imo ifwile ukukwata threshold ya iko yeka (independent). Noise content eilepingula iyi threshold.

3. Attention Mechanism

Abafwailisha kuti baumfwikisha attention mechanisms mu computer vision. Amenso ya nama kuti yapusanya ifintu (targets) ukupitila mu kulolesha bwangu bwangu pa cifulo conse. Lyena, amenso yabika amano (focus attention) pa cintu ico balefwaya. Uku kucita kulenga amenso ukumona ifishinka fyonse. Pa nshita imo ine, amenso yalasuula ifyebo ifishicindeme. Pa kumfwa ifingi, belengeni ifitabo pa attention mechanisms.

Squeeze-and-Excitation Network (SENet) yimininako inshila ipya iya deep learning iyibomfya attention mechanisms. Mu ma samples ayapusanapusana, ama feature channels ayapusanapusana yalabomba imilimo iyapusanapusana ku classification task. SENet ibomfya sub-network inono ukusanga ama weights (Learn a set of weights). Lyena, SENet ilacita multiply aya ma weights na ma features aya ma channels yene. Uku kucita kulalula ubukulu bwa ma features muli channel imo na imo. Kuti twamona uyu mulimo kwati kubika attention iyalekanalekana pa ma feature channels ayapusanapusana (Apply weighting to each feature channel).

Muli iyi nshila, sample imo na imo yalikwata ama weights ya iko yeka (independent set). Mu mashiwi yambi, ama weights ya ma samples yabili ayali yonse yaba ayapusana. Muli SENet, inshila ya kusangilamo ama weights ni: “Global Pooling → Fully Connected Layer → ReLU Function → Fully Connected Layer → Sigmoid Function.”

4. Soft Thresholding muli Deep Attention Mechanism

Deep Residual Shrinkage Network ibomfya umupangilwe wa SENet sub-network. Network ibomfya uyu mupangilwe ukubomfya soft thresholding muli deep attention mechanism. Sub-network (iyalangiwa mu bokoshi lwakashika) ila-sambilila (Learn a set of thresholds). Lyena, network ibomfya soft thresholding kuli feature channel imo na imo ukubomfya aya ma thresholds.

Muli iyi sub-network, system intanshi ilapenda ama absolute values ya ma features yonse muli input feature map. Lyena, system icita global average pooling na averaging ukusanga feature, iyo twita ati A. Mu nshila imbi (Identity path), system ibika feature map muli fully connected network inono pa numa ya global average pooling. Iyi fully connected network ibomfya Sigmoid function nge layer ya kulekelesha. Iyi function ilacita normalize ififumamo ukuba pa kati ka 0 na 1. Uyu mulimo ulafumya namba, iyo twita ati α. Kuti twalanda ukuti threshold ya kulekelesha ni α × A. Eico, threshold cisansho (product) ca manamba yabili. Namba imo yaba pa kati ka 0 na 1. Namba imbi ni average ya ma absolute values ya feature map. Iyi nshila ilashininkisha ukuti threshold yaba positive. Iyi nshila ilashininkisha no kuti threshold taikulisha sana.

Na kabili, ama samples ayapusanapusana yafumya ama thresholds ayapusanapusana. Eico, kuti twaumfwikisha iyi nshila kwati ni specialized attention mechanism. Iyi mechanism ilasanga ama features ayashikumine ku task ya nomba. Iyi mechanism ilalula aya ma features ukuba amanamba ayali mupepi na 0 ukubomfya ama convolutional layers yabili. Lyena, iyi mechanism ibomfya soft thresholding ukucita set aya ma features ukuya kuli zero. Nangu, iyi mechanism ilasanga ama features ayakumine ku task ya nomba. Iyi mechanism ilalula aya ma features ukuba amanamba ayali ukutali na 0 ukubomfya ama convolutional layers yabili. Pakulekelesha, iyi mechanism ilasunga aya ma features.

Pa kulekelesha, tula-stack-a (Stack many basic modules) ama basic modules ayengi. Tulasanshamo na ma convolutional layers, batch normalization, activation functions, global average pooling, na fully connected output layers. Uyu mulimo e upanga Deep Residual Shrinkage Network yonse.

5. Amaka ya Kubomba Ponse (Generalization Capability)

Deep Residual Shrinkage Network ni nshila ya feature learning iyo mwingabomfya ku fintu ifingi (general method). Umulandu waba wa kutila ama samples ilingi yakwata noise mu milimo iingi iya feature learning. Ama samples yalakwata ne fyebo ifishikumineko. Iyi noise ne fyebo ifishikumineko kuti fyaonaula amaka ya feature learning. Ku ca kumwenako:

Tontonkanyeni pa image classification. Icikope (Image) kuti cakwata ifintu fimbi ifingi pa nshita imo ine. Kuti twaumfwa ifi fintu kwati ni “noise.” Deep Residual Shrinkage Network limbi kuti yabomfya attention mechanism. Network yamona iyi “noise.” Lyena, network yabomfya soft thresholding ukucita set ama features ya iyi “noise” kuli zero. Uku kucita kuti kwawamyako image classification accuracy.

Tontonkanyeni pa speech recognition. Maka maka, tontonkanyeni pa fifulo ifili ne congo cikalamba pamo nga ukulanshanya mu mbali ya musebo nangu mu kati ka factory workshop. Deep Residual Shrinkage Network limbi kuti yawamyako speech recognition accuracy. Nangu fye, network ilepeela inshila (methodology). Iyi nshila yalikwata amaka ya kuwamyako speech recognition accuracy.

Reference

Minghang Zhao, Shisheng Zhong, Xuyun Fu, Baoping Tang, Michael Pecht, Deep residual shrinkage networks for fault diagnosis, IEEE Transactions on Industrial Informatics, 2020, 16(7): 4681-4690.

https://ieeexplore.ieee.org/document/8850096

BibTeX

@article{Zhao2020,
  author    = {Minghang Zhao and Shisheng Zhong and Xuyun Fu and Baoping Tang and Michael Pecht},
  title     = {Deep Residual Shrinkage Networks for Fault Diagnosis},
  journal   = {IEEE Transactions on Industrial Informatics},
  year      = {2020},
  volume    = {16},
  number    = {7},
  pages     = {4681-4690},
  doi       = {10.1109/TII.2019.2943898}
}

Icupo mu Masambililo (Academic Impact)

Iyi paper yalikwata ama citation ukucila 1,400 pali Google Scholar.

Ukulingana ne fipendo (statistics), abafwailisha balibomfya Deep Residual Shrinkage Network (DRSN) mu publications nangu amasambililo ukucila 1,000. Imibomfeshe yasanshamo ififulo ifingi. Ifi fifulo fyasanshamo mechanical engineering, electrical power, vision, healthcare, speech, text, radar, na remote sensing.