Deep Residual Shrinkage Network: Huk Artificial Intelligence Method Highly Noisy Data-paq

Deep Residual Shrinkage Network nisqaqa huk allinchasqa Deep Residual Network niraqmi. Chiqaptapuniqa, kayqa huñunmi Deep Residual Networkta, attention mechanismsta, hinallataq soft thresholding functionsta ima.

Huk partemanta qhawaspaqa, Deep Residual Shrinkage Networkpa llamk’ayninqa khaynatam entiendekunman: attention mechanismsta llamk’achin mana ancha allin featuresta riqsinanpaq, hinaspa soft thresholding functionswan ch’usaqman (zero) tukun; hukladumantaqa, allin featuresta riqsispa waqaychan. Kay ruwaymi deep neural networkpa atiyninta kallpachan allin featuresta hurqunanpaq noiseyuq signalsmanta.

1. Research Motivation

Ñawpaqta, samplesta classify ruwachkaspaqa, noiseqa (imaynan Gaussian noise, pink noise, Laplacian noise ima) manam karunchakuyta atikunchu. Aswan hatun rimaypi, samplesqa sapa kutim current classification taskwan mana tupaq willakuyta (information) apamun, chaypas noise hinallataqmi riqsisqa. Kay noiseqa classification performanceta waqllichinman. (Soft thresholdingqa ancha allinmi achka signal denoising algorithmspi).

Huk ejemplo: callekunapi rimachkaptinchikqa, audiopiqa autoq hornsnin utaq ruedanpa qapariyninmi chaqrukunman. Kay signalspi speech recognitionta ruwaptinchikqa, resultsqa manam allinchu kanqa kay background sounds rayku. Deep learningmanta qhawaspaqa, kay hornswan ruedakunaq featuresnintaqa deep neural network ukhupi chinkachina tiyan, speech recognition resultsta mana waqllichinanpaq.

Iskaykaq, kikin dataset ukhupipas, noiseqa sapa samplepi hukniraqmi. (Kayqa rikch’akunmi attention mechanismsman; huk image datasetta ejemplo hina qhawaspaqa, target objectpa maypi kasqanqa sapa imagepi hukniraqmi kanman; attention mechanismsqa atinmi focus ruwayta target objectpa locationninman sapa imagepi).

Ejemplopaq, cat-and-dog classifierta trainichkaspa, pichqa imagesta “dog” labelniyuqta qhawarisun. Ñawpaq kaq imagepiqa kanmanmi huk allqu huk ukuchawan, iskay kaqpi huk allqu huk wallatawan, kimsa kaqpi huk allqu huk wallpawan, tawa kaqpi huk allqu huk asnowan, pichqa kaqpitaq huk allqu huk patowan. Trainichkaspaqa, classifierqa sasachakunqapunim kay mana allin objetokunawan (ukuchakuna, wallatakuna, wallpakuna, asnokuna, patokuna), chaymi classification accuracyta urmachin. Sichus kay mana allin objetokunata riqsispa (ukuchakuna, wallatakuna, wallpakuna, asnokuna, patokuna) paykunaq featuresninta chinkachisunman chayqa, cat-and-dog classifierpa accuracyninqa aswan allinmi kanman.

2. Soft Thresholding

Soft thresholdingqa ancha allinmi achka signal denoising algorithms ukhupi. Chayqa chinkachinmi featuresta mayqinchus absolute valuesnin aswan huch’uy huk thresholdmanta, hinallataq shrink ruwan featuresta mayqinchus absolute valuesnin aswan hatun thresholdmanta zero ladoman. Kayqa ruwakunmanmi kay formulawan:

Soft thresholding outputpa derivativenin inputman qhawaspaqa kayhinam:

Hawa formulapi rikukusqanman hina, soft thresholdingpa derivativeninqa 1 utaq 0 kanman. Kayqa ReLU activation functionwan kaqllataqmi. Chayrayku, soft thresholdingqa yanapanmi deep learning algorithmsta mana gradient vanishing nitaq gradient exploding sasachakuykunapi urmananpaq.

Soft thresholding functionpiqa, threshold churayqa iskayta hunt’ananmi: ñawpaqta, thresholdqa positive number kanan; iskaykaq, thresholdqa manam input signalpa maximum valueninta yallinanchu, mana chayqa outputqa llapanmi zero kanqa.

Chaymantapas, aswan allinmi kanman threshold kay kimsa kaq conditionta hunt’aptin: sapa samplemi kikin independent thresholdniyuq kanan, hayk’a noise content kasqanman hina.

Imaraykuchus, noise contentqa sapa samplepi hukniraqmi. Ejemplopaq, kikin datasetpiqa Sample A pisi noiseyuq kanman, Sample Btaq aswan achka noiseyuq kanman. Chayna kaptinqa, signal denoising algorithmpi soft thresholdingta ruwaspaqa, Sample Aqa aswan huch’uy thresholdta llamk’achinan, Sample Btaq aswan hatun thresholdta llamk’achinan. Deep neural networkspi kay featureswan thresholds explicit physical definitionsninta chinkachiptinkupas, ukhupi logicankuqa kaqllam. Huk rimaypi, sapa samplemi kikin independent thresholdniyuq kanan, hayk’a noise kasqanman hina.

3. Attention Mechanism

Attention mechanismsqa facilmi entiendey computer vision fieldpi. Animalpa ñawinqa (visual systems) atinmi targetsta riqsiyta llapan areata utqaylla qhawaspa, chaymantataqmi attentionta churan target objectman aswan details hurqunanpaq, mana allin informationta saqispa. Aswan yachayta munaspaqa, attention mechanismsmanta qillqasqakunata qhaway.

Squeeze-and-Excitation Network (SENet)qa huk musuq deep learning methodmi attention mechanismsta llamk’achiq. Hukniraq samplespi, sapa feature channelpa contributionnin classification taskpiqa hukniraqmi. SENetqa huk huch’uy sub-networkta llamk’achin huk set of weightsta tarinanpaq, chaymantataq kay weightsta mirachin sapa channelpa featuresninwan, chay featurespa magnitudninta adjustananpaq. Kay ruwayqa khaynatam qhawakunman: Apply weighting to each feature channel (hukniraq levels of attentionta churaspa sapa feature channelman).

Kay ruwaypiqa, sapa samplemi kikin independent set of weightsniyuq. Huk rimaypi, iskay ima samplespa weightsninqa hukniraqmi. SENetpiqa, weights tarina ñanqa kaymi: “Global Pooling → Fully Connected Layer → ReLU Function → Fully Connected Layer → Sigmoid Function.”

4. Soft Thresholding with Deep Attention Mechanism

Deep Residual Shrinkage Networkqa hawapi rimusqanchik SENet sub-network structuremanta yachaqakun (inspiration) soft thresholdingta deep attention mechanismpi ruwananpaq. Sub-networkwan (puka cajapi rikukuq), huk Learn a set of thresholdsta yachakunman soft thresholdingta sapa feature channelman churananpaq.

Kay sub-networkpiqa, ñawpaqta llapan featurespa absolute valuesnintam yupan input feature mappi. Chaymanta, global average poolingwan averagingwan, huk featureta tarin, chaytaq A sutiwan riqsisqa. Huknin Identity pathpi, global average pooling pasasqa feature mapqa huk huch’uy fully connected networkman yaykun. Kay fully connected networkqa Sigmoid functionta llamk’achin final layer hina, outputta 0 mantawan 1 mantawan chawpipi kananpaq, chaymanta huk coefficient $\alpha$ nisqata qun. Final thresholdqa $\alpha \times A$ nisqawanmi riqsisqa. Chayrayku, thresholdqa kanqa: huk yupay (0 to 1) mirachisqa feature mappa absolute valuesninpa averagenwan. Kay methodqa manam threshold positive kanallanpaqchu yanapan, aswanpas mana nishuta hatun kananpaq.

Chaymantapas, hukniraq samplesqa hukniraq thresholdstami qun. Chayrayku, kayqa huk special attention mechanism hinam entiendekunman: riqsinmi features mana current taskwan tupaqta, hinaspa iskay convolutional layerswan kay featuresman chayan values cercaman 0, chaymantataq soft thresholdingwan zeroman tukun; utaq, riqsinmi features current taskwan tupaqta, hinaspa iskay convolutional layerswan kay featuresman chayan values karuman 0, hinaspa waqaychan.

Tukuyninpi, Stack many basic modules ruwaspa, convolutional layerswan, batch normalizationwan, activation functionswan, global average poolingwan, hinallataq fully connected output layerswan ima, hunt’asqa Deep Residual Shrinkage Networkmi ruwakun.

5. Generalization Capability

Deep Residual Shrinkage Networkqa, chiqaptapuni, huk general feature learning methodmi. Imaraykuchus achka feature learning taskspiqa, samplesqa imaymanatam noiseta utaq irrelevant informationta apamun. Kay noisewan irrelevant informationqa feature learningpa performancenta waqllichinman. Ejemplopaq:

Image classificationpi, sichus huk image achka hukniraq objetosta apamun chayqa, kay objetosqa “noise” hina riqsisqam. Deep Residual Shrinkage Networkqa atinmanmi attention mechanismta llamk’achiyta kay “noise” riqsinanpaq, chaymantataq soft thresholdingwan kay “noise“pa featuresninta zeroman churananpaq, chaywanmi image classification accuracyta allinchanman.

Speech recognitionpi, aswantaqa noisy environmentspi (imaynan callekunapi utaq factory workshopspi rimay), Deep Residual Shrinkage Networkqa speech recognition accuracyta allinchanman, utaq allin ñanta (methodology) qunman speech recognition accuracyta allinchananpaq.

Reference

Minghang Zhao, Shisheng Zhong, Xuyun Fu, Baoping Tang, Michael Pecht, Deep residual shrinkage networks for fault diagnosis, IEEE Transactions on Industrial Informatics, 2020, 16(7): 4681-4690.

https://ieeexplore.ieee.org/document/8850096

BibTeX

@article{Zhao2020,
  author    = {Minghang Zhao and Shisheng Zhong and Xuyun Fu and Baoping Tang and Michael Pecht},
  title     = {Deep Residual Shrinkage Networks for Fault Diagnosis},
  journal   = {IEEE Transactions on Industrial Informatics},
  year      = {2020},
  volume    = {16},
  number    = {7},
  pages     = {4681-4690},
  doi       = {10.1109/TII.2019.2943898}
}

Academic Impact

Kay paperqa 1400 masnin citationsniyuqmi Google Scholarpi.

Mana hunt’asqa statisticasman hina, Deep Residual Shrinkage Network (DRSN)qa achka kutita llamk’achisqa utaq allinchasqa karqan 1000 masnin publications/studies ukhupi, imaymana fieldspi: mechanical engineering, electrical power, vision, healthcare, speech, text, radar, remote sensing ima.