Deep Residual Shrinkage Network: Peteĩ Artificial Intelligence Method umi Highly Noisy Data-pe g̃uarã

Deep Residual Shrinkage Network niko peteĩ Deep Residual Network oñemyatyrõva. Añetehápe, kóva ombojoaju Deep Residual Network, attention mechanisms, ha soft thresholding functions.

Ikatu ja’e, Deep Residual Shrinkage Network omba’apo péicha: oipuru attention mechanisms ohechakuaa haguã umi unimportant features ha upéi oipuru soft thresholding functions oheja haguã chupekuéra zero-pe; ambue ládope, ohechakuaa umi important features ha oñongatu chupekuéra. Ko proceso omombarete pe deep neural network pokatu o-extract haguã useful features umi signals oguerekóvagui noise.

1. Research Motivation

Tenonderã, ñaclasifica vove umi samples, ndaikatúi ñamboyke pe noise —taha’e Gaussian noise, pink noise, térã Laplacian noise. Tuichave ñama’ẽramo, umi samples py’ỹinte oguereko información ndoúiva ko classification task rehe, ha upéva ikatu avei ñanembohe’i noise-rã. Ko noise ikatu ombyai pe classification performance. (Soft thresholding niko peteĩ paso tekotevẽte heta signal denoising algorithms-pe).

Techapyrã, ñañemongetávo tape yképe, pe audio oñembojehe’a car horns ha wheels punge. Jajapóvo speech recognition ko’ã signals rehe, pe resultado oñe-afectáta katuete ko’ã typu rupive. Deep learning punto de vista guive, umi features oúva horns ha wheels-gui oñelimina va’erã deep neural network ryepýpe, ani haguã o-afecta pe speech recognition resultado.

Mokõiha, oĩramo jepe peteĩ dataset-pe, pe noise cantidad py’ỹinte iñambue sample ha sample pa’ũme. (Kóva ojogua attention mechanisms-pe; jaipuru ramo peteĩ image dataset, pe target object renda ikatu iñambue umi ta’angápe, ha attention mechanisms ikatu o-focus pe target object rendaitépe).

Péicha, ña-entrena jave peteĩ cat-and-dog classifier, ñamoĩ 5 ta’anga oguerekóva etiqueta “dog”. Pe 1ª ta’anga ikatu oguereko peteĩ dog ha peteĩ mouse, pe 2ª peteĩ dog ha peteĩ goose, pe 3ª peteĩ dog ha peteĩ chicken, pe 4ª peteĩ dog ha peteĩ donkey, ha pe 5ª peteĩ dog ha peteĩ duck. Pe training aja, pe classifier katuete ojehe’áta umi objetos noñeikotevẽivare (irrelevants) taha’e mice, geese, chickens, donkeys, ha ducks, ha upéva omboguejy pe classification accuracy. Ikatúramo jahechakuaa ko’ã objetos irrelevants —mice, geese, chickens, donkeys, ha ducks— ha ñambogue umi features chupekuéra g̃uarã, ikatu ñamoporãve pe cat-and-dog classifier accuracy.

2. Soft Thresholding

Soft thresholding niko peteĩ paso principal heta signal denoising algorithms-pe. Kóva o-eliminate umi features oguerekóva absolute values imichĩvéva peteĩ threshold-gui, ha umi features oguerekóva absolute values ituichavéva, o-shrink chupekuéra zero gotyo. Ikatu jajapo ko fórmula rupive:

Pe derivative soft thresholding output rehegua, input rehe, ha’e:

Jahechaháicha, pe derivative soft thresholding-gui ha’e 1 térã 0. Kóva ojoguaite ReLU activation function-pe. Upévare, soft thresholding ikatu avei omboguejy pe riesgo deep learning algorithms ohasa haguã gradient vanishing ha gradient exploding.

Pe soft thresholding function-pe, ña-setea vove pe threshold, o-satisfy va’erã mokõi conditions: peteĩha, pe threshold ha’e va’erã positive number; mokõiha, pe threshold ndohasái va’erã pe maximum value input signal rehegua, aje’érõ pe output ha’éta zero memete.

Avei, iporãve pe threshold o-satisfy ramo peteĩ condition mbohapyha: peteĩteĩ sample oguereko va’erã threshold iñe’ẽvake (independent) o-dependéva pe noise content hese.

Kóva oiko pypore heta samples oguerekóui noise content iñambuéva. Techapyrã, py’ỹi oiko peteĩ dataset ryepýpe, Sample A oguereko sa’i noise, ha Sample B oguereko heta noise. Ko kásõpe, jajapóvo soft thresholding peteĩ denoising algorithm-pe, Sample A oipuru va’erã threshold michĩvéva, ha Sample B oipuru va’erã threshold tuichavéva. Deep neural networks-pe, ko’ã features ha thresholds operdéramo jepe i-definición física, pe lógica guasu opyta peteĩcha. He’ise, kada sample oguereko va’erã independent threshold oñe-determina-va pe noise content rupive.

3. Attention Mechanism

Attention mechanisms ndahasyietéi oñeikũmby haguã computer vision field-pe. Umi animales visual systems ikatu o-distingui targets pya’e oma’ẽvo pe área tuichakue rehe, ha upéi o-focus attention pe target object-pe o-extract haguã hetave details ha o-suppress umi irrelevant information. Reikuaaséramo hetave, ema’ẽmi umi literatura attention mechanisms rehegua.

Pe Squeeze-and-Excitation Network (SENet) ha’e peteĩ deep learning method ipyahúva oipurúva attention mechanisms. Umi samples iñambuéva apytépe, pe contribución umi feature channels ojapóva classification task-pe, py’ỹi iñambue. SENet oipuru peteĩ sub-network michĩva oikuaa haguã (Learn a set of weights) ha upéi o-multiply ko’ã weights umi features rehe o-adjust haguã pe magnitude umi features kada channel-pe. Ko proceso ikatu jahecha péicha: Apply weighting to each feature channel.

Ko enfoque-pe, peteĩteĩ sample oguereko independent set of weights. He’ise, umi weights oimeraẽva mokõi samples-pe g̃uarã, iñambue. SENet-pe, pe tape ojehupyty haguã weights ha’e: “Global Pooling → Fully Connected Layer → ReLU Function → Fully Connected Layer → Sigmoid Function”.

4. Soft Thresholding with Deep Attention Mechanism

Deep Residual Shrinkage Network oñemopyenda pe SENet sub-network structure oñemombe’u va’ekuére, o-implementa haguã soft thresholding peteĩ deep attention mechanism guýpe. Pe sub-network rupive (oĩva pe red box ryepýpe), ikatu jahechakuaa peteĩ conjunto de thresholds (Learn a set of thresholds) jajapo haguã soft thresholding kada feature channel-re.

Ko sub-network-pe, ñepyrũrã oñecalcula absolute values opaite features input feature map-pe g̃uarã. Upéi, global average pooling ha averaging rupive, ojehupyty peteĩ feature, ojeheróva A. Pe ambue path-pe, pe feature map ohasáma va’ekue global average pooling rupi, oike peteĩ small fully connected network-pe. Ko fully connected network oipuru Sigmoid function i-layer pahaguépe o-normalize haguã pe output 0 ha 1 pa’ũme, ha o-yield peteĩ coefficient ojeheróva α. Pe threshold pahague ikatu oñe-expressa péicha: α × A. Upévare, pe threshold ha’e pe producto peteĩ número 0 ha 1 pa’ũme, ha pe average umi absolute values feature map-gui. Ko método o-ensure pe threshold ha’eha positive, ha avei naicuichaitereíri.

Avei, samples iñambuéva oguereko thresholds iñambuéva. Upévare, ikatu ja’e kóva ha’eha peteĩ specialized attention mechanism: ohechakuaa features irrelevant ko task-pe g̃uarã, o-transforma chupekuéra values oĩva cerca de zero-gui (two convolutional layers rupive), ha o-set chupekuéra zero-pe oipurúvo soft thresholding; térã, ohechakuaa features relevant ko task-pe g̃uarã, o-transforma chupekuéra values mombyry zerógui, ha o-preserve chupekuéra.

Ipahápe, jajapo Stack many basic modules oñondive convolutional layers, batch normalization, activation functions, global average pooling, ha fully connected output layers, oñemopu’ã haguã pe Deep Residual Shrinkage Network kompletoite.

5. Generalization Capability

Deep Residual Shrinkage Network añetehápe niko peteĩ general feature learning method. Kóva oiko pypore heta feature learning tasks-pe, umi samples oguerekógui noise térã irrelevant information. Ko noise ha irrelevant information ikatu o-afecta pe feature learning performance. Techapyrã:

Image classification-pe, peteĩ imagen oguerekóramo heta ambue objetos, ko’ãva ikatu oñe-entende “noise” ramo. Deep Residual Shrinkage Network ikatu oipuru attention mechanism ohechakuaa haguã ko “noise”, ha upéi oipuru soft thresholding o-set haguã umi features ko “noise” rehegua zero-pe, ha péicha ikatu omoporãve image classification accuracy.

Speech recognition-pe, especialmente umi ambientes noisy-hápe (tapére ñañemongetávo térã peteĩ fábrica ryepýpe), Deep Residual Shrinkage Network ikatu omoporãve speech recognition accuracy, térã katu, oikuave’ẽ peteĩ metodología ikatúva omoporãve speech recognition accuracy.

Reference

Minghang Zhao, Shisheng Zhong, Xuyun Fu, Baoping Tang, Michael Pecht, Deep residual shrinkage networks for fault diagnosis, IEEE Transactions on Industrial Informatics, 2020, 16(7): 4681-4690.

https://ieeexplore.ieee.org/document/8850096

BibTeX

@article{Zhao2020,
  author    = {Minghang Zhao and Shisheng Zhong and Xuyun Fu and Baoping Tang and Michael Pecht},
  title     = {Deep Residual Shrinkage Networks for Fault Diagnosis},
  journal   = {IEEE Transactions on Industrial Informatics},
  year      = {2020},
  volume    = {16},
  number    = {7},
  pages     = {4681-4690},
  doi       = {10.1109/TII.2019.2943898}
}

Academic Impact

Ko paper oguereko hetave 1,400 citations Google Scholar-pe.

Ojehechaháicha estadísticamente (incomplete statistics), Deep Residual Shrinkage Network (DRSN) oñe-aplika directamente térã oñemboambue ha oñe-aplika hetave 1,000 publications/studies-pe, heta fields-pe: mechanical engineering, electrical power, vision, healthcare, speech, text, radar, ha remote sensing.