Deep Residual Shrinkage Network: Otu Artificial Intelligence Method maka Highly Noisy Data

Deep Residual Shrinkage Network bụ ụdị Deep Residual Network emelitere ma kwalite. N’ezie, ọ bụ njikọta nke Deep Residual Network, attention mechanisms, na soft thresholding functions.

N’ogo ụfọdụ, enwere ike ịghọta etu Deep Residual Shrinkage Network si arụ ọrụ otu a: ọ na-eji attention mechanisms amata unimportant features (atụmatụ ndị na-adịghị mkpa) ma jiri soft thresholding functions mee ka ha bụrụ zero; n’aka nke ọzọ, ọ na-amata important features ma debe ha. Usoro a na-eme ka ike deep neural network nwere i-extract useful features site na signals nwere noise dịkwuo elu.

1. Research Motivation

Nke mbụ, mgbe a na-eme classify samples, ọ dịghị ka aga-esi zere inwe noise—dịka Gaussian noise, pink noise, na Laplacian noise. N’ikwu ya n’ụzọ sara mbara, samples na-enwekarị ozi ma ọ bụ ihe ndị na-enweghị njikọ na classification task a na-arụ ugbu a, nke a nwekwara ike ịbụ ihe a na-akpọ noise. Noise a nwere ike imetụta arụmọrụ classification n’ụzọ na-adịghị mma. (Soft thresholding bụ isi ihe na ọtụtụ signal denoising algorithms.)

Dịka ọmụmaatụ, n’oge mkparịta ụka n’akụkụ ụzọ, audio nwere ike ịgwakọta ya na ụda opi ụgbọ ala na taya. Mgbe a na-eme speech recognition na signals ndị a, ihe a na-anụ ga-enwerịrị nsogbu site na ụda ndị a na-adị n’azụ (background sounds). Site na elele anya nke deep learning, features ndị dabara na opi ụgbọ ala na taya kwesịrị ka ewepụ ha n’ime deep neural network ka ha ghara imetụta nsonaazụ speech recognition ahụ.

Nke abụọ, ọbụlagodi n’ime otu dataset, oke noise na-adịkarị iche site na sample gaa na nke ọzọ. (Nke a nwere njikọ na attention mechanisms; iji image dataset dịka ọmụmaatụ, ebe target object dị nwere ike ịdị iche na images dị iche iche, attention mechanisms nwere ike ilekwasị anya na ebe target object ahụ dị na image ọ bụla.)

Dịka ọmụmaatụ, mgbe a na-azụ (training) otu cat-and-dog classifier, weregodụ images ise a kpọrọ “dog” (nkịta). Image nke mbụ nwere ike inwe nkịta na oke, nke abụọ enwee nkịta na ọbọgwụ (goose), nke atọ enwee nkịta na ọkụkọ, nke anọ enwee nkịta na jakị, nke ise enwee nkịta na ọbọgwụ (duck). N’oge training, classifier ahụ ga-enwerịrị nsogbu site na ihe ndị na-adịghị mkpa dịka oke, ọbọgwụ, ọkụkọ, jakị, na ọbọgwụ, nke a ga-eme ka classification accuracy belata. Ọ bụrụ na anyị nwere ike ịmata ihe ndị a na-adịghị mkpa—oke, ọbọgwụ, ọkụkọ, jakị, na ọbọgwụ—ma wepụ features ha, ọ ga-ekwe omume ime ka accuracy nke cat-and-dog classifier ahụ dịkwuo mma.

2. Soft Thresholding

Soft thresholding bụ isi ihe na ọtụtụ signal denoising algorithms. Ọ na-ewepụ features ndị nwere absolute values dị ala karịa otu threshold, ma na-eme ka features ndị nwere absolute values dị elu karịa threshold a, bịaruo nso na zero (shrinks towards zero). Enwere ike ime ya site na iji usoro a:

Ihe derivative nke soft thresholding output n’ebe input nọ bụ:

Dịka egosiri n’elu, derivative nke soft thresholding bụ 1 ma ọ bụ 0. Akparamagwa a yiri nke ReLU activation function. Ya mere, soft thresholding nwekwara ike ibelata ihe egwu nke deep learning algorithms ịnweta nsogbu gradient vanishing na gradient exploding.

N’ime soft thresholding function, i-set threshold ahụ ga-emerịrị ka ọnọdụ abụọ a zuo oke: nke mbụ, threshold ga-abụrịrị positive number; nke abụọ, threshold agaghị aka maximum value nke input signal, ma ọ bụghị ya, output ga-abụ zero kpamkpam.

Na mgbakwunye, ọ ka mma ka threshold ahụ mezuo ọnọdụ nke atọ: sample ọbụla kwesịrị inwe independent threshold nke ya dabere na noise content o nwere.

Nke a bụ maka na oke noise na-adịkarị iche n’etiti samples. Dịka ọmụmaatụ, ọ na-emekarị n’ime otu dataset na Sample A nwere obere noise ebe Sample B nwere nnukwu noise. N’ọnọdụ a, mgbe a na-eme soft thresholding n’ime denoising algorithm, Sample A kwesịrị iji obere threshold, ebe Sample B kwesịrị iji threshold buru ibu. Ọ bụ ezie na features na thresholds ndị a anaghị enwe explicit physical definitions n’ime deep neural networks, isi eziokwu ahụ ka dịkwa otu. N’ikwu ya n’ụzọ ọzọ, sample ọbụla kwesịrị inwe independent threshold nke ya nke ekpebiri site na specific noise content o nwere.

3. Attention Mechanism

Ọ dị mfe ịghọta Attention mechanisms n’ime ngalaba computer vision. Anya anụmanụ nwere ike ịmata ihe ọ na-achọ site na iji ọsọ lele mpaghara niile, ma mechaa tinye uche (focus attention) n’ihe ahụ ọ na-achọ iji hụ ya nke ọma, ebe ọ na-eleghara ihe ndị ọzọ na-adịghị mkpa anya. Maka nkọwa zuru ezu, biko rụtụ aka na edemede gbasara attention mechanisms.

Squeeze-and-Excitation Network (SENet) bụ usoro deep learning dị ọhụrụ nke na-eji attention mechanisms. N’ime samples dị iche iche, onyinye nke feature channels dị iche iche na-enye na classification task na-adịkarị iche. SENet na-eji obere sub-network iji Learn a set of weights, wee mụbaa weights ndị a na features nke channels ndị ahụ iji gbanwee nha features n’ime channel ọ bụla. Enwere ike ịhụ usoro a dị ka i-Apply weighting to each feature channel n’ogo dị iche iche.

N’ụzọ a, sample ọbụla nwere independent set of weights nke ya. N’ikwu ya n’ụzọ ọzọ, weights maka samples abụọ ọbụla dị iche. Na SENet, ụzọ esi enweta weights bụ “Global Pooling → Fully Connected Layer → ReLU Function → Fully Connected Layer → Sigmoid Function.”

4. Soft Thresholding with Deep Attention Mechanism

Deep Residual Shrinkage Network nwetara mmụọ site na usoro SENet sub-network ahụ a kpọtụrụ aha n’elu iji mejuputa soft thresholding n’okpuru deep attention mechanism. Site na sub-network ahụ (nke egosiri n’ime igbe uhie), enwere ike i-Learn a set of thresholds iji tinye soft thresholding na feature channel ọ bụla.

N’ime sub-network a, a na-ebu ụzọ gbakọọ absolute values nke features niile dị na input feature map. Mgbe ahụ, site na global average pooling na averaging, a na-enweta otu feature, nke akpọrọ A. N’ụzọ nke ọzọ, feature map ahụ mgbe global average pooling gasịrị na-abanye n’ime obere fully connected network. Fully connected network a na-eji Sigmoid function dị ka layer ikpeazụ ya iji mee ka output dị n’etiti 0 na 1, na-enye otu coefficient akpọrọ α. Enwere ike igosipụta final threshold dị ka α×A. Ya mere, threshold bụ nsonaazụ nke nọmba dị n’etiti 0 na 1 mụbaa site na average nke absolute values nke feature map ahụ. Usoro a na-eme ka threshold abụghị naanị positive, kamakwa ọ naghị ebu oke ibu.

Ọzọkwa, samples dị iche iche na-enweta thresholds dị iche iche. N’ihi ya, n’ogo ụfọdụ, enwere ike ịkọwa nke a dị ka otu specialized attention mechanism: ọ na-amata features na-enweghị njikọ na task dị ugbu a, gbanwee ha gaa na values dị nso na zero site na iji two convolutional layers, ma jiri soft thresholding mee ka ha bụrụ zero; n’aka nke ọzọ, ọ na-amata features nwere njikọ na task dị ugbu a, gbanwee ha gaa na values dị anya na zero site na iji two convolutional layers, ma debe ha.

N’ikpeazụ, site na i-Stack many basic modules tinyere convolutional layers, batch normalization, activation functions, global average pooling, na fully connected output layers, a na-ewu Deep Residual Shrinkage Network zuru oke. Ihe ọzọ pụrụ iche bụ Identity shortcut nke na-enyere aka na training.

5. Generalization Capability

Deep Residual Shrinkage Network, n’ezie, bụ usoro feature learning zuru ụwa ọnụ (general). Nke a bụ n’ihi na n’ime ọtụtụ ọrụ feature learning, samples na-enwekarị ụfọdụ noise yana ozi na-enweghị njikọ. Noise a na ozi na-enweghị njikọ nwere ike imetụta arụmọrụ nke feature learning. Dịka ọmụmaatụ:

Na image classification, ọ bụrụ na image nwere ọtụtụ ihe ndị ọzọ n’otu oge, enwere ike ịghọta ihe ndị a dị ka “noise.” Deep Residual Shrinkage Network nwere ike iji attention mechanism chọpụta “noise” a, wee jiri soft thresholding mee ka features ndị dabara na “noise” a bụrụ zero, si otú a nwee ike ime ka image classification accuracy dịkwuo mma.

Na speech recognition, ọkachasị na gburugburu ebe nwere oke mkpọtụ dịka ebe mkparịta ụka n’akụkụ ụzọ ma ọ bụ n’ime factory workshop, Deep Residual Shrinkage Network nwere ike ime ka speech recognition accuracy dịkwuo mma, ma ọ bụ opekata mpe, nye otu usoro nwere ike ime ka speech recognition accuracy dịkwuo mma.

Reference

Minghang Zhao, Shisheng Zhong, Xuyun Fu, Baoping Tang, Michael Pecht, Deep residual shrinkage networks for fault diagnosis, IEEE Transactions on Industrial Informatics, 2020, 16(7): 4681-4690.

https://ieeexplore.ieee.org/document/8850096

BibTeX

@article{Zhao2020,
  author    = {Minghang Zhao and Shisheng Zhong and Xuyun Fu and Baoping Tang and Michael Pecht},
  title     = {Deep Residual Shrinkage Networks for Fault Diagnosis},
  journal   = {IEEE Transactions on Industrial Informatics},
  year      = {2020},
  volume    = {16},
  number    = {7},
  pages     = {4681-4690},
  doi       = {10.1109/TII.2019.2943898}
}

Academic Impact

Akwụkwọ a (paper) enwetala ihe karịrị citations 1,400 na Google Scholar.

Dabere na ọnụ ọgụgụ na-ezughị ezu, Deep Residual Shrinkage Network (DRSN) etinyere ya n’ọrụ ozugbo ma ọ bụ gbanwee ya ma tinye ya n’ọrụ n’ihe karịrị publications/studies 1,000 n’ime ọtụtụ ngalaba, gụnyere mechanical engineering, electrical power, vision, healthcare, speech, text, radar, na remote sensing.