Deep Residual Shrinkage Network: ን Highly Noisy Data ዝኸውን ናይ Artificial Intelligence Method

Deep Residual Shrinkage Network ናይ Deep Residual Network improved variant (ዝተመሓየሸ ስሪት) እዩ። ብመሰረቱ፣ እዚ ናይ Deep Residual Network, Attention mechanisms, ከምኡውን Soft thresholding functions ዝተዋሃሃደ (integration) እዩ።

ብዝተወሰነ ደረጃ፣ working principle ናይ Deep Residual Shrinkage Network ከምዚ ዝስዕብ ክንርድኦ ንኽእል: Attention mechanisms ተጠቂሙ unimportant features (ዘይአገዳሲ features) identify ይገብር፣ ብድሕሪኡ soft thresholding functions ተጠቂሙ ናብ zero set ይገብሮም፤ በቲ ሓደ ወገን ድማ፣ important features identify ብምግባር retain ይገብሮም። እዚ process እዚ፣ deep neural network ካብ noise ዘለዎ signals ጠቃሚ ዝኾኑ features ናይ extract ምግባር ዓቅሚ (ability) enhance ይገብሮ።

1. Research Motivation (ናይ ምርምር ድራ𝒾)

መጀመርታ፣ samples classify ክንገብር ከለና፣ ከም Gaussian noise, pink noise, ከምኡውን Laplacian noise ዝአመሰሉ noise ክህልዉ natural እዩ። ብዝሰፍሐ አገላልጻ፣ samples ምስቲ current classification task ዘይተሓሓዝ information ክሕዙ ይኽእሉ እዮም፣ እዚ ድማ ከም noise ክውሰድ ይከኣል። እዚ noise ኣብ classification performance negative effect ክህልዎ ይኽእል እዩ። (Soft thresholding ኣብ ብዙሕ signal denoising algorithms ከም key step እዩ ዝውሰድ።)

ንኣብነት፣ ኣብ ጽርግያ ኴንካ conversation ክትገብር ከለኻ፣ እቲ audio ምስ ናይ መኪና ሆን (horns) ወይ ድማ ጫውጫውታ ናይ ጎማ (wheels) ክሕወስ ይኽእል እዩ። ኣብዚ signals እዚ speech recognition ክንሰርሕ ከለና፣ እቲ result በዚ background sounds ክጽሎ ግድን እዩ። ካብ Deep learning perspective ክንርእዮ ከለና፣ እቶም horns ከምኡውን wheels ዝወክሉ features፣ ኣብ ውሽጢ deep neural network ክጠፍኡ (eliminated ክኮኑ) ኣለዎም። እዚ ድማ ነቲ speech recognition results ከይጸልዎ ይከላኸል።

ካልኣይ፣ ኣብ ሓደ dataset እንተኾነ እውን፣ amount of noise ካብ sample ናብ sample ክፈላለ ይኽእል እዩ። (እዚ ምስ attention mechanisms ይመሳሰል እዩ፤ image dataset ከም ኣብነት እንተወሰድና፣ location ናይቲ target object ኣብ ዝተፈላለየ images ክፈላለ ይኽእል እዩ፣ attention mechanisms ድማ ኣብ ነፍሲ ወከፍ image ነቲ target object ዘለዎ location focus ክገብር ይኽእል።)

ንኣብነት፣ Cat-and-dog classifier train ክንገብር ከለና፣ 5 images “dog” ዝብል label ዘለወን ንውሰድ። እታ 1st image ከልብን ኣንጭዋን (mouse) ክትሕዝ ትኽእል፣ 2nd image ከልብን ደርሆን (chicken)፣ 3rd image ከልብን ገይስን (goose)፣ ወዘተ ክሕዛ ይኽእላ። Training ክንገብር ከለና፣ እቲ classifier በቶም irrelevant objects (ከም ኣንጭዋ፣ ደርሆ፣ ወዘተ) interference ክገጥሞ ይኽእል እዩ፣ እዚ ድማ accuracy ክንክይ ይገብሮ። ነዞም irrelevant objects (ከም mice, geese, chickens, donkeys, ducks) identify ጌርና፣ ነቶም corresponding features eliminate ክንገብሮም እንተክኢልና፣ accuracy ናይቲ cat-and-dog classifier improve ክንገብሮ ንኽእል ኢና።

2. Soft Thresholding

Soft thresholding ኣብ ብዙሕ signal denoising algorithms ከም core step እዩ ዝውሰድ። እዚ method እዚ፣ absolute values ካብ feature ነቲ ዝተወሰነ threshold ዝነኣሱ features eliminate ይገብር (features whose absolute values are lower than a certain threshold)፣ ነቶም ካብ threshold ዝበልጹ features ድማ ናብ zero shrink ይገብሮም። እዚ ብዝስዕብ formula implement ክግበር ይከኣል:

Derivative ናይ soft thresholding output with respect to input ከምዚ እዩ:

ከምዚ ኣብ ላዕሊ ዝረኣናዮ፣ derivative ናይ soft thresholding ወይ 1 ወይ ድማ 0 እዩ። እዚ property እዚ ምስ ReLU activation function ሓደ ዓይነት እዩ። ስለዚ፣ soft thresholding ን deep learning algorithms ካብ gradient vanishing ከምኡውን gradient exploding risk ክከላኸል ይኽእል እዩ።

ኣብ soft thresholding function፣ setting ናይ threshold ክልተ conditions ከማልእ ኣለዎ: 1st, እቲ threshold positive number ክኸውን ኣለዎ፤ 2nd, እቲ threshold ካብ maximum value ናይ input signal ክበልጽ የብሉን፣ እንተዘይኮይኑ እቲ output ምሉእ ብምሉእ zero ይኸውን።

ብተወሳኺ፣ እቲ threshold ሳልሳይ condition እንተዘማልአ ይምረጽ: each sample (ነፍሲ ወከፍ sample) base on its noise content ናይ ገዛእ ርእሱ independent threshold ክህልዎ ኣለዎ።

ምኽንያቱ noise content ኣብ መንጎ samples ስለ ዝፈላለ እዩ። ንኣብነት፣ ኣብ ሓደ dataset፣ Sample A ዝነኣሰ noise ክህልዎ ይኽእል፣ Sample B ድማ ብዙሕ noise ክህልዎ ይኽእል። ኣብዚ case፣ soft thresholding ክንገብር ከለና፣ Sample A ንእሽቶ threshold ክጥቀም ኣለዎ፣ Sample B ድማ ዓቢ threshold ክጥቀም ኣለዎ። ምንም እኳ እዞም features ከምኡውን thresholds ኣብ deep neural networks ፊዚካዊ ትርጉሞም (explicit physical definitions) እንተሰአኑ፣ እቲ basic logic ግን ሓደ እዩ። ማለት፣ each sample ናይ ገዛእ ርእሱ independent threshold ክህልዎ ኣለዎ።

3. Attention Mechanism

Attention mechanisms ኣብ field ናይ Computer Vision ብቀሊሉ ክንርድኦ ንኽእል ኢና። Visual system ናይ እንስሳት፣ ንሙሉእ area scan ብምግባር target distinguish ክገብር ይኽእል እዩ፣ ብድሕሪኡ attention ናብቲ target object focus ብምግባር more details extract ይገብር፣ ነቲ irrelevant information ድማ suppress ይገብሮ። ንዝርዝር ሓበሬታ፣ literature regarding attention mechanisms ተወከሱ።

Squeeze-and-Excitation Network (SENet) ሓደ relatively new ዝኾነ deep learning method ኮይኑ attention mechanisms ዝጥቀም እዩ። ኣብ different samples፣ contribution ናይ different feature channels ን classification task ዝፈላለ እዩ። SENet ንእሽቶ sub-network ብምጥቃም Learn a set of weights (ሓደ set weights ይመሃር)፣ ብድሕሪኡ ነዞም weights ምስ features ናይ respective channels multiply ይገብሮም፣ እዚ ድማ magnitude ናይ features adjust ይገብር። እዚ process እዚ፣ Apply weighting to each feature channel ተባሂሉ ክውሰድ ይከኣል።

በዚ approach እዚ፣ every sample ናይ ገዛእ ርእሱ independent set of weights ይህልዎ። ብኻልእ አዘራርባ፣ weights ናይ arbitrary two samples ዝተፈላለዩ እዮም። ኣብ SENet፣ weights ንምርካብ ዝጥቀመሉ specific path “Global Pooling → Fully Connected Layer → ReLU Function → Fully Connected Layer → Sigmoid Function” እዩ።

4. Soft Thresholding with Deep Attention Mechanism

Deep Residual Shrinkage Network ካብቲ ኣብ ላዕሊ ዝተጠቅሰ SENet sub-network structure inspiration ይወስድ፣ እዚ ድማ soft thresholding under deep attention mechanism implement ንምግባር እዩ። በቲ sub-network (ኣብቲ red box ዘሎ) ኣቢሉ፣ Learn a set of thresholds ይገብር፣ ብድሕሪኡ ን each feature channel soft thresholding apply ይገብር።

ኣብዚ sub-network፣ መጀመርታ absolute values ናይ ኩሉ features ኣብ input feature map calculate ይግበር። ብድሕሪኡ፣ through global average pooling and averaging፣ ሓደ feature ይርከብ፣ እዚ ድማ A ተባሂሉ denote ይግበር። ኣብቲ ካልእ path፣ እቲ feature map after global average pooling ናብ ሓደ small fully connected network input ይግበር። እዚ fully connected network ን Sigmoid function ከም final layer ይጥቀም፣ እዚ ድማ ን output ኣብ መንጎ 0 and 1 normalize ይገብሮ፣ እዚ coefficient α ተባሂሉ ይጽዋዕ። እቲ final threshold ከም α × A ተባሂሉ express ክግበር ይክእል። ስለዚ፣ እቲ threshold product ናይ ሓደ number between 0 and 1 ከምኡውን average of absolute values of the feature map እዩ። እዚ method፣ እቲ threshold positive ምዃኑ ጥራይ ዘይኮነ፣ excessively large (ካብ ዓቅሚ ንላዕሊ ዓቢ) ከይከውን ensure ይገብር።

ብተወሳኺ፣ different samples ዝተፈላለየ thresholds ይህልዎም። Consequently, እዚ ከም specialized attention mechanism ተገይሩ ክውሰድ ይከኣል: features irrelevant to the current task identify ይገብር፣ በቲ two convolutional layers ኣቢሉ ናብ values close to zero transform ይገብሮም፣ ብድሕሪኡ Soft thresholding ተጠቂሙ set to zero ይገብሮም፤ Alternatively, features relevant to the current task identify ይገብር፣ ናብ values far from zero transform ይገብሮም እሞ preserve ይገብሮም (Identity path).

ኣብ መወዳእታ፣ Stack many basic modules (ብዙሓት basic modules ብምደራረብ) ምስ convolutional layers, batch normalization, activation functions, global average pooling, ከምኡውን fully connected output layers፣ እቲ complete Deep Residual Shrinkage Network ይስራሕ።

5. Generalization Capability

Deep Residual Shrinkage Network ብሓቂ general feature learning method እዩ። ምኽንያቱ፣ ኣብ ብዙሕ feature learning tasks፣ samples more or less ገለ noise ወይ ድማ irrelevant information ስለ ዝሕዙ እዩ። እዚ noise and irrelevant information ን performance ናይ feature learning affect ክገብሮ ይኽእል እዩ። ንኣብነት:

ኣብ Image classification፣ ሓንቲ image ብዙሕ other objects እንተሒዛ፣ እዞም objects ከም “noise” ክንወስዶም ንኽእል ኢና። Deep Residual Shrinkage Network ነቲ attention mechanism ተጠቂሙ ነዚ “noise” notice ክገብር ይኽእል፣ ብድሕሪኡ soft thresholding ተጠቂሙ ነቲ corresponding features ናይዚ “noise” ናብ zero set ይገብሮ፣ በዚ ድማ image classification accuracy improve ክገብር ይኽእል።

ኣብ Speech recognition፣ ብፍላይ ኣብ relatively noisy environments (ንኣብነት roadside conversation ወይ factory workshop)፣ Deep Residual Shrinkage Network ን speech recognition accuracy improve ክገብር ይኽእል፣ ወይ at least ን accuracy improve ዝገብር methodology offer ይገብር።

Reference

Minghang Zhao, Shisheng Zhong, Xuyun Fu, Baoping Tang, Michael Pecht, Deep residual shrinkage networks for fault diagnosis, IEEE Transactions on Industrial Informatics, 2020, 16(7): 4681-4690.

https://ieeexplore.ieee.org/document/8850096

BibTeX

@article{Zhao2020,
  author    = {Minghang Zhao and Shisheng Zhong and Xuyun Fu and Baoping Tang and Michael Pecht},
  title     = {Deep Residual Shrinkage Networks for Fault Diagnosis},
  journal   = {IEEE Transactions on Industrial Informatics},
  year      = {2020},
  volume    = {16},
  number    = {7},
  pages     = {4681-4690},
  doi       = {10.1109/TII.2019.2943898}
}

Academic Impact

እዚ paper ኣብ Google Scholar ልዕሊ 1,400 citations ረኺቡ እዩ።

Based on incomplete statistics፣ Deep Residual Shrinkage Network (DRSN) ኣብ ልዕሊ 1,000 publications/studies ተወኪሱ ወይ modified ተገይሩ apply ተገይሩ እዩ። እዚ ድማ mechanical engineering, electrical power, vision, healthcare, speech, text, radar, ከምኡውን remote sensing ዝአመሰሉ wide range of fields የጠቓልል።