Deep Residual Shrinkage Network nyan keuh saboh versi nyang ka geu-peugot leubeh get nibak Deep Residual Network. Pokok jih, Deep Residual Shrinkage Network nyan geu-gabung Deep Residual Network, attention mechanisms, ngon soft thresholding functions.
Jinoe, tanyoe jeut ta-pahami kiban cara keurija Deep Residual Shrinkage Network nyan lagèe nyoe. Phon-phon, network nyan geu-ngui attention mechanisms keu geu-tanda pat features nyang hana peunteng. Lheuh nyan, network geu-ngui soft thresholding functions keu geu-böh (set to zero) features nyang hana peunteng nyan. Seubalek jih, network nyan geu-tanda features nyang peunteng dan geu-simpan nyan. Proses nyoe geu-peukuat kemampuan deep neural network. Proses nyoe geu-bantu network nyan bak geu-tarik features nyang useuë (useful) nibak sinyal nyang na noise.
1. Research Motivation (Motivasi Peunugèt)
Phon-phon, noise nyan hana meu kiban taböh (inevitable) watèe algoritma geu-klasifikasi samples. Contoh noise nyan lagèe Gaussian noise, pink noise, ngon Laplacian noise. Leubeh luwah lom, samples nyan kayém na informasi nyang hana hubongan ngon tugas klasifikasi jinoe. Tanyoe jeut ta-anggap informasi nyang hana meuhubong nyan sibagoe noise. Noise nyoe jeut geu-peutron performa klasifikasi. (Soft thresholding nyan keuh langkah konci bak le that algoritma signal denoising.)
Misal jih, cuba gata bayangkan na ureueng meututur bak binèh jalan. Audio nyan teuntèe na su klakson moto ngon su grudoë moto. Tanyoe mungkén ta-peulaku speech recognition bak sinyal nyan. Su-su background nyan pasti geu-ganggu hasè jih. Nibak sudut pandang deep learning, deep neural network seuharoh jih geu-peugadoeh features nyang meuhubong ngon su klakson dan su grudoë moto nyan. Ngon cara geu-eliminasi nyan, features nyan hana geu-ganggu hasè speech recognition teuh.
Keu-dua, jumlah noise nyan kayém bidé-bidé tiyeup samples. Perubahan nyoe teujadi bah pih dalam dataset nyang sama. (Perubahan nyoe na saban bacut ngon attention mechanisms. Tanyoe cok saboh contoh image dataset. Lokasi objek target mungkén bidé-bidé bak tiyeup gambar. Attention mechanisms jeut geu-fokus bak lokasi spesifik objek target nyan bak tiyeup gambar.)
Misal jih, tanyoe ta-training saboh cat-and-dog classifier ngon limong boh gambar nyang na label “dog” (asèe). Gambar 1 mungkén na asèe ngon tikoh. Gambar 2 mungkén na asèe ngon angsa. Gambar 3 mungkén na asèe ngon manok. Gambar 4 mungkén na asèe ngon keulidèe. Gambar 5 mungkén na asèe ngon iték. Watèe ta-training, objek nyang hana meuhubong nyan geu-ganggu classifier. Objek-objek nyan tramasok tikoh, angsa, manok, keulidèe, ngon iték. Gangguan nyoe geu-akibat keu akurasi klasifikasi nyang meukureueng. Meunyo tanyoe jeut ta-tanda objek-objek nyang hana meuhubong nyan. Lheuh nyan, tanyoe jeut ta-eliminasi features nyang meuhubong ngon objek nyan. Ngon cara nyoe, tanyoe jeut ta-peutimang akurasi nibak cat-and-dog classifier nyan.
2. Soft Thresholding
Soft thresholding nyan keuh langkah inti (core step) bak le algoritma signal denoising. Algoritma nyan geu-eliminasi features meunyo nilai absolut nibak features nyan leubeh ubeut nibak threshold teutentu. Algoritma nyan geu-tarik (shrinks) features u arah nol meunyo nilai absolut features nyan leubeh rayek nibak threshold nyan. Peneliti jeut geu-implementasi soft thresholding ngon cara rumus di yup nyoe:
\[y = \begin{cases} x - \tau & x > \tau \\ 0 & -\tau \le x \le \tau \\ x + \tau & x < -\tau \end{cases}\]Turunan (derivative) nibak output soft thresholding teuhadap input nyan keuh:
\[\frac{\partial y}{\partial x} = \begin{cases} 1 & x > \tau \\ 0 & -\tau \le x \le \tau \\ 1 & x < -\tau \end{cases}\]Rumus di ateuh nyan geu-tunyoek bahwa turunan nibak soft thresholding nyan 1 atawa 0. Sifat nyoe saban persis lagèe sifat ReLU activation function. Maka nibak nyan, soft thresholding jeut geu-kurangi risiko gradient vanishing ngon gradient exploding bak algoritma deep learning.
Bak fungsi soft thresholding, cara ta-atoe threshold nyan beu meunuhi dua syarat. Phon, threshold nyan beu positif (positive number). Keu-dua, threshold nyan han jeut leubeh rayek nibak nilai maksimum sinyal input. Meunyo han, output jih habéh nol mandum.
Meutamah lom, threshold nyan leubeh get meunyo meunuhi syarat nyang keu-lhèe. Tiyeup samples beu na threshold droe jih masing-masing (independent threshold) meunurot kadar noise bak sample nyan.
Alaisuh jih, kadar noise nyan kayém meubidé antara samples. Misal jih, Sample A mungkén na noise bacut, seudangkan Sample B na noise leubeh le dalam dataset nyang sama. Dalam hal nyoe, Sample A seuharoh jih geu-ngui threshold nyang ubeut watèe soft thresholding. Sample B seuharoh jih geu-ngui threshold nyang leubeh rayek. Features ngon thresholds nyoe memang ka gadoeh definisi fisik nyang jaleh bak deep neural networks. Tapi, logika dasar jih mantong sama. Deungon narit laén, tiyeup sample beu na threshold nyang independen. Kadar noise spesifik nyan nyang teuntukan threshold nyan.
3. Attention Mechanism
Peneliti mudahlah geu-pahami attention mechanisms bak bidang computer vision. Sistem visual binatang jeut geu-bidé target ngon cara geu-scan mandum area ngon bagah. Lheuh nyan, sistem visual nyan geu-fokus attention bak objek target. Buet nyoe geu-bi keu sistem nyan jeut geu-tarik leubeh le detail. Bak watèe nyang sama, sistem nyan geu-teukan (suppress) informasi nyang hana meuhubong. Keu leubeh jaleh, neukalon laju literatur teuntang attention mechanisms.
Squeeze-and-Excitation Network (SENet) nyan saboh metode deep learning nyang lumayan baro nyang geu-ngui attention mechanisms. Bak samples nyang meubidé, feature channels nyang meubidé geu-sumbang (contribute) cara laén bak tugas klasifikasi. SENet geu-ngui sub-network ubeut keu geu-dapot saboh set weights (Learn a set of weights). Lheuh nyan, SENet geu-kalai weights nyoe ngon features nibak channels nyan masing-masing (Apply weighting to each feature channel). Operasi nyoe geu-ubah rayek ubeut features bak tiyeup channel. Tanyoe jeut ta-kalon proses nyoe lagèe ta-apliaksi attention (perhatian) level nyang meubidé keu feature channels nyang meubidé (Weighting).
Bak cara nyoe, tiyeup sample na saboh set weights nyang independen. Deungon narit laén, weights keu dua boh sample bebas nyan pasti bidé. Bak SENet, jalan (path) spesifik keu ta-dapot weights nyan keuh “Global Pooling → Fully Connected Layer → ReLU Function → Fully Connected Layer → Sigmoid Function.”
4. Soft Thresholding with Deep Attention Mechanism
Deep Residual Shrinkage Network geu-ngui struktur nibak sub-network SENet. Network nyan geu-ngui struktur nyoe keu geu-implementasi soft thresholding di yup deep attention mechanism. Sub-network nyan (nyang geu-tanda bak kotak mirah) geu-meurunoe saboh set thresholds (Learn a set of thresholds). Lheuh nyan, network nyan geu-aplikasi Soft thresholding keu tiyeup feature channel ngon geu-ngui thresholds nyan.
Bak sub-network nyoe, sistem phon-phon geu-itong nilai absolut nibak mandum features dalam input feature map. Lheuh nyan, sistem geu-peulaku global average pooling dan averaging keu geu-dapot saboh feature, nyang geu-tanda sibagoe A. Bak jalan laén (path laén), sistem geu-input feature map nyan u dalam small fully connected network lheuh global average pooling. Fully connected network nyoe geu-ngui Sigmoid function sibagoe layer akhé. Fungsi nyoe geu-normalisasi output antara 0 ngon 1. Proses nyoe geu-hasè saboh koefisien, nyang geu-tanda sibagoe α. Tanyoe jeut ta-tuléh threshold akhé sibagoe α × A. Jadi, threshold nyan keuh hasè kali dua boh angka. Saboh angka nyan antara 0 ngon 1. Angka saboh teuk nyan rata-rata nibak nilai absolut feature map. Cara nyoe geu-jamin bahwa threshold nyan positif. Cara nyoe cit geu-jamin threshold nyan han jeut keu rayek that.
Meutamah lom, samples nyang meubidé geu-hasè thresholds nyang meubidé. Akibat jih, tanyoe jeut ta-pahami cara nyoe sibagoe attention mechanism nyang khusuh. Mekanisme nyan geu-tanda features nyang hana meuhubong ngon tugas jinoe. Mekanisme nyan geu-ubah features nyan jeut keu nilai nyang rap nol meulalui dua convolutional layers. Lheuh nyan, mekanisme nyan geu-set features nyan keu nol ngon cara soft thresholding. Atawa, mekanisme nyan geu-tanda features nyang na hubongan ngon tugas jinoe. Mekanisme nyan geu-ubah features nyan jeut keu nilai nyang ji-oh nibak nol meulalui dua convolutional layers. Akhé jih, mekanisme nyan geu-pertahankan features nyan (Identity path).
Nyang teurakhé, tanyoe ta-susôn (stack) padum-padum boh basic modules (Stack many basic modules). Tanyoe cit ta-pasoe convolutional layers, batch normalization, activation functions, global average pooling, ngon fully connected output layers. Proses nyoe geu-bangun Deep Residual Shrinkage Network nyang leungkap.
5. Generalization Capability (Kemampuan Generalisasi)
Deep Residual Shrinkage Network nyan keuh cara umum (general method) keu feature learning. Alaisuh jih, samples nyan kayém na noise bak le tugas feature learning. Samples cit na informasi nyang hana meuhubong. Noise ngon informasi hana meuhubong nyoe mungkén geu-ganggu performa feature learning. Misal jih:
Cuba gata kalon bak image classification. Saboh gambar mungkén na le objek laén bak watèe nyang sama. Tanyoe jeut ta-pahami objek-objek nyoe sibagoe “noise.” Deep Residual Shrinkage Network mungkén sanggup geu-manfaat attention mechanism. Network nyan geu-perhati “noise” nyoe. Lheuh nyan, network geu-ngui soft thresholding keu geu-set features nyang meuhubong ngon “noise” nyan keu nol. Buet nyoe potènsi jih jeut geu-peucaë (improve) akurasi image classification.
Cuba gata kalon bak speech recognition. Khusuh jih, bak lingkungan nyang bising lagèe ureueng meututur bak binèh jalan atawa dalam pabrék. Deep Residual Shrinkage Network mungkén jeut geu-peucaë akurasi speech recognition. Atawa paléng kureueng, network nyan geu-tawarkan saboh metodologi. Metodologi nyoe sanggup keu geu-peutingkat akurasi speech recognition.
Reference (Referensi)
Minghang Zhao, Shisheng Zhong, Xuyun Fu, Baoping Tang, Michael Pecht, Deep residual shrinkage networks for fault diagnosis, IEEE Transactions on Industrial Informatics, 2020, 16(7): 4681-4690.
https://ieeexplore.ieee.org/document/8850096
BibTeX
@article{Zhao2020,
author = {Minghang Zhao and Shisheng Zhong and Xuyun Fu and Baoping Tang and Michael Pecht},
title = {Deep Residual Shrinkage Networks for Fault Diagnosis},
journal = {IEEE Transactions on Industrial Informatics},
year = {2020},
volume = {16},
number = {7},
pages = {4681-4690},
doi = {10.1109/TII.2019.2943898}
}
Academic Impact (Dampak Akademik)
Paper nyoe ka meuteumé leubeh nibak 1.400 citations bak Google Scholar.
Meunurot statistik nyang hana leungkap, peneliti ka geu-aplikasi Deep Residual Shrinkage Network (DRSN) bak leubeh nibak 1.000 publikasi/studi. Aplikasi-aplikasi nyoe geu-cakup bidang nyang luwah. Bidang-bidang nyoe tramasok mechanical engineering, electrical power, vision, healthcare, speech, text, radar, ngon remote sensing.