中国物理B ›› 2022, Vol. 31 ›› Issue (8): 80503-080503.doi: 10.1088/1674-1056/ac5886

• • 上一篇    下一篇

Hyperparameter on-line learning of stochastic resonance based threshold networks

Weijin Li(李伟进), Yuhao Ren(任昱昊), and Fabing Duan(段法兵)   

  1. College of Automation, Qingdao University, Qingdao 266071, China
  • 收稿日期:2022-01-06 修回日期:2022-02-16 接受日期:2022-02-25 出版日期:2022-07-18 发布日期:2022-07-18
  • 通讯作者: Fabing Duan E-mail:fabingduan@qdu.edu.cn
  • 基金资助:
    Project supported by the Natural Science Foundation of Shandong Province, China (Grant No. ZR2021MF051).

Hyperparameter on-line learning of stochastic resonance based threshold networks

Weijin Li(李伟进), Yuhao Ren(任昱昊), and Fabing Duan(段法兵)   

  1. College of Automation, Qingdao University, Qingdao 266071, China
  • Received:2022-01-06 Revised:2022-02-16 Accepted:2022-02-25 Online:2022-07-18 Published:2022-07-18
  • Contact: Fabing Duan E-mail:fabingduan@qdu.edu.cn
  • Supported by:
    Project supported by the Natural Science Foundation of Shandong Province, China (Grant No. ZR2021MF051).

摘要: Aiming at training the feed-forward threshold neural network consisting of nondifferentiable activation functions, the approach of noise injection forms a stochastic resonance based threshold network that can be optimized by various gradient-based optimizers. The introduction of injected noise extends the noise level into the parameter space of the designed threshold network, but leads to a highly non-convex optimization landscape of the loss function. Thus, the hyperparameter on-line learning procedure with respective to network weights and noise levels becomes of challenge. It is shown that the Adam optimizer, as an adaptive variant of stochastic gradient descent, manifests its superior learning ability in training the stochastic resonance based threshold network effectively. Experimental results demonstrate the significant improvement of performance of the designed threshold network trained by the Adam optimizer for function approximation and image classification.

关键词: noise injection, adaptive stochastic resonance, threshold neural network, hyperparameter learning

Abstract: Aiming at training the feed-forward threshold neural network consisting of nondifferentiable activation functions, the approach of noise injection forms a stochastic resonance based threshold network that can be optimized by various gradient-based optimizers. The introduction of injected noise extends the noise level into the parameter space of the designed threshold network, but leads to a highly non-convex optimization landscape of the loss function. Thus, the hyperparameter on-line learning procedure with respective to network weights and noise levels becomes of challenge. It is shown that the Adam optimizer, as an adaptive variant of stochastic gradient descent, manifests its superior learning ability in training the stochastic resonance based threshold network effectively. Experimental results demonstrate the significant improvement of performance of the designed threshold network trained by the Adam optimizer for function approximation and image classification.

Key words: noise injection, adaptive stochastic resonance, threshold neural network, hyperparameter learning

中图分类号:  (Noise)

  • 05.40.Ca
02.50.-r (Probability theory, stochastic processes, and statistics) 84.35.+i (Neural networks)