Convergence of Gradient Method for Elman Networks
-
摘要: 考虑有限样本集上Elman网络梯度学习法的确定性收敛性.证明了误差函数的单调递减性.给出了一个弱收敛性结果和一个强收敛结果,表明误差函数的梯度收敛于0,权值序列收敛于固定点.通过数值例子验证了理论结果的正确性.Abstract: The gradient method for training Elman networks with finite training sample set is considered. The monotonicity of the error function in the iteration is shown. A weak and a strong convergence results are proved, indicating that the gradient of the error function goes to zero and the weight sequence goes to a fixed point, respectively. A numerical example is given to support the theoretical findings.
-
Key words:
- Elman network /
- gradient learning algorithm, convergence /
- monotonicity /
-
[1] Elman J L. Finding structure in time[J].Cognitive Science,1990,14(2):179-211. doi: 10.1207/s15516709cog1402_1 [2] Tsoi A C, Back A D.Locally recurrent globally feedforward networks:a critical review of architectures[J].IEEE Transactions on Neural Networks,1994,5(2):229-239. doi: 10.1109/72.279187 [3] WANG De-liang,LIU Xiao-mei,Ahalt S C.On temporal generalization of simple recurrent networks[J].Neural Networks,1996,9(7):1099-1118. doi: 10.1016/0893-6080(96)00034-2 [4] Kremer S C. On the computational power of Elman-style recurrent net works[J].IEEE Transactions on Neural Networks,1995,6(4):1000-1004. doi: 10.1109/72.392262 [5] Pham D T,Liu X. Training of elman networks and dynamic system modeling[J].International Journal of Systems Science,1996,27(2):221-226. doi: 10.1080/00207729608929207 [6] Cartling B.On the implicit acquisition of a context-free grammar by a simple recurrent neural network[J].Neurocomputing,2008,71(7/9):1527-1537. doi: 10.1016/j.neucom.2007.05.006 [7] LI Xiang,CHEN Zeng-qiang,YUAN Zhu-zhi,et al.Generating chaos by an Elman network[J].IEEE Transactions on Circuits and Systems-Ⅰ,2001,48(9):1126-1131. doi: 10.1109/81.948441 [8] Ekici S,Yildirim S,Poyraz M.A transmission line fault locator based on Elman recurrent networks[J].Applied Soft Computing,DOI: 10.1016/J.asoc.2008.04.011. [9] Neto L B, Coelho P H G,Soares de Mello J C C B,et al.Flow estimation using an Elman networks[A].In:Wunsch D,Ed.Proceedings of 2004 IEEE International Joint Conference on Neural Networks[C].Budapest, Hungary:IEEE Press,2004,831-836. [10] Demuth H B, Beale M H, Hagan M T.Neural Network Toolbox User'Sguide[M].atick, MA: The Mathworks Inc, 2007. [11] Jesús O D,Hagan M T.Back propagation algorithms for a broad class of dynamic networks[J].IEEE Transactions on Neural Networks,2007,18(1):14-27. doi: 10.1109/TNN.2006.882371 [12] Williams R J, Zisper D. A learning algorithm for continually runningfully recurrent neural networks[J].Neural Computation,1989,1(2):270-280. doi: 10.1162/neco.1989.1.2.270 [13] Ku C C, Lee K Y.Diagonal recurrent neural networks for dynamic systems control[J].IEEE Transaction on Neural Networks,1995,6(1):144-156. doi: 10.1109/72.363441 [14] XU Dong-po,LI Zheng-xue,WU Wei,et al.Convergence of gradient descent algorithm for diagonal recurrent neural networks[A]. In:CUI Guang-zhao,Ed.International Conference on Bio-Inspired Computing: Theories and Applications[C].Zhengzhou,China:IEEE Press,2007. [15] Kuan C M, Hornik K, White H.A convergence results for learning inrecurrent neural networks[J].Neural Computation,1994,6(3):420-440. doi: 10.1162/neco.1994.6.3.420 [16] WU Wei,FNG Guo-rui,LI Zheng-xue,et al.Convergence of an online gradient method for BP neural networks[J].IEEE Transactionson Neural Networks,2005,16(3):533-540. doi: 10.1109/TNN.2005.844903 [17] WU Wei,SHAO Hong-mei,QU Di. Strong convergence for gradient methods for BP networks training[A].In:ZHAO Ming-sheng,SHI Zhong-zhi,Eds.Proceedings of 2005 International Conference on Neural Networks and Brains[C].Beijing,China:IEEE Press,2005,332-334. [18] Gori M, Maggini M.Optimal convergence of on-line back propagation[J].IEEE Transaction on Neural Networks,1996,7(1):251-254. doi: 10.1109/72.478415 [19] Ortega J,Rheinboldt W.Iterative Solution of Nonlinear Equations in Several Variables[M].New York:Academic Press,1970. [20] 袁亚湘,孙文瑜.最优化理论与方法[M].北京:科学出版社,2001,149.
点击查看大图
计量
- 文章访问数: 2954
- HTML全文浏览量: 88
- PDF下载量: 813
- 被引次数: 0