Convergence of Gradient Method for Elman Networks

WU Wei; XU Dong-po; LI Zheng-xue

Article Contents

Article Navigation > Applied Mathematics and Mechanics > 2008 > 29(9): 1117-1123

WU Wei, XU Dong-po, LI Zheng-xue. Convergence of Gradient Method for Elman Networks[J]. Applied Mathematics and Mechanics, 2008, 29(9): 1117-1123.

Citation:

WU Wei, XU Dong-po, LI Zheng-xue. Convergence of Gradient Method for Elman Networks[J]. Applied Mathematics and Mechanics, 2008, 29(9): 1117-1123.

Citation:

WU Wei, XU Dong-po, LI Zheng-xue. Convergence of Gradient Method for Elman Networks[J]. Applied Mathematics and Mechanics, 2008, 29(9): 1117-1123.

PDF( 311 KB)

Convergence of Gradient Method for Elman Networks

Department of Applied Mathematics, Dalian University of Technology, Dalian, Liaoning 116024, P. R. China

Received Date: 2007-12-05
Rev Recd Date: 2008-07-19
Publish Date: 2008-09-15

Abstract

Abstract

The gradient method for training Elman networks with finite training sample set is considered. The monotonicity of the error function in the iteration is shown. A weak and a strong convergence results are proved, indicating that the gradient of the error function goes to zero and the weight sequence goes to a fixed point, respectively. A numerical example is given to support the theoretical findings.
- Elman network,
- gradient learning algorithm, convergence,
- monotonicity,

FullText(HTML)

References(20)

References

[1]	Elman J L. Finding structure in time[J].Cognitive Science,1990,14(2):179-211. doi: 10.1207/s15516709cog1402_1
[2]	Tsoi A C, Back A D.Locally recurrent globally feedforward networks:a critical review of architectures[J].IEEE Transactions on Neural Networks,1994,5(2):229-239. doi: 10.1109/72.279187
[3]	WANG De-liang,LIU Xiao-mei,Ahalt S C.On temporal generalization of simple recurrent networks[J].Neural Networks,1996,9(7):1099-1118. doi: 10.1016/0893-6080(96)00034-2
[4]	Kremer S C. On the computational power of Elman-style recurrent net works[J].IEEE Transactions on Neural Networks,1995,6(4):1000-1004. doi: 10.1109/72.392262
[5]	Pham D T,Liu X. Training of elman networks and dynamic system modeling[J].International Journal of Systems Science,1996,27(2):221-226. doi: 10.1080/00207729608929207
[6]	Cartling B.On the implicit acquisition of a context-free grammar by a simple recurrent neural network[J].Neurocomputing,2008,71(7/9):1527-1537. doi: 10.1016/j.neucom.2007.05.006
[7]	LI Xiang,CHEN Zeng-qiang,YUAN Zhu-zhi,et al.Generating chaos by an Elman network[J].IEEE Transactions on Circuits and Systems-Ⅰ,2001,48(9):1126-1131. doi: 10.1109/81.948441
[8]	Ekici S,Yildirim S,Poyraz M.A transmission line fault locator based on Elman recurrent networks[J].Applied Soft Computing,DOI: 10.1016/J.asoc.2008.04.011.
[9]	Neto L B, Coelho P H G,Soares de Mello J C C B,et al.Flow estimation using an Elman networks[A].In:Wunsch D,Ed.Proceedings of 2004 IEEE International Joint Conference on Neural Networks[C].Budapest, Hungary:IEEE Press,2004,831-836.
[10]	Demuth H B, Beale M H, Hagan M T.Neural Network Toolbox User'Sguide[M].atick, MA: The Mathworks Inc, 2007.
[11]	Jesús O D,Hagan M T.Back propagation algorithms for a broad class of dynamic networks[J].IEEE Transactions on Neural Networks,2007,18(1):14-27. doi: 10.1109/TNN.2006.882371
[12]	Williams R J, Zisper D. A learning algorithm for continually runningfully recurrent neural networks[J].Neural Computation,1989,1(2):270-280. doi: 10.1162/neco.1989.1.2.270
[13]	Ku C C, Lee K Y.Diagonal recurrent neural networks for dynamic systems control[J].IEEE Transaction on Neural Networks,1995,6(1):144-156. doi: 10.1109/72.363441
[14]	XU Dong-po,LI Zheng-xue,WU Wei,et al.Convergence of gradient descent algorithm for diagonal recurrent neural networks[A]. In:CUI Guang-zhao,Ed.International Conference on Bio-Inspired Computing: Theories and Applications[C].Zhengzhou,China:IEEE Press,2007.
[15]	Kuan C M, Hornik K, White H.A convergence results for learning inrecurrent neural networks[J].Neural Computation,1994,6(3):420-440. doi: 10.1162/neco.1994.6.3.420
[16]	WU Wei,FNG Guo-rui,LI Zheng-xue,et al.Convergence of an online gradient method for BP neural networks[J].IEEE Transactionson Neural Networks,2005,16(3):533-540. doi: 10.1109/TNN.2005.844903
[17]	WU Wei,SHAO Hong-mei,QU Di. Strong convergence for gradient methods for BP networks training[A].In:ZHAO Ming-sheng,SHI Zhong-zhi,Eds.Proceedings of 2005 International Conference on Neural Networks and Brains[C].Beijing,China:IEEE Press,2005,332-334.
[18]	Gori M, Maggini M.Optimal convergence of on-line back propagation[J].IEEE Transaction on Neural Networks,1996,7(1):251-254. doi: 10.1109/72.478415
[19]	Ortega J,Rheinboldt W.Iterative Solution of Nonlinear Equations in Several Variables[M].New York:Academic Press,1970.
[20]	袁亚湘,孙文瑜.最优化理论与方法[M].北京：科学出版社,2001,149.