Section 1 of 3 - Prev - Next
All sections - 1 - 2 - 3

Archive-name: ai-faq/neural-nets/part7
Last-modified: 2002-10-14
URL: ftp://ftp.sas.com/pub/neural/FAQ7.html
Maintainer: saswss@unx.sas.com (Warren S. Sarle)

Copyright 1997, 1998, 1999, 2000, 2001, 2002 by Warren S. Sarle, Cary, NC,
USA. Answers provided by other authors as cited below are copyrighted by
those authors, who by submitting the answers for the FAQ give permission for
the answer to be reproduced as part of the FAQ in any of the ways specified
in part 1 of the FAQ. 

This is part 7 (of 7) of a monthly posting to the Usenet newsgroup
comp.ai.neural-nets. See the part 1 of this posting for full information
what it is all about.

========== Questions ========== 
********************************

Part 1: Introduction
Part 2: Learning
Part 3: Generalization
Part 4: Books, data, etc.
Part 5: Free software
Part 6: Commercial software
Part 7: Hardware and miscellaneous

   Neural Network hardware?
   What are some applications of NNs?
      General
      Agriculture
      Automotive
      Chemistry
      Criminology
      Face recognition
      Finance and economics
      Games, sports, gambling
      Industry
      Materials science
      Medicine
      Music
      Robotics
      Weather forecasting
      Weird
   What to do with missing/incomplete data?
   How to forecast time series (temporal sequences)?
   How to learn an inverse of a function?
   How to get invariant recognition of images under translation, rotation,
   etc.?
   How to recognize handwritten characters?
   What about pulsed or spiking NNs?
   What about Genetic Algorithms and Evolutionary Computation?
   What about Fuzzy Logic?
   Unanswered FAQs
   Other NN links?

------------------------------------------------------------------------

Subject: Neural Network hardware?
=================================

Overview articles: 

 o Clark S. Lindsey and Thomas Lindblad (1998), "Review of hardware neural
   networks: A user's perspective", 
   http://www.particle.kth.se/~lindsey/elba2html/elba2html.html 

 o P. D. Moerland and E. Fiesler (1997), "Neural Network Adaptations to
   Hardware Implementations", in Handbook of Neural Computation, 
   http://www.idiap.ch/~perry/moerland-97.1.bib.abs.html 

The journal, IEEE Transactions on Neural Networks, plans to have a
special issue on neural networks hardware implementations in September,
2003. 

Various NN hardware information can be found at the following web sites: 

 o Pacific Northwest National Laboratory:
   http://www.emsl.pnl.gov:2080/proj/neuron/neural/systems/commercial.html 
 o Dr. Denise Gorse, University College London:
   http://www.cs.ucl.ac.uk/staff/D.Gorse/research/pRAM.html 
 o Neural Chips and Evolvable Hardware:
   http://glendhu.com/ai/neuralchips/ 

------------------------------------------------------------------------

Subject: What are some applications of NNs?
===========================================

There are vast numbers of published neural network applications. If you
don't find something from your field of interest below, try a web search.
Here are some useful search engines:
http://www.google.com/
http://search.yahoo.com/
http://www.altavista.com/
http://www.deja.com/

General
-------

 o The Pacific Northwest National Laboratory: 
   http://www.emsl.pnl.gov:2080/proj/neuron/neural/ including a list of
   commercial applications at 
   http://www.emsl.pnl.gov:2080/proj/neuron/neural/products/ 
 o The Stimulation Initiative for European Neural Applications: 
   http://www.mbfys.kun.nl/snn/siena/cases/ 
 o The DTI NeuroComputing Web's Applications Portfolio: 
   http://www.globalweb.co.uk/nctt/portfolo/ 
 o The Applications Corner, NeuroDimension, Inc.: 
   http://www.nd.com/appcornr/purpose.htm 
 o The BioComp Systems, Inc. Solutions page: http://www.bio-comp.com 
 o Chen, C.H., ed. (1996) Fuzzy Logic and Neural Network Handbook, NY:
   McGraw-Hill, ISBN 0-07-011189-8. 
 o The series Advances in Neural Information Processing Systems containing
   proceedings of the conference of the same name, published yearly by
   Morgan Kauffman starting in 1989 and by The MIT Press in 1995. 

Agriculture
-----------

 o P.H. Heinemann, Automated Grading of Produce: 
   http://server.age.psu.edu/dept/fac/Heinemann/phhdocs/visionres.html
 o Deck, S., C.T. Morrow, P.H. Heinemann, and H.J. Sommer, III. 1995.
   Comparison of a neural network and traditional classifier for machine
   vision inspection. Applied Engineering in Agriculture. 11(2):319-326. 
 o Tao, Y., P.H. Heinemann, Z. Varghese, C.T. Morrow, and H.J. Sommer III.
   1995. Machine vision for color inspection of potatoes and apples.
   Transactions of the American Society of Agricultural Engineers.
   38(5):1555-1561. 

Automotive
----------

 o "No Hands Across America Journal" - steering a car: 
   http://cart.frc.ri.cmu.edu/users/hpm/project.archive/reference.file/Journal.html
   Photos: 
   http://www.techfak.uni-bielefeld.de/ags/ti/personen/zhang/seminar/intelligente-autos/tour.html

Chemistry
---------

 o PNNL, General Applications of Neural Networks in Chemistry and Chemical
   Engineering: 
   http://www.emsl.pnl.gov:2080/proj/neuron/neural/bib/chemistry.html. 
 o Prof. Dr. Johann Gasteiger, Neural Networks and Genetic Algorithms in
   Chemistry: 
   http://www2.ccc.uni-erlangen.de/publications/publ_topics/publ_topics-12.html
 o Roy Goodacre, pyrolysis mass spectrometry: 
   http://gepasi.dbs.aber.ac.uk/roy/pymshome.htm and Fourier transform
   infrared (FT-IR) spectroscopy: 
   http://gepasi.dbs.aber.ac.uk/roy/ftir/ftirhome.htm contain applications
   of a variety of NNs as well as PLS (partial least squares) and other
   statistical methods. 
 o Situs, a program package for the docking of protein crystal structures to
   single-molecule, low-resolution maps from electron microscopy or small
   angle X-ray scattering: http://chemcca10.ucsd.edu/~situs/ 
 o An on-line application of a Kohonen network with a 2-dimensional output
   layer for prediction of protein secondary structure percentages from UV
   circular dichroism spectra: http://www.embl-heidelberg.de/~andrade/k2d/. 

Criminology
-----------

 o Computer Aided Tracking and Characterization of Homicides and Sexual
   Assaults (CATCH): 
   http://lancair.emsl.pnl.gov:2080/proj/neuron/papers/kangas.spie99.abs.html

Face recognition
----------------

 o Face Recognition Home Page: http://www.cs.rug.nl/~peterkr/FACE/face.html 
 o Konen, W., "Neural information processing in real-world face-recognition
   applications," 
   http://www.computer.muni.cz/pubs/expert/1996/trends/x4004/konen.htm 
 o Jiang, Q., "Principal Component Analysis and Neural Network Based Face
   Recognition," http://people.cs.uchicago.edu/~qingj/ThesisHtml/ 
 o Lawrence, S., Giles, C.L., Tsoi, A.C., Back, A.D. (1997), "Face
   Recognition: A Convolutional Neural Network Approach," IEEE Transactions
   on Neural Networks, 8, 98-113, 
   http://www.neci.nec.com/~lawrence/papers/face-tnn97/latex.html 

Finance and economics
---------------------

 o Athanasios Episcopos, References on Neural Net Applications to Finance
   and Economics: http://www.compulink.gr/users/episcopo/neurofin.html 
 o Franco Busetti, Heuristics and artificial intelligence in finance and
   investment: http://www.geocities.com/francorbusetti/ 
 o Trippi, R.R. & Turban, E. (1993), Neural Networks in Finance and
   Investing, Chicago: Probus. 
 o Zirilli, J.S. (1996), Financial Prediction Using Neural Networks,
   International Thomson Publishing, ISBN 1850322341, 
   http://www6.bcity.com/mjfutures/ 
 o Andreas S. Weigend, Yaser Abu-Mostafa, A. Paul N. Refenes (eds.) (1997) 
   Decision Technologies for Financial Engineering: Proceedings of the Fourth
   International Conference on Neural Networks in the Capital Markets (Nncm
   '96) Publisher: World Scientific Publishing Company, ISBN: 9810231245 

Games, sports, gambling
-----------------------

 o General:

   Jay Scott, Machine Learning in Games: 
   http://satirist.org/learn-game/index.html

   METAGAME Game-Playing Workbench: 
   ftp://ftp.cl.cam.ac.uk/users/bdp/METAGAME

   R.S. Sutton, "Learning to predict by the methods of temporal
   differences", Machine Learning 3, p. 9-44 (1988). 

   David E. Moriarty and Risto Miikkulainen (1994). "Evolving Neural
   Networks to Focus Minimax Search," In Proceedings of Twelfth National
   Conference on Artificial Intelligence (AAAI-94, Seattle, WA), 1371-1377.
   Cambridge, MA: MIT Press, 
   http://www.cs.utexas.edu/users/nn/pages/publications/neuro-evolution.html

   Games World '99 at http://gamesworld99.free.fr/menuframe.htm

 o Backgammon:

   G. Tesauro and T.J. Sejnowski (1989), "A Parallel Network that learns to
   play Backgammon," Artificial Intelligence, vol 39, pp. 357-390. 

   G. Tesauro and T.J. Sejnowski (1990), "Neurogammon: A Neural Network
   Backgammon Program," IJCNN Proceedings, vol 3, pp. 33-39, 1990. 

   G. Tesauro (1995), "Temporal Difference Learning and TD-Gammon,"
   Communications of the ACM, 38, 58-68, 
   http://www.research.ibm.com/massive/tdl.html 

   Pollack, J.P. and Blair, A.D. (1997), "Co-Evolution in the Successful
   Learning of Backgammon Strategy," Brandeis University Computer Science
   Technical Report CS-97-193, 
   http://www.demo.cs.brandeis.edu/papers/long.html#hcgam97

 o Bridge:

   METAGAME: ftp://ftp.cl.cam.ac.uk/users/bdp/bridge.ps.Z

   He Yo, Zhen Xianjun, Ye Yizheng, Li Zhongrong (19??), "Knowledge
   acquisition and reasoning based on neural networks - the research of a
   bridge bidding system," INNC '90, Paris, vol 1, pp. 416-423. 

   M. Kohle and F. Schonbauer (19??), "Experience gained with a neural
   network that learns to play bridge," Proc. of the 5th Austrian Artificial
   Intelligence meeting, pp. 224-229. 

 o Checkers/Draughts: 

   Mark Lynch (1997), "NeuroDraughts: an application of temporal difference
   learning to draughts," 
   http://www.ai.univie.ac.at/~juffi/lig/Papers/lynch-thesis.ps.gz Software
   available at 
   http://satirist.org/learn-game/archive/NeuroDraughts-1.00.zip

   K. Chellapilla and D. B. Fogel, "Co-Evolving Checkers Playing Programs
   using Only Win, Lose, or Draw," SPIE's AeroSense'99: Applications and
   Science of Computational Intelligence II, Apr. 5-9, 1999, Orlando,
   Florida, USA, http://vision.ucsd.edu/~kchellap/Publications.html 

   David Fogel (1999), Evolutionary Computation: Toward a New Philosophy
   of Machine Intelligence (2nd edition), IEEE, ISBN: 078035379X 

   David Fogel (2001), Blondie24: Playing at the Edge of AI, Morgan Kaufmann
   Publishers, ISBN: 1558607838
   According to the publisher, this is: 

   ... the first book to bring together the most advanced work in the
   general use of evolutionary computation for creative results. It is
   well suited for the general computer science audience. 

   Here's the story of a computer that taught itself to play checkers
   far better than its creators ever could. Blondie24 uses a program
   that emulates the basic principles of Darwin evolution to discover on
   its own how to excel at the game. Through this entertaining story,
   the book provides the reader some of the history of AI and explores
   its future. 

   Unlike Deep Blue, the celebrated chess machine that beat Garry
   Kasparov, the former world champion chess player, this evolutionary
   program didn't have access to other games played by human grand
   masters, or databases of moves for the endgame. It created its own
   means for evaluating the patterns of pieces that it experienced by
   evolving artificial neural networks--mathematical models that loosely
   describe how a brain works. 

See http://www.natural-selection.com/NSIPublicationsOnline.htm for a variety
of online papers by Fogel. 

Not NNs, but classic papers:

A.L. Samuel (1959), "Some studies in machine learning using the game of
checkers," IBM journal of Research and Development, vol 3, nr. 3, pp.
210-229. 

A.L. Samuel (1967), "Some studies in machine learning using the game of
checkers 2 - recent progress," IBM journal of Research and Development, vol
11, nr. 6, pp. 601-616. 

o Chess:

Sebastian Thrun, NeuroChess: 
http://satirist.org/learn-game/systems/neurochess.html

Luke Pellen, Octavius: http://home.seol.net.au/luke/octavius/

Louis Savain (AKA Nemesis), Animal, a spiking neural network that the author
hopes will learn to play a passable game of chess after he implements the
motivation mechanism: 
http://home1.gte.net/res02khr/AI/Temporal_Intelligence.htm

o Dog racing:

H. Chen, P. Buntin Rinde, L. She, S. Sutjahjo, C. Sommer, D. Neely (1994),
"Expert Prediction, Symbolic Learning, and Neural Networks: An Experiment on
Greyhound Racing," IEEE Expert, December 1994, 21-27, 
http://ai.bpa.arizona.edu/papers/dog93/dog93.html

o Football (Soccer):

Kuonen Diego, "Statistical Models for Knock-out Soccer Tournaments", 
http://dmawww.epfl.ch/~kuonen/CALCIO/ (not neural nets, but relevant)

o Go:

David Stoutamire (19??), "Machine Learning, Game Play, and Go," Center for
Automation and Intelligent Systems Research TR 91-128, Case Western Reserve
University. http://www.stoutamire.com/david/publications.html 

David Stoutamire (1991), Machine Learning Applied to Go, M.S. thesis, Case
Western Reserve University, ftp://ftp.cl.cam.ac.uk/users/bdp/go.ps.Z 

Schraudolph, N., Dayan, P., Sejnowski, T. (1994), "Temporal Difference
Learning of Position Evaluation in the Game of Go," In: Neural Information
Processing Systems 6, Morgan Kaufmann 1994, 
ftp://bsdserver.ucsf.edu/Go/comp/td-go.ps.Z 

P. Donnelly, P. Corr & D. Crookes (1994), "Evolving Go Playing Strategy in
Neural Networks", AISB Workshop on Evolutionary Computing, Leeds, England, 
ftp://www.joy.ne.jp/welcome/igs/Go/computer/egpsnn.ps.Z or 
ftp://ftp.cs.cuhk.hk/pub/neuro/GO/techreports/egpsnn.ps.Z 

Markus Enzenberger (1996), "The Integration of A Priori Knowledge into a Go
Playing Neural Network," 
http://www.cgl.ucsf.edu/go/Programs/neurogo-html/neurogo.html 

Norman Richards, David Moriarty, and Risto Miikkulainen (1998), "Evolving
Neural Networks to Play Go," Applied Intelligence, 8, 85-96, 
http://www.cs.utexas.edu/users/nn/pages/publications/neuro-evolution.html 

Dahl, F. A. (1999), "Honte, a Go-playing program using neural nets", 
http://www.ai.univie.ac.at/icml-99-ws-games/papers/dahl.ps.gz 

o Go-Moku:

Freisleben, B., "Teaching a Neural Network to Play GO-MOKU," in I.
Aleksander and J. Taylor, eds, Artificial Neural Networks 2, Proc. of
ICANN-92, Brighton UK, vol. 2, pp. 1659-1662, Elsevier Science Publishers,
1992 

Katz, W.T. and Pham, S.P. "Experience-Based Learning Experiments using
Go-moku", Proc. of the 1991 IEEE International Conference on Systems, Man,
and Cybernetics, 2: 1405-1410, October 1991. 

o Olympics:

E.M.Condon, B.L.Golden, E.A.Wasil (1999), "Predicting the success of nations
at the Summer Olympics using neural networks", Computers & Operations
Research, 26, 1243-1265. 

o Pong:

http:// www.engin.umd.umich.edu/~watta/MM/pong/pong5.html 

o Reversi/Othello:

David E. Moriarty and Risto Miikkulainen (1995). Discovering Complex Othello
Strategies through Evolutionary Neural Networks. Connection Science, 7,
195-209, 
http://www.cs.utexas.edu/users/nn/pages/publications/neuro-evolution.html 

Yoshioka, T., Ishii, S., and Ito, M., Strategy acquisition for the game
``Othello'' based on reinforcement learning, IEICE Transactions on
Information and Systems E82-D 12, 1618-1626, 1999, 
http://mimi.aist-nara.ac.jp/~taku-y/ 

o Tic-Tac-Toe/Noughts and Crosses:

Fogel, David Bb (1993), "Using evolutionary programming to construct neural
networks that are capable of playing tic-tac-toe," Intern. Conf. on Neural
Networks 1993, IEEE, San Francisco, CA, pp. 875-880. 

Richard S. Sutton and Andrew G. Barto (1998), Reinforcement Learning: An
Introduction The MIT Press, ISBN: 0262193981, 
http://www-anw.cs.umass.edu/~rich/book/the-book.html 

Yongzheng Zhang, Chen Teng, Sitan Wei (2000), "Game playing with
Evolutionary Strategies and Modular Neural Networks: Tic-Tac-Toe," 
http://www.cs.dal.ca/~mheywood/GAPproject/EvolvingGamePlay.html 

Rob Ellison, "Neural Os and Xs," 
http://www.catfood.demon.co.uk/beta/game.html (An online Javascript demo,
but you may not live long enough to teach the network to play a mediocre
game. I'm not sure what kind of network it uses, but maybe you can figure
that out if you read the source.) 

http://listserv.ac.il/~dvorkind/TicTacToe/main_doc.htm, Java classes by Tsvi
Dvorkind, using reinforcement learning. 

Industry
--------

 o PNNL, Neural Network Applications in Manufacturing: 
   http://www.emsl.pnl.gov:2080/proj/neuron/neural/bib/manufacturing.html. 
 o PNNL, Applications in the Electric Power Industry: 
   http://www.emsl.pnl.gov:2080/proj/neuron/neural/bib/power.html. 
 o PNNL, Process Control: 
   http://www.emsl.pnl.gov:2080/proj/neuron/neural/bib/process.html. 
 o Raoul Tawel, Ken Marko, and Lee Feldkamp (1998), "Custom VLSI ASIC for
   Automotive Applications with Recurrent Networks", 
   http://www.jpl.nasa.gov/releases/98/ijcnn98.pdf 
 o Otsuka, Y. et al. "Neural Networks and Pattern Recognition of Blast
   Furnace Operation Data" Kobelco Technology Review, Oct. 1992, 12 
 o Otsuka, Y. et al. "Applications of Neural Network to Iron and Steel
   Making Processes" 2. International Conference on Fuzzy Logic and Neural
   Networks, Iizuka, 1992 
 o Staib, W.E. "Neural Network Control System for Electric Arc Furnaces"
   M.P.T. International, 2/1995, 58-61 
 o Portmann, N. et al. "Application of Neural Networks in Rolling
   Automation" Iron and Steel Engineer, Feb. 1995, 33-36 
 o Gorni, A.A. (2000), "The modelling of hot rolling processes using neural
   networks: A bibliographical review", 
   http://www.geocities.com/SiliconValley/5978/neural_1998.html 
 o Murat, M. E., and Rudman, A. J., 1992, Automated first arrival picking: A
   neural network approach: Geophysical Prospecting, 40, 587-604. 

Materials science
-----------------

 o Phase Transformations Research Group (search for "neural"): 
   http://www.msm.cam.ac.uk/phase-trans/pubs/ptpuball.html 

Medicine
--------

 o PNNL, Applications in Medicine and Health: 
   http://www.emsl.pnl.gov:2080/proj/neuron/neural/bib/medicine.html. 

Music
-----

 o Mozer, M. C. (1994), "Neural network music composition by prediction:
   Exploring the benefits of psychophysical constraints and multiscale
   processing," Connection Science, 6, 247-280, 
   http://www.cs.colorado.edu/~mozer/papers/music.html. 
 o Griffith, N., and Todd, P.M., eds. (1999), Musical Networks: Parallel
   Distributed Perception and Performance, Cambridge, MA: The MIT Press,
   ISBN 0-262-07181-9. 

Robotics
--------

 o Institute of Robotics and System Dynamics: 
   http://www.robotic.dlr.de/LEARNING/ 
 o UC Berkeley Robotics and Intelligent Machines Lab: 
   http://robotics.eecs.berkeley.edu/ 
 o Perth Robotics and Automation Laboratory: 
   http://telerobot.mech.uwa.edu.au/ 
 o University of New Hampshire Robot Lab: 
   http://www.ece.unh.edu/robots/rbt_home.htm 

Weather forecasting and atmospheric science
-------------------------------------------

 o UBC Climate Prediction Group: 
   http://www.ocgy.ubc.ca/projects/clim.pred/index.html 
 o Artificial Intelligence Research In Environmental Science: 
   http://www.salinas.net/~jpeak/airies/airies.html 
 o MET-AI, an mailing list for meteorologists and AI researchers: 
   http://www.comp.vuw.ac.nz/Research/met-ai 
 o Caren Marzban, Ph.D., Research Scientist, National Severe Storms
   Laboratory: http://www.nhn.ou.edu/~marzban/ 
 o David Myers's references on NNs in atmospheric science: 
   http://terra.msrc.sunysb.edu/~dmyers/ai_refs 

Weird
-----

Zaknich, Anthony and Baker, Sue K. (1998), "A real-time system for the
characterisation of sheep feeding phases from acoustic signals of jaw
sounds," Australian Journal of Intelligent Information Processing Systems
(AJIIPS), Vol. 5, No. 2, Winter 1998. 

Abstract
This paper describes a four-channel real-time system for the detection and
measurement of sheep rumination and mastication time periods by the analysis
of jaw sounds transmitted through the skull. The system is implemented using
an 80486 personal computer, a proprietary data acquisition card (PC-126) and
a custom made variable gain preamplifier and bandpass filter module. Chewing
sounds are transduced and transmitted to the system using radio microphones
attached to the top of the sheep heads. The system's main functions are to
detect and estimate rumination and mastication time periods, to estimate the
number of chews during the rumination and mastication periods, and to
provide estimates of the number of boli in the rumination sequences and the
number of chews per bolus. The individual chews are identified using a
special energy threshold detector. The rumination and mastication time
periods are determined by neural network classifier using a combination of
time and frequency domain features extracted from successive 10 second
acoustic signal blocks. 

------------------------------------------------------------------------

Subject: What to do with missing/incomplete data? 
==================================================

The problem of missing data is very complex. 

For unsupervised learning, conventional statistical methods for missing data
are often appropriate (Little and Rubin, 1987; Schafer, 1997; Schafer and
Olsen, 1998). There is a concise introduction to these methods in the
University of Texas statistics FAQ at 
http://www.utexas.edu/cc/faqs/stat/general/gen25.html. 

For supervised learning, the considerations are somewhat different, as
discussed by Sarle (1998). The statistical literature on missing data deals
almost exclusively with training rather than prediction (e.g., Little,
1992). For example, if you have only a small proportion of cases with
missing data, you can simply throw those cases out for purposes of training;
if you want to make predictions for cases with missing inputs, you don't
have the option of throwing those cases out! In theory, Bayesian methods
take care of everything, but a full Bayesian analysis is practical only with
special models (such as multivariate normal distributions) or small sample
sizes. The neural net literature contains a few good papers that cover
prediction with missing inputs (e.g., Ghahramani and Jordan, 1997; Tresp,
Neuneier, and Ahmad 1995), but much research remains to be done. 

References: 

   Donner, A. (1982), "The relative effectiveness of procedures commonly
   used in multiple regression analysis for dealing with missing values,"
   American Statistician, 36, 378-381. 

   Ghahramani, Z. and Jordan, M.I. (1994), "Supervised learning from
   incomplete data via an EM approach," in Cowan, J.D., Tesauro, G., and
   Alspector, J. (eds.) Advances in Neural Information Processing Systems
   6, San Mateo, CA: Morgan Kaufman, pp. 120-127. 

   Ghahramani, Z. and Jordan, M.I. (1997), "Mixture models for Learning from
   incomplete data," in Greiner, R., Petsche, T., and Hanson, S.J. (eds.) 
   Computational Learning Theory and Natural Learning Systems, Volume IV:
   Making Learning Systems Practical, Cambridge, MA: The MIT Press, pp.
   67-85. 

   Jones, M.P. (1996), "Indicator and stratification methods for missing
   explanatory variables in multiple linear regression," J. of the American
   Statistical Association, 91, 222-230. 

   Little, R.J.A. (1992), "Regression with missing X's: A review," J. of the
   American Statistical Association, 87, 1227-1237. 

   Little, R.J.A. and Rubin, D.B. (1987), Statistical Analysis with Missing
   Data, NY: Wiley. 

   McLachlan, G.J. (1992) Discriminant Analysis and Statistical Pattern
   Recognition, Wiley. 

   Sarle, W.S. (1998), "Prediction with Missing Inputs," in Wang, P.P.
   (ed.), JCIS '98 Proceedings, Vol II, Research Triangle Park, NC, 399-402,
   ftp://ftp.sas.com/pub/neural/JCIS98.ps. 

   Schafer, J.L. (1997), Analysis of Incomplete Multivariate Data, London:
   Chapman & Hall, ISBN 0 412 04061 1. 

   Schafer, J.L., and Olsen, M.K. (1998), "Multiple imputation for
   multivariate missing-data problems: A data analyst's perspective," 
   http://www.stat.psu.edu/~jls/mbr.pdf or 
   http://www.stat.psu.edu/~jls/mbr.ps 

   Tresp, V., Ahmad, S. and Neuneier, R., (1994), "Training neural networks
   with deficient data", in Cowan, J.D., Tesauro, G., and Alspector, J.
   (eds.) Advances in Neural Information Processing Systems 6, San Mateo,
   CA: Morgan Kaufman, pp. 128-135. 

   Tresp, V., Neuneier, R., and Ahmad, S. (1995), "Efficient methods for
   dealing with missing data in supervised learning", in Tesauro, G.,
   Touretzky, D.S., and Leen, T.K. (eds.) Advances in Neural Information
   Processing Systems 7, Cambridge, MA: The MIT Press, pp. 689-696. 

------------------------------------------------------------------------

Subject: How to forecast time series (temporal sequences)?
==========================================================

In most of this FAQ, it is assumed that the training cases are statistically
independent. That is, the training cases consist of pairs of input and
target vectors, (X_i,Y_i), i=1,...,N, such that the conditional
distribution of Y_i given all the other training data, (X_j,
j=1,...,N, and Y_j, j=1,...i-1,i+1,...N) is equal to the
conditional distribution of Y_i given X_i regardless of the values in the
other training cases. Independence of cases is often achieved by random
sampling. 

The most common violation of the independence assumption occurs when cases
are observed in a certain order relating to time or space. That is, case 
(X_i,Y_i) corresponds to time T_i, with T_1 < T_2 < ... <
T_N. It is assumed that the current target Y_i may depend not only on 
X_i but also on (X_i,Y_i) in the recent past. If the T_i are equally
spaced, the simplest way to deal with this dependence is to include
additional inputs (called lagged variables, shift registers, or a tapped
delay line) in the network. Thus, for target Y_i, the inputs may include 
X_i, Y_{i-1}, X_{i-1}, Y_{i-1}, X_{i-2}, etc. (In some
situations, X_i would not be known at the time you are trying to forecast 
Y_i and would therefore be excluded from the inputs.) Then you can train
an ordinary feedforward network with these targets and lagged variables. The
use of lagged variables has been extensively studied in the statistical and
econometric literature (Judge, Griffiths, Hill, Lütkepohl and Lee, 1985). A
network in which the only inputs are lagged target values is called an
"autoregressive model." The input space that includes all of the lagged
variables is called the "embedding space." 

If the T_i are not equally spaced, everything gets much more complicated.
One approach is to use a smoothing technique to interpolate points at
equally spaced intervals, and then use the interpolated values for training
instead of the original data. 

Use of lagged variables increases the number of decisions that must be made
during training, since you must consider which lags to include in the
network, as well as which input variables, how many hidden units, etc.
Neural network researchers have therefore attempted to use partially
recurrent networks instead of feedforward networks with lags (Weigend and
Gershenfeld, 1994). Recurrent networks store information about past values
in the network itself. There are many different kinds of recurrent
architectures (Hertz, Krogh, and Palmer 1991; Mozer, 1994; Horne and Giles,
1995; Kremer, 199?). For example, in time-delay neural networks (Lang,
Waibel, and Hinton 1990), the outputs for predicting target Y_{i-1} are
used as inputs when processing target Y_i. Jordan networks (Jordan, 1986)
are similar to time-delay neural networks except that the feedback is an
exponential smooth of the sequence of output values. In Elman networks
(Elman, 1990), the hidden unit activations that occur when processing target
Y_{i-1} are used as inputs when processing target Y_i. 

However, there are some problems that cannot be dealt with via recurrent
networks alone. For example, many time series exhibit trend, meaning that
the target values tend to go up over time, or that the target values tend to
go down over time. For example, stock prices and many other financial
variables usually go up. If today's price is higher than all previous
prices, and you try to forecast tomorrow's price using today's price as a
lagged input, you are extrapolating, and extrapolating is unreliable. The
simplest methods for handling trend are: 

 o First fit a linear regression predicting the target values from the time,
   Y_i = a + b T_i + noise, where a and b are regression
   weights. Compute residuals R_i = Y_i - (a + b T_i). Then
   train the network using R_i for the target and lagged values. This
   method is rather crude but may work for deterministic linear trends. Of
   course, for nonlinear trends, you would need to fit a nonlinear
   regression. 

 o Instead of using Y_i as a target, use D_i = Y_i - Y_{i-1} for
   the target and lagged values. This is called differencing and is the
   standard statistical method for handling nondeterministic (stochastic)
   trends. Sometimes it is necessary to compute differences of differences. 

For an elementary discussion of trend and various other practical problems
in forecasting time series with NNs, such as seasonality, see Masters
(1993). For a more advanced discussion of NN forecasting of economic series,
see Moody (1998). 

There are several different ways to compute forecasts. For simplicity, let's
assume you have a simple time series, Y_1, ..., Y_99, you want to
forecast future values Y_f for f > 99, and you decide to use three
lagged values as inputs. The possibilities include: 

Single-step, one-step-ahead, or open-loop forecasting: 
   Train a network with target Y_i and inputs Y_{i-1}, Y_{i-2},
   and Y_{i-3}. Let the scalar function computed by the network be
   designated as Net(.,.,.) taking the three input values as arguments
   and returning the output (predicted) value. Then:
   forecast Y_100 as Net(Y_99,Y_98,Y_97)
   forecast Y_101 as Net(Y_100,Y_99,Y_98)
   forecast Y_102 as Net(Y_101,Y_100,Y_99)
   forecast Y_103 as Net(Y_102,Y_101,Y_100)
   forecast Y_104 as Net(Y_103,Y_102,Y_101)
   and so on. 

Multi-step or closed-loop forecasting: 
   Train the network as above, but:
   forecast Y_100 as P_100 = Net(Y_99,Y_98,Y_97)
   forecast Y_101 as P_101 = Net(P_100,Y_99,Y_98)
   forecast Y_102 as P_102 = Net(P_101,P_100,Y_99)
   forecast Y_103 as P_103 = Net(P_102,P_101,P_100)
   forecast Y_104 as P_104 = Net(P_103,P_102,P_101)
   and so on. 

N-step-ahead forecasting: 
   For, say, N=3, train the network as above, but:
   compute P_100 = Net(Y_99,Y_98,Y_97)
   compute P_101 = Net(P_100,Y_99,Y_98)
   forecast Y_102 as P_102 = Net(P_101,P_100,Y_99)

Section 1 of 3 - Prev - Next
All sections - 1 - 2 - 3

© allanswers.org | Terms of use