Modern Theory of Detection and Estimation

Modern Theory of Detection and Estimation
Course Number: 214 - 14994
ECTS credits: 6
YEAR 3/ UPPER Division

PREREQUISITES/STUDENTS ARE EXPECTED TO HAVE COMPLETED:
Statistics, Calculus II, Systems & Circuits

COMPETENCES AND SKILLS THAT WILL BE ACQUIRED AND LEARNING RESULTS:

After this course students will understand the principles of estimation and decision problems, and will become familiarized with the fundamental differences between the analytic and machine approaches that can be followed to solve them.  Students will understand that, for the correct understanding of these problems, it is necessary to master three basic probability theory elements: 1) the likelihood, 2) the difference between a priori and a posteriori uncertainty, and 3) Bayes' Theorem.  They will also understand the concepts of generalization and sufficient stastistics, as well as the bias vs variance tradeoff.  Finally, it will become apparent the advantages (both analytical and computational) inherent to Gaussian problems and linear solutions. (PO a)

From a practical point of view, students will learn to identify the convenience of following an analytical or machine approach for concrete situations.  They will acquire the necessary knowledge to face an analytical resolution of a decision or estimation problem when complete statistical information is available, knowing also some semianalytical approaches for scenarios with partial information.  When no statistical information is available, they will know how to design a regression or classification model, using data sets for learning its parameters: splitting the available data into training, validation and test sets, and applying algorithms for model order selection and parameter adjustment.  Furthermore, different criteria for measuring the quality of deciders and estimators, as well as their generalization capabilities, will be introduced.  Finally, students will study how these tools for estimation and detection can be adapted to deal with temporal series, and to implement adaptive solutions. (PO b)

During the course, students will study the previous concepts from a theoretical point of view, and will also apply them for the resolution of several study cases in practical sessions. During these sessions, students' work will help them improve the following general skills:
* Ability to identify and understand particular estimation and decision problems, and to propose practical solutions taking into account the characteristics of such problems (availability of historic data, possible computational constraints, etc) (PO e)
* Ability to design the experiments for the evaluation of the implemented estimators and deciders. (PO b)
* Knowledge of a simulation and mathematical modeling software application, which is widely used in engineering (Matlab) (PO k)

DESCRIPTION OF CONTENTS:

Block 0 - Introduction to Statistical and Machine Learning
   0.1. Estimation and classification concepts
   0.2. Examples of application of estimators and classifiers
   0.3. Analytical, semianalytical and machine methods
   0.4. Previous knowledge

Block 1 - Analytical and Machine Estimation
   1.1. General view of the estimation problem: Analytical and Machine approaches
   1.2. Design of estimators under an analytical approach
      *ML estimation of deterministic parameters
      * Bayesian Estimation Theory.  Cost functions.  MSE, ML and MAP estimation.  Gaussian case.
      * Minimum Mean Square Error linear estimator
      * Bias and Variance of estimators
   1.3. Design of estimators under a machine approach
      * Design of machine estimators: general approach
      * Least Squares Linear Regression
      * Semilinear regression

Block 2 - Analytical and Machine Decision
   2.1. General view of the decision problem: Analytical and Machine approaches
   2.2. Design of classifiers under an analytical approach
      * ML and MAP decision
      * Minimization of the expected cost: Optimal Bayesian decider
      * Binary classification.  LRT tests.  False Alarm, Miss, and Detection Probabilities.  Characteristic Curves (OC). Gaussian likelihoods.
   2.3. Design of classifiers under a machine approach
      * Train, validation and test data sets.  Generalization
      * Linear machine classifiers
      * Non-linear machine classifiers: semilinear models

Block 3 - Temporal Series Filtering
   3.1. Transversal scheme for linear filtering.  Frequent configuration setups
   3.2. Mean Square Error minimization: Wiener-Hopf equation, the Wiener Filter, Canonical shape of the error surface
   3.3. Adaptive filtering.  Steepest Descent algorithm.  Stochastic approximations: the LMS filter

LEARNING ACTIVITES AND METHODOLOGY:

THEORY (3 ECTS)
Theory sessions consist of lectures in which the basic concepts of the course will be introduced, illustrating them with a large number of examples (POs a and e)

PROBLEMS (1.5 ECTS)
Exercises and problems similar to those to be proposed in the exam will be solved.  Students will have problem statements available at the beginning of the course, so that they can work on them before they are solved in class. (POs a and e)

PRACTICAL SESSIONS (1.5 ECTS)
Sessions in which students will apply the concepts presented in the course with the help of a computer.  Students will deal with estimation and classification problems with real data, and will have to evaluate the performance of the implemented systems (PO b). During these practical sessions students will use Matlab as the simulation tool. (PO k)

ASSESSMENT SYSTEM:

The final mark of the course will be obtained by following the principles of continuous assessment:

- Exercises and theory questions to be solved by the students by taking an intermediate exam (30% of the course mark);
- During some of the practical sessions, students we be presented short estimation and decision/classification problems to be solved with the help of Matlab: 20% of the course mark;
- Final exam consisting of a theory part with questions and short exercises, followed by several problems: 50% of the course mark.

Students who do not follow the continuous evaluation procedure, will be assessed according to the general rules established by the University.

BIBLIOGRAPHY:
    C. M. Bishop. Neural Networks for Pattern Recognition. Oxford University Press, Oxford (United Kingdom). 1995
    C.M. Bishop. Pattern Recognition and Machine Learning. New York, NY: Springer. 2006
    H. L. Van Trees. Detection, Estimation, and Modulation Theory (vol. I). New York, NY: Wiley. 1968
    R.O. Duda, P.E. Hart, D.G. Stork. Pattern Classification. New York, NY: Wiley. 2001