Information Theory and Source Coding

Probability, random variables (trials, events, definition of probability, probability density function, mean values, central limit theorem). Stochastic processes (sample function, time averages, ergodicity, power density spectrum). Information (metrics, information sources, entropy, redundancy). Coding and data compaction (source coding theorem, entropy coding, Lempel-Ziv coding). Mutual information and channel capacity (mutual information, information channel, joint entropy of discrete sources, differential entropy, information capacity theorem).

Analogue signal coding – basic formatting (ideal and flat toped sampling, reconstruction of continuous time signals, band-pass signal sampling; quantization, granular and overload noise, dynamic range). Audio signals (sound and hearing, properties of audio signal, perceptual properties of human hearing, frequency masking, redundancy and irrelevance, properties of speech, vocal tract modeling, speech redundancy).

Speech coding (non-linear quantization, A- law compression, predictive coding; scalar quantization (DPCM, ADPCM), vector quantization (CELP). Audio signal coding (standard coding formats: CD, DVD-audio, DSD; lossy compression, MP2, MP3, AAC)

Basic principles of information transmission and related backgrounds. Entropy as the basic measure of information. Source coding and basic data compaction algorithms. Fundamental limits of reliable communication over noisy channel. Properties of analogue signals that are important for coding schemes.

Distinction between redundancy and irrelevance.  Redundancy removal in advanced speech coding. Basic principles of perceptual coding of audio signals.

Lectures, tutorial, homeworks