Information Theory and Coding

Subject description

Introduction: definition of information, code, coding and communication system.
Entropy: discrete random systems, entropy of a discrete random variable, entropy of a continuous random variable.
Information: average information, mutual information of two discrete variables, mutual information of two continuous variables.
Discrete information sources: entropy of a stationary source, ergodic stationary sources, memoryless sources, sources with memory, and source redundancy.
Information source coding: fixed-length and variable-length coding, Kraft – McMillan inequality, Huffman code, arithmetic code, LZW code.
Secrecy coding: cryptosystems with a secret key, DES and AES cryptosystems, cryptosystems with a public key, RSA cryptosystem, digital signature.
Communication channels I: continuous communication channels, the capacity of a continuous communication channel, discrete communication channels, the capacity of a discrete communication channel.
Communication channels II: error-detecting codes, error-correcting codes, optimal decoding, Shannon theorem of secure coding, inversion of Shannon theorem.
Channel coding: linear block codes, cyclic codes, Hamming codes, Golay codes, the Reed-Muller codes, convolutional codes, the Viterbi algorithm, Turbo codes.

The subject is taught in programs

Objectives and competences

The objective of the course is to provide the student with the knowledge of the most important concepts and methods of information theory, information source coding, cryptography and communication-channel coding that form the basis for the design and development of communication interfaces that are essential components of intelligent systems in automation. The acquired knowledge forms the basis for the understanding and development of the technologies that involve obtaining and processing information from the environment.

Teaching and learning methods

The lectures provide a theoretical background of all the considered models and methods together with simple computational examples that illustrate the key characteristics of all the presented methods. A textbook and other study material, such as lecture notes with solved example problems and lecture slides, are available to the students. As part of the lectures, the students receive optional homework assignments including theoretical questions as well as computational exercises that enable the students to promptly verify the acquired knowledge. Practical work is carried out as part of the laboratory exercises, where students solve given programming problems. As part of the laboratory exercises, students also carry out additional elective projects within which the selected methods of information theory and coding should be implemented. The results of the elective projects are reported in written reports.

Expected study results

After successful completion of the course, students should be able to:

present the basic mathematical models describing the main phenomena in technical communication systems,
explain the measures of entropy, information and capacity of channels that are used for the quantitative descriptions of communication systems,
use encryption methods to ensure the security and confidentiality of communications,
use digital certificates to verify the identity of people and other information sources in different communication systems,
analyse the advantages and disadvantages of communication systems, where secure and confidential communication must be ensured,
build examples of communication systems that provide secure and confidential communication between information sources and receivers, and
to evaluate the level of security and confidentiality of communications in the given communication systems.

Basic sources and literature

N. Pavešić: Informacija in kodi, (2. izdaja), Založba FE in FRI, 2010.
V. Štruc, S. Dobrišek: Informacija in kodi : dopolnilni učbenik z vajami. 1. izd., Založba FE, 2016
T. M. Cover, J. A. Thomas: Elements of Information Theory, Wiley-Interscience, New York, 2006.
Roberto Togneri, Christopher J. S. deSilva: Fundamentals of information Theory and Coding Design, Chapman & Hall / CRC, 2002.