Information Theory and Coding
Information Theory and Coding
Course URL :
18-753
Blackboard location.
Login with your Andrew ID and password, and search for
18-753.
Motivation :
Information theory underlies the mathematical theory of communication.
However, since its introduction several decades back, information theory
has been extended to several other fields, such as computer science,
economics, and statistics. The first half of this course will introduce
the basic notions of information theory, and relate it to primarily
communication system concepts, with some examples of its application in
other fields. Coding theory, which is the practical realization of the
communication limits specified by information theory, will be covered in
the second half of the course. However, a generalized treatment of
coding theory needs knowledge of finite field algebra, which will be
hard to cover in a half-semester. Therefore, the course will concentrate
on binary coding schemes, which nonetheless are very widely used. A
class project, involving independent reading, will allow students to
investigate any advanced topic related to information theory and coding.
This course may be of relevance to non-ECE students too.
Course description :
The first half of the course comprises of the concepts of entropy,
mutual information, the Asymptotic Equipartition property, applications
to source coding (data compression), applications to channel capacity
(channel coding), differential entropy and its application to waveform
channel capacities, and a subset of advanced topics such as Kolmogorov
complexity, timing (covert) communications, or rate-distortion theory,
as time permits.
The second half of the course comprises Hamming codes, cyclic codes (CRC
and BCH codes), a brief introduction to Reed-Solomon codes, and perhaps
universal codes (Lempel-Ziv coding). Students will be encouraged to
choose non-traditional applications of information theory or coding for
the course research project.
Pre-requisites :
At least senior level standing with 36-217, or permission of instructor.
Advisory Note: While the course material is fun, and while the emphasis
will primarily be on developing intuition, there will be a fair amount
of (undergraduate level) mathematics required, which you should either
know, or should be willing to pick up.