home
publications
teaching
short CV
personal
the pub

Algorithms and Computation in Signal Processing or
How to Write Fast Code
18799B (CMU, ECE)
A new version of this course was taught in spring 2008 (website)
Basic Information
 Course number: 18799B
 Spring 2005, TR: 1:303:00pm, DH 1209
 Instructor: Markus Püschel (PH B16, pueschel at ece, 84259), TA: Srinivas Chellappa (PH B10, schellap at andrew, 87104), Admin: Carol Patterson (PH B15, carol at ece, 87286)
 12 units
 Office hours: Tuesdays 45pm, Thursdays after class til 4pm.
 Requirements: solid C programming skills, senior undergraduate or graduate student
Course Description
The fast evolution and increasing complexity of computing platforms pose a major challenge for developers of high quality software for signal processing and other applications: it becomes increasingly harder to harness the available computing power; conversely, straightforward implementations may loose as much as one or two orders of magnitude in performance. Creating optimal implementations requires the developer to have an understanding of algorithms, capabilities and limitations of compilers, and the target platform's microarchitecture. For these reasons, a recent trend in numerical computing is towards "selfadaptable" software to achieve optimal performance and portability at reduced coding effort.
This interdisciplinary course introduces the student to the foundations and stateoftheart techniques in high performance software development for signal processing and other numerical functionality including transforms, filters, and basic linear algebra algorithms. Topics include: 1) fundamental tools in algorithm theory and analysis; 2) fast signal processing and numerical algorithms; 3) how to write software that overcomes compiler limitations; 4) the role of the memory hierarchy and other microarchitectural features in software development; 5) how to use special instruction sets, such as SSE/MMX on Pentium; 6) an introduction to the concepts of selfadaptable software and software generators.
The course is targeted to signal processing students and CE/CS students who want to acquire a better understanding of algorithms and advanced software implementation techniques.
The students will be required to complete several implementation projects in C throughout the semester.
Topics Covered
 Foundations of algorithm analysis
 cost and complexity
 Ocalculus
 cost analysis and recurrences
 Computer architecture
 architecture and microarchitecture
 memory hierarchy including cache structure
 execution units
 special instruction sets (especially short vector instructions)
 Compilers (strengths, limitations, how to use)
 In detail: algorithms, complexity, and cuttingedge software or code generation for various numerical kernels
 Discrete Fourier transform, filters, other signal transforms (FFTW and SPIRAL)
 Linear algebra kernels (ATLAS)
 Higher linear algebra functionality (LAPACK)
 Sparse linear algebra (BeBOP)
 Other kernels
Goals of this Course
 Understand the connection between algorithms, implementation, and computer architecture
 Learn a guideline how to write fast numerical code and apply it in your research project
 Learn some fundamental numerical algorithms
 Learn how to analyze numerical algorithms
Textbook
There is no textbook for this class. The part that is foundation (algorithms, computer architecture etc.) will be compiled from several standard books. The core part, which analyzes cutting edge implementations for numerical problems is compiled from research papers and the instructor's own experience.
Grading
 50% research project
 Topic: Very fast, ideally adaptive implementation of (or code generation for) a numerical problem
 Team up in pairs (preferably)
 End of January/early February: Suggest to me a problem or I give you one
 Show "milestones" during semester
 Write 4 page standard conference paper (template will be provided)
 Give short presentation last week of April
 15% midterm
 Mostly about algorithm analysis
 Some multiple choice
 25% homework
 Exercises on algorithms analysis
 Implementation exercises. Purpose: study the effect of program optimizations, compilers, special instructions, etc. Tasks: writing and submitting C code & creating runtime/performance plots
 Some templates will be provided
 10% class participation
 It is important to attend (many things I teach cannot be found in books)
 I encourage you to ask questions
 I will provide some anonymous feedback mechanism
Final Exam
Homework
Midterm
Thursday, 03. March, 1:30pm.
Research Project
 Template for 4 page paper:
 Everybody reads this: conference.pdf
 For latex use: conferencelatex.tgz
 Creating bibliography: latex conference; bibtex conference; latex conference
 Creating a pdf: dvips t letter o conference.ps Ppdf G0 conference.dvi; ps2pdf conference.ps
 For Word (discouraged) use this: conferenceword.doc
 Presentation

 Timeline:
 first version of paper due on April 20th (contains everything except some, but not all, experimental results and optimizations)
 presentations last week of April (exact time will be decided April 12th)
 final version of paper due: one week after your presentation
 Projects:
 Biometrics registration and identification (Woon Ho Jung), final paper
 Software radio (Bryan Chen and Vijay Chandrasekhar), final paper
 LU factorization (John Cole), final paper
 ATLAS for embedded VLIW (Roland Wunderlich), final paper
 Shortest path problem (Sungchul Han and Sukchan Kang), final paper
 Feature set computation for biomedical imaging (Tad Merryman and Eizan Miyamoto), final paper
 Power optimization for signal transforms (Peter Milder and Marek Telgarsky), final paper
 Vector ATLAS (Joohoon Lee and Dongkeun Lee), final paper
Lectures (including pdfs, paper links may need CMU IP)
 1. Lecture (11. Jan.): Technicalities, overview and motivation (slides)
 2. Lecture (13. Jan.): Problem, algorithm, complexity, asymptotic runtime analysis of divideandconquer algorithms (slides, notes)
 3. Lecture (18. Jan.): Cost analysis, solving recurrences (slides, mynotes)
 4. Lecture (20. Jan.): Overview architecture and microarchitecture (slides, mynotes)
 5. Lecture (25. Jan.): Guide to benchmarking, overview LAPACK and BLAS, Matrixmatrix multiplication (MMM), complexity, algorithms (slides, mynotes)
 6. Lecture (27. Jan.): The ATLAS code generator: MMM (slides, mynotes, paper)
 7. Lecture (01. Feb.): Recitation (assignment 1)
 8. Lecture (03. Feb.): Modelbased ATLAS (slides, mynotes, paper)
 9. Lecture (08. Feb.): Dense and sparse matrixvector multiplication, Bebop/Sparsity (slides including link to relevant paper, mynotes)
 10. Lecture (10. Feb.): Signal transforms and algorithms: Overview (slides)
 11. Lecture (15. Feb.): Discrete Fourier transform (DFT), interpretations of the DFT, structured matrices, Kronecker product (mynotes)
 12. Lecture (17. Feb.): Structured matrices: various interpretations, CooleyTukey FFT (mynotes)
 13. Lecture (22. Feb.): GoodThomas, Rader, Bluestein FFT, complexity of the DFT, complex arithmetic (mynotes)
 14. Lecture (24. Feb.): Feedback 2nd assignment, adaptive FFT library FFTW (slides, mynotes)
 15. Lecture (01. Mar.): Cancelled. Oneonone meetings.
 16. Lecture (03. Mar.): Midterm.
 17. Lecture (15. Mar.): More on FFTW, the problem with strided data access (A tensor I) (slides, mynotes)
 18. Lecture (17. Mar.): Comparison ATLAS, Sparsity, FFTW, short guide to writing fast code, convolution (filtering), correlation (slides, mynotes, thesisfilterwavelet)
 19. Lecture (22. Mar., Srinivas Chellappa): Feedback midterm
 20. Lecture (24. Mar.): Feedback 3rd assignment (slides)
 21./22. Lecture (29./31. Mar., Franz Franchetti): SIMD vector instructions: state of the art and history; comparison to original SIMD, vector computers, VLIW and superscalar processors; programming interfaces, SSE with Intel C++ compiler; adapting algorithms to SIMD: MMM, WHT, DFT; BlueGene/L supercomputer; guide to writing fast vector code (slides1, slides2, notes)
 23. Lecture (05. Apr.): Cancelled. Oneonone meetings.
 24./25. Lecture (07./12 Apr.): Spiral: Code generation for DSP transforms (slides, mynotes)
 26. Lecture (14. Apr.): Cancelled. Spring carnival.
 27. Lecture (19. Apr.): Spiral demo, Gauss elimination and LU factorization (slides, mynotes)
 28. Lecture (21. Apr.): LU factorization (cont'd), matrix inversion, determinant, small guide to giving presentations (slides)
 29./31. Lecture (26./28. Apr.): Research project presentations (final project papers above)
