Data is pervasive these days and novel solutions critically depend on the ability of both scientific and business communities to derive insights from the data deluge. Experts in the area of big data analytics are more sought after than ever. However, the big data ecosystem is sprawling and convoluted.
At the same time, distributed systems are becoming more and more the norm for large applications since their requirements often exceed the capabilities of a single machine. Being aware of the advantages and pitfalls of distributed systems is key to mastering the move from small to large-scale projects. Finally, big data processing is one of the most important applications of functional programming and platforms are designed to run in a distributed setup.
This course wishes to prepare you for big data processing in a unique way. You will be introduced to the principles of functional programming, learn the particular challenges of distributed systems, and how big data processing systems use functional programming to respond to these challenges. Learn from experts in the field how to get the most out of existing popular platforms like Hadoop and Apache Spark using functional programming, how to write code that performs well, and how to address other common non-functional requirements like resilience and resource-constraints.
The course goes beyond the user level to delve into the architectural considerations that led to the development of these platforms, thereby unlocking their full potential for novel applications. This allows decision makers to lead their teams towards better solutions and practitioners to become leaders within their organizations in one of the fastest growing areas in ICT.
What You'll Learn:
- understanding the foundations of functional programming
- ability to apply them in practical applications
- knowledge and understanding of the foundations of distributed systems
- competence in analyzing and evaluating the implications on data processing
- familiarity with the architecture and properties of big data processing systems and the ability to make educated decisions on when to apply which system in a project
- ability to develop programs and solutions for big data analytics tasks
Module 1: Introduction to Lambda Calculus
- Video: Introduction to Lambda Calculus
- Definition and Applications of Scopes
- Practice: Scopes
- Video: Alpha Conversion
- Video: Church Booleans
- Practice: Beta Reduction
- Video: Summary Lambda Calculus
- Activity: Discussion and Reflection
Module 2: From Lambda Calculus to Functional Programming
- Introduction to Type Systems
- The Case for Types
- Video: Typed Arithmetic Expressions
- Typing Rules and Type Checking
- Function Types and Type Inferencing
- Discussion: Type Systems
- Functional Programming
- Activity: Introduction to SML
- Video: Pattern Matching and Recursion
- Lists and Higher-Order Functions
- Video: Summary Functional Programming
- Exercise: Hands-on Functional Programming
Module 3: Introduction to Distributed Systems
- Introduction to Distributed Systems
- Video: Introduction and Motivation
- What makes Distributed Systems hard?
- Quiz: Distributed Systems
- Video: The CAP Theorem
- The Importance of Parallelism
- Video: Summary Distributed Systems
- Activity: Discussion and Reflection
Module 4: Big Data Processing Systems
- Video: The Inertia of Big Data
- Distributed Storage: GFS and HDFS
- MapReduce and Hadoop
- Video: From Hadoop to Second Generation Systems Spark
- Video: Summary Big Data Processing Systems
- Exercise: Getting Started with Spark
Module 5: Programming for Big Data Analytics
- Programming for Big Data Analytics in Scala
- Demo 1
- Demo 2
If you successfully complete this course, you will earn a professional education certificate and you are eligible to receive 2.0 Continuing Education Units (CEUs).
This course is primarily geared towards working professionals.
Basic understanding of programming in mainstream programming languages like Java or Python. Basic understanding of computer systems.
If you have any questions about this course or the TU Delft online learning environment, please visit our Help & Support page.