Data Engineering

Data Engineering examines how software engineering practices are applied to the development of modern data pipeline solutions that drive data driven decisions and businesses. The module begins by exploring parallelism concepts which allow students to understand the benefits of building distributed data platforms. Data Engineering then moves into concepts of dealing with large sources of data, including distributed databases, data warehousing, and data lakes. With a thorough understanding of how distribution and large-scale data operates, the module moves to examining data streaming and transaction processing. Finally, the module ends by considering data pipeline solutions in the cloud and how these enable the delivery of data-to-data scientists.

Data Engineering blends the tools and methods of data management and processing with software engineering principles. The module will continue the experience provided in Software Engineering, so students can further experience working in agile development teams. The tools used in the module will enable students to build more sophisticated solutions that those in Software Engineering, focusing on technology that allows data to be managed and processed at scale.

Data Engineering continues the team-working and system development via a technology-stack approach of Software Engineering. Students are expected to feel comfortable applying the team-working techniques provided in Software Engineering. Data Engineering provides a capstone to the Software Engineering theme and in many regards the software development work students undertake in Computer Science. On completion of this module, students will have delivered at least two significant software solutions as members of a team.