Systems for processing large amounts of data

Subject description

Data collection: smart phones, sensors and internet-connected devices, web, cleaning and preparation of data, data anonymization and de-identification.

Data retention; scalable relational databases, NoSQL databases, understanding the compromise between the consistency of data, performance and availability.

Data processing: event-oriented processing, processing parallelization (map-reduce), extraction of structured data from unstructured.

Analyses: efficient algorithms for processing and analysis of data, machine learning

Visualization, procedures and challenges of visualizing large amounts of data, other modalities of presentation of data (soundification, etc.).

Applications of the presented techniques: systems for context detection, smart systems (applications of smart cities, smart transport, etc.), medical applications, social networks, financial systems

The subject is taught in programs

Objectives and competences

Is familiar with the concept of "big data". Able to evaluate the amount of data, the rate of events, their diversity, and the key challenges associated with large amounts of data.

Knows the difference and can choose among relational or NoSQL database, and evaluate the appropriateness of use.

Knows the strengths and weaknesses of map-reduce model and evaluates it in comparison with relational databases.

Can apply basic analytical and visualization techniques for working with large amounts of data in a use-case.

Teaching and learning methods

Lectures or mentoring

Seminar

Expected study results

Understanding the concept of "big data": data volume, events and their diversity, and key challenges associated with large amounts of data.

Understanding of relational databases, their capabilities and limitations.

Understanding the capabilities, strengths and weaknesses of NoSQL databases.

Understanding of map-reducer model, its strengths and weaknesses, as well as a comparison with relational databases.

Understanding of basic analytical and visualization techniques for working with large amounts of data.

Basic sources and literature

European Commission: http://www.internet-of-things-research.eu/pdf/Converging_Technologies_for_Smart_Environments_and_Integrated_Ecosystems_IERC_Book_Open_Access_2013.pdf
Tom White: Hadoop: The Definitive Guide, 3rd Edition; Storage and Analysis at Internet Scale; O'Reilly Media
Jure Leskovec, Anand Rajaraman, Jeﬀrey D. Ullman: Mining of Massive Datasets, http://i.stanford.edu/~ullman/mmds/book.pdf
Jimmy Lin, Chris Dyer: Data-Intensive Text Processing with MapReduce, http://lintool.github.io/MapReduceAlgorithms/MapReduce-book-final.pdf
Tamara Munzner: Visualization Analysis and Design (2014 Draft) http://www.cs.ubc.ca/~tmm/courses/533/book/vispmp-draft.pdf
Scott Murray: Interactive Data Visualization for the Web: An Introduction to Designing with D3, O'Reilly Media