Big Data Storage and Processing Continuous Assessment 2026 | CCT College Dublin

Big Data Storage and Processing Continuous Assessment

Learning Outcomes

On successful completion of this module the learner will be able to:

1. Critically assess the data storage and management requirements of a given data project from a modern perspective and evaluate limitations of legacy approaches to Big Data. (Linked to PLO 3)
2. Assess the design concepts and architectural patterns of distributed Big Data systems and analyse the components that form their technology stack. (Linked to PLO 1, PLO 2)
3. Critically evaluate and select a Big Data environment suitable for retrieving and processing a given Big Data set, perform data management operations and select appropriate analytic algorithms for the required scale and speed. (Linked to PLO 2, PLO 3)
4. Assess the functional differences between common big data environments and the particularities

Assessment Task

Students are advised to review and adhere to the submission requirements documented after the assessment task.

It is Required that you use GitHub Classroom as your version control repository etc with regular commits of code and report versions. You may be called to a Viva to defend your work.

Please find the GitHub Classroom link below:

https://classroom.github.com/a/dVkmg3sO

You may not upload a PDF document for your report, It MUST be a word document. This is Not a Scientific Paper and must be formatted in standard report format including index, appendix, etc.

In this continuous assessment, You are required to identify and carry out an analysis of a large dataset using the energy AND the weather data Supplied. This data should be stored as requested below

Your project must incorporate the following elements:

• Utilisation of a distributed data processing environment (e.g., Hadoop Map-reduce or Spark), for some part of the analysis.
• Source dataset(s) can be stored into an appropriate SQL/ NoSQL database(s) prior to processing by MapReduce / Spark (HBase / HIVE / Spark SQL /Cassandra / MongoDB / etc.) The data can be populated into the NoSQL database using an appropriate tool (Hadoop/ Spark etc.)
• Post Map-reduce processing dataset(s) can be stored into an appropriate NoSQL database(s) (Follow a similar choice as in the previous step)
• Store the data and then follow-up analysis on the output data. It can be extracted from the NoSQL database into another format, using an appropriate tool, if necessary (e.g. extract to CSV to import into R/ Python etc.).
• Devise and implement a test strategy in order to perform a comparative analysis of the capabilities of any two databases (MySQL, MongoDB, Cassandra, HBase and CouchDB) in terms of the performance. You should record a set of appropriate metrics and perform a quantitative analysis for comparison purposes between the two chosen database systems.
• Provide evidence and justification of your choice of sentiment extraction techniques.

Deliverables

The results of the analysis must be presented in the form of a project report. This report should discuss the storage and processing of big data. The report should be 2500 ± 10% words in length (excluding references, titles, and code) and must follow the Harvard styles format in addition to employing appropriate referencing methods and academic writing style. The report should include the following:

Big Data

1. Details of the data storage and processing activities carried out, including preparation of the data and processing the data in a MapReduce/ Spark environment;[0-30]
2. Comparative analysis for at least two databases (one SQL and at least one NOSQL) using YCSB.[0-30]
3. A discussion of the rationale and justification for the choices you have made in terms of data processing and storage, programming language choice, that you have implemented.[0-20]
4. Design the architecture for the processing of big data using all the necessary technologies (HADOOP/SPARK,NOSQL/SQL databases and programming). Present your Design in the form of a diagram and discussion in your report .[0-20]

Note that MapReduce-style processing in this instance is considered to include platforms such as Apache Spark.

Submission

Submission Requirements All assessment submissions must meet the minimum requirements listed below. Failure to do so may have implications for the mark awarded.

All assessment submissions must:

• 2500 words +- 10% (excluding references, titles, citations and quotes)
• Word Document for report (No PDF’s), Jupyter notebook for code, Screencast for practical demonstration.
• Be submitted by the deadline date specified or be subject to late submission penalties
• Be submitted via Moodle upload
• Use Harvard Referencing when citing third party material
• Be the student’s own work.
• Include the CCT assessment cover page.

Continuous Assessment Rubric

Additional Information

• Lecturers are not required to review draft assessment submissions.
• In accordance with CCT policy, feedback to learners may be provided in written, audio or video format and can be provided as individual learner feedback, small group feedback or whole class feedback.
• Results and feedback will only be issued when assessments have been marked and moderated / reviewed by a second examiner.
• Additional feedback may be requested by contacting your lecturer AFTER the publication of results, Additional feedback may be provided as individual, small group or whole class feedback. Lecturers are not obliged to respond to email requests for additional feedback where this is not the specified process or to respond to further requests for feedback following the additional feedback.
• Following receipt of feedback, where a student believes there has been an error in the marks or feedback received, they should avail of the recheck and review process and should not attempt to get a revised mark / feedback by directly approaching the lecturer. Lecturers are not authorised to amend published marks outside of the recheck and review process or the Board of Examiners process.
• Students are advised that disagreement with an academic judgement is not grounds for review.
• For additional support with academic writing and referencing students are advised to contact the CCT Library Service or access the CCT Learning Space.
• For additional support with subject matter content students are advised to contact the CCT Student Mentoring Academy
• For additional support with IT subject content, students are advised to access the CCT Support Hub.