Mapreduce algorithm for weather dataset

Weather forecasting plays a vital role in human daily routine, business and their decisions. The technology for weather forecasting is evolving rapidly due to the critical needs in obtaining the accurate prediction results. From the literature exploration, the researchers have found that weather dat...

Full description

Bibliographic Details
Main Author:	Khalid Adam, Ismail Hammad
Format:	Thesis
Language:	English English English
Published:	2017
Subjects:	T Technology (General)
Online Access:	http://umpir.ump.edu.my/id/eprint/19680/ http://umpir.ump.edu.my/id/eprint/19680/ http://umpir.ump.edu.my/id/eprint/19680/1/Mapreduce%20algorithm%20for%20weather%20dataset%20-Table%20of%20contents.pdf http://umpir.ump.edu.my/id/eprint/19680/2/Mapreduce%20algorithm%20for%20weather%20dataset%20-Abstract.pdf http://umpir.ump.edu.my/id/eprint/19680/3/Mapreduce%20algorithm%20for%20weather%20dataset%20-References.pdf

id	ump-19680
recordtype	eprints
spelling	ump-196802018-01-11T03:00:34Z http://umpir.ump.edu.my/id/eprint/19680/ Mapreduce algorithm for weather dataset Khalid Adam, Ismail Hammad T Technology (General) Weather forecasting plays a vital role in human daily routine, business and their decisions. The technology for weather forecasting is evolving rapidly due to the critical needs in obtaining the accurate prediction results. From the literature exploration, the researchers have found that weather data is important to be analysed in form of structure data. Most of data in weather is represented in unstructured data with different attributes such as temperature, humidity, visibility, and pressure. These data were captured by different types of sensors. The weather data consists of high volumes, high velocity and variety of data which is reflects to the characteristics of Big Data. In addition, these characteristics also contribute to the complexity on the data processing and prediction. Big Data analytics is a new concept to process the Big Data. For weather data, this new concept will help to organise the data into structure data. The well-known method for Big Data analytics is MapReduce Model. Nevertheless, the usage of MapReduce Model in processing weather dataset is not widely explored. Therefore, this research is focus on analysing the weather dataset using MapReduce Algorithm. The historical dataset in 10 years’ period (1997 to 2007) has been used and this dataset is obtained from NOAA. This original dataset is stored in Hadoop Distributed File System. Next, MapReduce Algorithm is developed using Java programming. The algorithm is tested using small and big dataset. The temperature, humidity and visibility attributes from the dataset has been extracted by the MapReduce Algorithm into structure data. Graphical analysis has been used to represent the result from the MapReduce Algorithm. Results from the proposed algorithm have been compared with the existing model known as AWK (Alfred Aho, Peter Weinberger, and Brian Kernighan) model. The purpose of the comparison is to investigate the capability of the proposed model in parallel processing. The comparison results shown that MapReduce Algorithm has produced 37%, 25% and 11% less compared to AWK in term of processing time for 10GB, 5GB and 1GB data, respectively. This result has revealed the significant impact to the used of MapReduce Algorithm in weather prediction. In addition, the MapReduce results have discovered the significant pattern of temperature, humidity and visibility information which is valuable for the weather prediction. 2017-05 Thesis NonPeerReviewed application/pdf en http://umpir.ump.edu.my/id/eprint/19680/1/Mapreduce%20algorithm%20for%20weather%20dataset%20-Table%20of%20contents.pdf application/pdf en http://umpir.ump.edu.my/id/eprint/19680/2/Mapreduce%20algorithm%20for%20weather%20dataset%20-Abstract.pdf application/pdf en http://umpir.ump.edu.my/id/eprint/19680/3/Mapreduce%20algorithm%20for%20weather%20dataset%20-References.pdf Khalid Adam, Ismail Hammad (2017) Mapreduce algorithm for weather dataset. Masters thesis, Universiti Malaysia Pahang. http://iportal.ump.edu.my/lib/item?id=chamo:101358&theme=UMP2
repository_type	Digital Repository
institution_category	Local University
institution	Universiti Malaysia Pahang
building	UMP Institutional Repository
collection	Online Access
language	English English English
topic	T Technology (General)
spellingShingle	T Technology (General) Khalid Adam, Ismail Hammad Mapreduce algorithm for weather dataset
description	Weather forecasting plays a vital role in human daily routine, business and their decisions. The technology for weather forecasting is evolving rapidly due to the critical needs in obtaining the accurate prediction results. From the literature exploration, the researchers have found that weather data is important to be analysed in form of structure data. Most of data in weather is represented in unstructured data with different attributes such as temperature, humidity, visibility, and pressure. These data were captured by different types of sensors. The weather data consists of high volumes, high velocity and variety of data which is reflects to the characteristics of Big Data. In addition, these characteristics also contribute to the complexity on the data processing and prediction. Big Data analytics is a new concept to process the Big Data. For weather data, this new concept will help to organise the data into structure data. The well-known method for Big Data analytics is MapReduce Model. Nevertheless, the usage of MapReduce Model in processing weather dataset is not widely explored. Therefore, this research is focus on analysing the weather dataset using MapReduce Algorithm. The historical dataset in 10 years’ period (1997 to 2007) has been used and this dataset is obtained from NOAA. This original dataset is stored in Hadoop Distributed File System. Next, MapReduce Algorithm is developed using Java programming. The algorithm is tested using small and big dataset. The temperature, humidity and visibility attributes from the dataset has been extracted by the MapReduce Algorithm into structure data. Graphical analysis has been used to represent the result from the MapReduce Algorithm. Results from the proposed algorithm have been compared with the existing model known as AWK (Alfred Aho, Peter Weinberger, and Brian Kernighan) model. The purpose of the comparison is to investigate the capability of the proposed model in parallel processing. The comparison results shown that MapReduce Algorithm has produced 37%, 25% and 11% less compared to AWK in term of processing time for 10GB, 5GB and 1GB data, respectively. This result has revealed the significant impact to the used of MapReduce Algorithm in weather prediction. In addition, the MapReduce results have discovered the significant pattern of temperature, humidity and visibility information which is valuable for the weather prediction.
format	Thesis
author	Khalid Adam, Ismail Hammad
author_facet	Khalid Adam, Ismail Hammad
author_sort	Khalid Adam, Ismail Hammad
title	Mapreduce algorithm for weather dataset
title_short	Mapreduce algorithm for weather dataset
title_full	Mapreduce algorithm for weather dataset
title_fullStr	Mapreduce algorithm for weather dataset
title_full_unstemmed	Mapreduce algorithm for weather dataset
title_sort	mapreduce algorithm for weather dataset
publishDate	2017
url	http://umpir.ump.edu.my/id/eprint/19680/ http://umpir.ump.edu.my/id/eprint/19680/ http://umpir.ump.edu.my/id/eprint/19680/1/Mapreduce%20algorithm%20for%20weather%20dataset%20-Table%20of%20contents.pdf http://umpir.ump.edu.my/id/eprint/19680/2/Mapreduce%20algorithm%20for%20weather%20dataset%20-Abstract.pdf http://umpir.ump.edu.my/id/eprint/19680/3/Mapreduce%20algorithm%20for%20weather%20dataset%20-References.pdf
first_indexed	2023-09-18T22:28:10Z
last_indexed	2023-09-18T22:28:10Z
_version_	1777416107111481344

Mapreduce algorithm for weather dataset

Similar Items