A model of multilayer tiered architecture for big data

Authors

  • Sonia Ordóñez Salinas Universidad Distrital Francisco José de Caldas, Bogotá
  • Alba Consuelo Nieto Lemus Universidad Distrital Francisco José de Caldas, Bogotá

DOI:

https://doi.org/10.18046/syt.v14i37.2257

Keywords:

Big data, data warehouse, multi-layered tiered architecture, repetitive structured data, non-repetitive unstructured data, hadoop, mapreduce, noSql.

Abstract

Until recently, the issue of analytical data was related to Data Warehouse, but due to the necessity of analyzing new types of unstructured data, both repetitive and non-repetitive, Big Data arises. Although this subject has been widely studied, there is not available a reference architecture for Big Data systems involved with the processing of large volumes of raw data, aggregated and non-aggregated. There are not complete proposals for managing the lifecycle of data or standardized terminology, even less a methodology supporting the design and development of that architecture. There are architectures in small-scale, industrial and product-oriented, which limit their scope to solutions for a company or group of companies, focused on technology but omitting the functionality. This paper explores the requirements for the formulation of an architectural model that supports the analysis and management of data: structured, repetitive and non-repetitive unstructured; there are some architectural proposals –industrial or technological type– to propose a logical model of multi-layered tiered architecture, which aims to respond to the requirements covering both Data Warehouse and Big Data.

Author Biographies

  • Sonia Ordóñez Salinas, Universidad Distrital Francisco José de Caldas, Bogotá

    Ph.D. in Systems, M.Sc. in Systems and Computing and Statistics of the Universidad Nacional de Colombia (Bogota) and Systems Engineer of the Universidad Distrital Francisco José de Caldas (Bogota). She has a widely experience in statistic models to transform text in graphs, recovery systems and databases, processing of natural language, data mining, and social networks. She currently is a full time professor in the Universidad Distrital Francisco José de Caldas and she is the leader of the GESDATOS research group, associated to the same university.

  • Alba Consuelo Nieto Lemus, Universidad Distrital Francisco José de Caldas, Bogotá

    Systems engineer of the Universidad Nacional de Colombia (Bogota), M.Sc. in in Systems Engineering and Computing of the Universidad de los Andes (Bogota). She has a wide professional experience in software development, software architectures, data management and software quality management, both in the public and private sectors. She currently works as a full time professor of the Universidad Distrital Francisco José de Caldas and she is a member of the GESTADOS and ARQUISOFT research groups, associated to the same university.

References

Agrawal, D. (2009). The reality of Real-Time Business Intelligence. En: M, Castellanos, U, Dayal. & T, Sellis. (Eds.), Lecture Notes in Business Information Processing. Vol. 27. Business Intelligence for the Real-Time Enterprise (pp. 75-88). Berlin Heidelberg : Germany : Springer-Verlag Berlin Heidelberg : Germany

Apache Hive TM. (n.d.). Retrieved from https://hive.apache.org/

Apache Impala. (n.d). Retrieved from: http://www.cloudera.com/products/apache-hadoop/impala.html

Apache Sqoop (2016, march 4). Retrieved from: http://sqoop.apache.org/

Apache SparkTM-Lightning-fast cluster computing. (n.d.). Retrieved from: http://spark.apache.org/

Apache Thrift - Home. (n.d.). Retrieved from https://thrift.apache.org/

Apache ZooKeeper - Home. (n.d.). Retrieved from https://zookeeper.apache.org/

Architecture - Apache Drill. (n.d.). Retrieved from http://drill.apache.org/architecture/

Bedi, P., Jindal, V., & Gautam, A. (2014). Beginning with big data simplified. In: Data Mining and Intelligent Computing (ICDMIC), 2014 International Conference on. IEEE. doi:10.1109/ICDMIC.2014.6954229

Brewer, E. (2012). CAP twelve years later: How the “rules” have changed. Computer. 45(2), 23-29.

Carter, S. (2013, Feb, 21). Social and BIG Data! #socbiz #ibmsocialbiz #bigdata #socialbusiness. Retrieved from: http://socialbusinesssandy.com/tag/big-data-2/page/14/

Chandarana, P. & Vijayalakshmi, M. (2014). Big data analytics frameworks. In Circuits, Systems, Communication and Information Technology Applications (CSCITA), 2014 international conference on (pp. 430-434. IEEE.

Cox, M. & Ellsworth, D. (1997). Application-controlled demand paging for out-of-core visualization [NASA Reports]. Retrieved from: http://www.nas.nasa.gov/assets/pdf/techreports/1997/nas-97-010.pdf

Cuzzocrea, A. (2014). Privacy and security of big data: current challenges and future research perspectives. In: Proceedings of the First International Workshop on Privacy and Security of Big Data (pp. 45-47). New York, NY: ACM. http://doi.acm.org/10.1145/2663715.2669614

Demchenko, Y., Laat, C. & Membrey, P. (2014). Defining architecture components of the big data ecosystem. In: Collaboration Technologies and Systems (CTS), 2014 International Conference on, 104-112. IEEE.

Díaz, Ma. (2011). Evaluación de la herramienta de código libre Apache Hadoop [thesis]. Universidad Carlos III de Madrid Escuela Politécnica Superior: Leganés, España.

Gudivada, V., Rao, D. & Raghavan, V. (2014). NoSQL systems for big data management. In: 2014 IEEE World Congress on Services (pp. 190-197). IEEE.

HDFS architecture guide. (2013, April 8). Retrieved from: http://hadoop.apache.org/docs/r1.2.1/hdfs_design.html

Hewlett Packard. (2013). HP Reference Architecture for MapR M5 [technical white paper]. Retrieved from: https://www.mapr.com/sites/default/files/hp_reference_architecture_for_mapr_m5.pdf

Inmon, W. (2005). Building the data warehouse [4a ed.]. Indianapolis, IN: Wiley.

Inmon, W.,Strauss, D. & Neushloss, G. (2008). DW 2.0: The Architecture for the Next Generation of Data Warehousing. Burlington, MA: Morgan Kaufmann

Inmon. H. & Linstedt, D. (2014). Data architecture: A primer for the data scientist: big data, data warehouse and data vault. Waltham, MA: Morgan Kaufmann.

Katal, A., Wazid, M. & Goudar, R. (2013). Big data: Issues, challenges, tools and good practices. In: Contemporary Computing (IC3), 2013 Sixth International Conference on (pp. 404-409). IEEE.

Kimball, R. (2011). The evolving role of the enterprise data warehouse in the era of big data analytics [Kimball Group white paper]. Retrieved from: http://www.montage.co.nz/assets/Brochures/DataWarehouseBigDataAnalyticsKimball.pdf

Kimball, R. (2012). Newly emerging best practices for big data [Kimball Group, white paper]. Retrieved from: http://www.kimballgroup.com/wp-content/uploads/2012/09/Newly-Emerging-Best-Practices-for-Big-Data1.pdf

Kimball, R., Ross, M., Thorthwaite, W., Becker, B. & Mundy, J. (2008). The data warehouse lifecycle toolkit [2a ed.]. Indianapolis, IN: Wiley.

Lomotey, R. K., & Deters, R. (2014). Towards knowledge discovery in big data. In: Service Oriented System Engineering (SOSE), 2014 IEEE 8th International Symposium on (pp. 181-191). IEEE.

MacDonald, A. (2015). Integrating SAP HANA and hadoop. Boston, MA: SAP Press.

Maiorescu, T. (2010). General Information on Business Intelligence and OLAP systems architecture. In: Computer and Automation Engineering (ICCAE), 2010 The 2nd International Conference on (V.2, pp. 294-297). IEEE.

Manikandan, S. G., & Ravi, S. (2014). Big data analysis using Apache Hadoop. In: IT Convergence and Security (ICITCS), 2014 International Conference on. doi: 10.1109/ICITCS.2014.7021746

Manning, C. & Schütze. H. (1999). Foundations of statistical natural language processing. Cambridge, MA: The MIT.

Marz, N. (n.d). Storm, distributed and fault-tolerant realtime computation. Retrieved from: http://cloud.berkeley.edu/data/storm-berkeley.pdf

Muntean, M., & Surcel, T. (2013). Agile BI - The Future of BI. Informatica Económica, 17(3), 114–124.

Nam, T., Choi, K., Ok, C. & Yeom, K. (2014). Service composition framework for big data service. In: Future Internet of Things and Cloud (FiCloud), 2014 International Conference on (pp. 328-333). IEEE.

Nandimath, J., Banerjee, E., Patil, A., Kakade, P., & Vaidya, S. (2013). Big Data analysis using Apache Hadoop. In: 2013 IEEE 14th International Conference on Information Reuse & Integration (IRI) (pp. 700-703). IEEE.

Oracle Corp. (2015). An enterprise architect's guide to big data [Oracle enterprise architecture - white paper.]. Retrieved from: http://www.oracle.com/technetwork/topics/entarch/articles/oea-big-data-guide-1522052.pdf

Pal, A. & Agrawal, S. (2014). An experimental approach towards big data for analyzing memory utilization on a hadoop cluster using HDFS and MapReduce. In: Networks & Soft Computing (ICNSC), 2014 First International Conference on (pp. 442-447). IEEE.

Schaffner, J., Bog, A., Krüger, J., & Zeier, A. (2009). A hybrid row-column OLTP database architecture for operational reporting. In: M. Castellanos, U. Dayal, & T. Sellis (Eds.), Business intelligence for the real-time enterprise (pp. 61-74). Berlin Heidelberg, Germany: Springer.

Todman, C. (2001). Designing a data warehouse: Supporting customer relationship management. Nueva Jersey, NJ: Prentice Hall.

Vaish, G. (2013). Getting started with NoSQL. Birmingham UK: Packt.

Welcome to ApacheTM Hadoop®! (n.d.). Retrieved from: https://hadoop.apache.org/

YiChuan, S. & Yao, X. (2012). Research of Real-time Data Warehouse Storage Strategy Based on Multi-level Caches. Physics Procedia, 25, 2315–2321.

Zhang, R., Hildebrand, D., & Tewari, R. (2014). In unity there is strength: Showcasing a unified Big Data platform with MapReduce Over both object and file storage. In: Big Data (Big Data), 2014 IEEE International Conference on (pp. 960-966). IEEE.

Downloads

Published

2016-08-05

Issue

Section

Discussion papers