A journal of IEEE and CAA , publishes high-quality papers in English on original theoretical/experimental research and development in all areas of automation
Volume 7 Issue 1
Jan.  2020

IEEE/CAA Journal of Automatica Sinica

  • JCR Impact Factor: 6.171, Top 11% (SCI Q1)
    CiteScore: 11.2, Top 5% (Q1)
    Google Scholar h5-index: 51, TOP 8
Turn off MathJax
Article Contents
Hira Zahid, Tariq Mahmood, Ahsan Morshed and Timos Sellis, "Big Data Analytics in Telecommunications: Literature Review and Architecture Recommendations," IEEE/CAA J. Autom. Sinica, vol. 7, no. 1, pp. 18-38, Jan. 2020. doi: 10.1109/JAS.2019.1911795
Citation: Hira Zahid, Tariq Mahmood, Ahsan Morshed and Timos Sellis, "Big Data Analytics in Telecommunications: Literature Review and Architecture Recommendations," IEEE/CAA J. Autom. Sinica, vol. 7, no. 1, pp. 18-38, Jan. 2020. doi: 10.1109/JAS.2019.1911795

Big Data Analytics in Telecommunications: Literature Review and Architecture Recommendations

doi: 10.1109/JAS.2019.1911795
Funds:  This work was supported in part by the Big Data Analytics Laboratory (BDA-LAB) at the Institute of Business Administration under the research grant approved by the Higher Education Commission of Pakistan (www.hec.gov.pk) and in part by the Darbi company (www.darbi.io)
More Information
  • This paper focuses on facilitating state-of-the-art applications of big data analytics (BDA) architectures and infrastructures to telecommunications (telecom) industrial sector. Telecom companies are dealing with terabytes to petabytes of data on a daily basis. IoT applications in telecom are further contributing to this data deluge. Recent advances in BDA have exposed new opportunities to get actionable insights from telecom big data. These benefits and the fast-changing BDA technology landscape make it important to investigate existing BDA applications to telecom sector. For this, we initially determine published research on BDA applications to telecom through a systematic literature review through which we filter 38 articles and categorize them in frameworks, use cases, literature reviews, white papers and experimental validations. We also discuss the benefits and challenges mentioned in these articles. We find that experiments are all proof of concepts (POC) on a severely limited BDA technology stack (as compared to the available technology stack), i.e., we did not find any work focusing on full-fledged BDA implementation in an operational telecom environment. To facilitate these applications at research-level, we propose a state-of-the-art lambda architecture for BDA pipeline implementation (called LambdaTel) based completely on open source BDA technologies and the standard Python language, along with relevant guidelines. We discovered only one research paper which presented a relatively-limited lambda architecture using the proprietary AWS cloud infrastructure. We believe LambdaTel presents a clear roadmap for telecom industry practitioners to implement and enhance BDA applications in their enterprises.


  • loading
  • 1The complete list of indexed resources is not made public by Google.
    2Mobile Network Operators.
    3Where necessary, we have itemized the paper discussion of validation-related papers to enhance readability.
    4We have adapted this architecture from one of our previous works [86].
  • [1]
    D. Laney, " 3D data management: Controlling data volume, velocity, and variety,” META Group, Tech. Rep., Feb. 2001. [Online]. Available: http://blogs.gartner.com/doug-laney/files/2012/01/ad949-3D-Data-Management-Controlling-Data-Volume-Velocity-and-Variety.pdf
    P. Zikopoulos, C. Eaton et al., Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming Data. McGraw-Hill Osborne Media, 2011.
    M. E, " The world according to linq,” Communications of the ACM, vol. 10, no. 54, pp. 45–51, 2011.
    F. X. Diebold, " Big data dynamic factor models for macroeconomic measurement and forecasting,” in Advances in Economics and Econometrics: Theory and Applications, Eighth World Congress of the Econometric Society, ”(edited by M. Dewatripont, LP Hansen and S. Turnovsky), 2003, pp. 115–122.
    J. Liebowitz, Big Data and Business Analytics, 1st ed., Amazon, Ed. CRC Press, 2013.
    F. J. Ohlhorst, Big Data Analytics: Turning Big Data into Big Money, 1st ed., Amazon, Ed. John Wiley & Sons, 2012.
    G. C. Deka, NoSQL: Database for Storage and Retrieval of Data in Cloud, Amazon, Ed. Chapman and Hall/CRC, 2017.
    C. M. Ricardo and S. D. Urban, Databases Illuminated, 3rd ed., Amazon, Ed. Jones & Bartlett Learning, 2015.
    M. D. D. Silva and H. L. Tavares, Redis Essentials, Amazon, Ed. Packt Publishing, 2015.
    M. D. D. Silva and H. L. Tavares, MongoDB: The Definitive Guide: Powerful and Scalable Data Storage, 2nd ed., Amazon, Ed. O’Reilly Media, 2013.
    P. Singh, " 10 reasons why big data and analytics projects fail,” https://analyticsindiamag.com/10-reasons-big-data-analytics-projects-fail/, 2017.
    B. Violino, " How to avoid big data analytics failures,” https://www.infoworld.com/article/3212945/big-data/how-to-avoid-big-data-analytics-failures.html, 2017.
    H. Demirkan and B. Dal, " The data economy: Why do so many analytics projects fail?” [Online]. Available: http://analytics-magazine.org/the-data-economy-why-do-so-many-analytics-projects-fail/, 2014.
    M. Asay, " 85% of big data projects fail, but your developers can help yours succeed,” [Online]. Available: https://www.techrepublic.com/article/85-of-big-data-projects-fail-but-your-developers-can-help-yours-succeed/, 2017.
    Datafloq, " Top reasons of hadoop - big data project failures,” https://datafloq.com/read/top-reasons-of-hadoop-big-data-project-failures/2185, 2017.
    N. Marz and J. Warren, Big Data: Principles and Best Practices of Scalable Realtime Data Systems. Manning Publications Co., 2015.
    M. Chen, S. Mao, and Y. Liu, " Big data: a survey,” Mobile Networks and Applications, vol. 19, no. 2, pp. 171–209, 2014. doi: 10.1007/s11036-013-0489-0
    A. Kumari, S. Tanwar, S. Tyagi, N. Kumar, M. Maasberg, and K.- K. R. Choo, " Multimedia big data computing and internet of things applications: A taxonomy and process model,” J. Network and Computer Applications, vol. 124, pp. 169–195, Dec. 2018. doi: 10.1016/j.jnca.2018.09.014
    A. Bahga and V. Madisetti, Big Data Science & Analytics: A Hands-On Approach, Amazon, Ed. VPT, 2016.
    M. Turck, " Firing on all cylinders: The 2017 big data landscape,” http://mattturck.com/bigdata2017/, 2017.
    J. Manyika, M. Chui, M. G. Institute, B. Brown, J. Bughin, R. Dobbs, C. Roxburgh, and A. Byers, Big Data: The Next Frontier for Innovation, Competition, and Productivity. McKinsey, 2011. [Online]. Available: https://books.google.com.pk/books?id=APsUMQAACAAJ
    S. Parise, " Big data: a revolution that will transform how we live, work, and think, by viktor mayer-schonberger and kenneth cukier,” J. Information Technology Case and Application Research, vol. 18, no. 3, pp. 186–190, Sept. 2016. [Online]. Available: https://doi.org/10.1080/15228053.2016.1220197
    J. Bughin, " Reaping the benefits of big data in telecom,” J. Big Data, vol. 3, no. 1, 2016.
    S. Han, C. -L. I, G. Li, S. Wang, and Q. Sun, " Big data enabled mobile network design for 5g and beyond,” IEEE Communications Magazine, vol. 55, no. 9, pp. 150–157, 2017. [Online]. Available: https://doi.org/10.1109/mcom.2017.1600911
    D. Sipus, " Big data analytics for communication service providers,” in Proc. 39th IEEE Int. Conv. Information and Communication Technology, Electronics and Microelectronics, May 2016.
    M. S. Parwez, D. Rawat, and M. Garuba, " Big data analytics for user activity analysis and user anomaly detection in mobile wireless network,” IEEE Trans. Industrial Informatics, 2017.
    I. Chih-Lin, Y. Liu, S. Han, S. Wang, and G. Liu, " On big data analytics for greener and softer ran,” IEEE Access, vol. 3, pp. 3068–3075, 2015. doi: 10.1109/ACCESS.2015.2469737
    R. F. Baumeister and M. R. Leary, " Writing narrative literature reviews,” Review of General Psychology, vol. 1, no. 3, pp. 311–320, 1997. doi: 10.1037/1089-2680.1.3.311
    C. M. Murphy, " Writing an effective review article,” Journal of Medical Toxicology, vol. 8, no. 2, pp. 89–90, Jun 2012. [Online]. Available: https://doi.org/10.1007/s13181-012-0234-2
    A. Leon-Garcia and I. Widjaja, Communication Networks, 2nd ed., Amazon, Ed. USA: McGraw-Hill Education, 2003.
    L. Goleniewski and K. W. Jarrett, Telecommunications Essentials: The Complete Global Source, 2nd ed., K. W. Jarrett, Ed. USA: Addison Wesley Professional, 2006.
    Gartner, " It glossary,” https://www.gartner.com/it-glossary/big-data, 2018.
    T. White, Hadoop: The Definitive Guide, 3rd ed., Amazon, Ed. USA: Yahoo Press, 2012.
    F. Cruz, P. Gomes, R. Oliveira, and J. Pereira, " Assessing NoSQL databases for telecom applications,” in Proc. 13th IEEE Conf. Commerce and Enterprise Computing, Sept. 2011.
    H. Daki, A. El Hannani, A. Aqqal, A. Haidine, A. Dahbi, and H. Ouahmane, " Towards adopting big data technologies by mobile networks operators: A moroccan case study,” in Proc. 2nd IEEE Int. Conf. Cloud Computing Technologies and Applications, 2016, pp. 154–161.
    L. George, HBase: The Definitive Guide: Random Access to Your PlanetSize Data, 1st ed., Amazon, Ed. USA: O’Reilly Media, 2011.
    M. A. Abbasi, Learning Apache Spark 2.0, 1st ed., Amazon, Ed. USA: Packt Publishing - ebooks Account, 2017.
    N. Garg, Learning Apache Kafka, Second Edition, 2nd ed., Amazon, Ed. USA: Packt Publishing, 2015.
    F. Hueske and V. Kalavri, Stream Processing With Apache Flink: Fundamentals, Implementation, and Operation of Streaming Applications, 1st ed., Amazon, Ed. USA: O’Reilly Media, 2018.
    S. T. Allen, M. Jankowski, and P. Pathirana, Storm Applied: Strategies for Real-Time Event Processing, 1st ed., Amazon, Ed. USA: Manning Publications, 2015.
    P. J. Sadalage and M. Fowler, NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence, 1st ed., Amazon, Ed. USA: Addison-Wesley Professional, 2012.
    G. Scholar, " Inclusion guidelines for webmasters,” https://scholar.google.com/intl/en/scholar/inclusion.html, 2018.
    Mendeley, " Mendeley feed,” www.mendeley.com, 2018.
    E. J. Khatib, R. Barco, P. Muñoz, I. De La Bandera, and I. Serrano, " Self-healing in mobile networks with big data,” IEEE Communications Magazine, vol. 54, no. 1, pp. 114–120, 2016. doi: 10.1109/MCOM.2016.7378435
    S. Bi, R. Zhang, Z. Ding, and S. Cui, " Wireless communications in the era of big data,” IEEE Communications Magazine, vol. 53, no. 10, pp. 190–199, 2015. doi: 10.1109/MCOM.2015.7295483
    B. R. Chang, H. F. Tsai, Z.-Y. Lin, and C. -M. Chen, " Access-controlled video/voice over ip in hadoop system with bpnn intelligent adaptation,” in Proc. IEEE Int. Conf. Information Security and Intelligence Control, 2012, pp. 325–328.
    A. Drosou, I. Kalamaras, S. Papadopoulos, and D. Tzovaras, " An enhanced graph analytics platform (gap) providing insight in big network data,” J. Innovation in Digital Ecosystems, vol. 3, no. 2, pp. 83–97, 2016. doi: 10.1016/j.jides.2016.10.005
    S. B. Elagib, A.-H. A. Hashim, and R. Olanrewaju, " CDR analysis using big data technology,” in Proc. IEEE Int. Conf. Computing, Control, Networking, Electronics and Embedded Systems Engineering, 2015, pp. 467–471.
    J. George, C. -A. Chen, R. Stoleru, and G. Xie, " Hadoop MapReduce for mobile clouds,” IEEE Trans. Cloud Computing, pp. 1–1, 2016. [Online]. Available: https://doi.org/10.1109/tcc.2016.2603474
    Y. He, F. R. Yu, N. Zhao, H. Yin, H. Yao, and R. C. Qiu, " Big data analytics in mobile cellular networks,” IEEE Access, vol. 4, pp. 1985–1996, 1985.
    X. Lu, F. Su, H. Liu, W. Chen, and X. Cheng, " A unified OLAP/OLTP big data processing framework in telecom industry,” in Proc. 16th IEEE Int. Symp. Communications and Information Technologies, Sept. 2016, pp. 290–295.
    Y. Ouyang, L. Shi, A. Huet, M. M. Hu, and X. Dai, " Predicting 4g adoption with apache spark: A field experiment,” in Proc.16th Int. Symp. Communications and Information Technologies, 2016, pp. 235-240.
    A. Imran, A. Zoha, and A. Abu-Dayya, " Challenges in 5G: how to empower son with big data for enabling 5G,” IEEE Network, vol. 28, no. 6, pp. 27–33, 2014. doi: 10.1109/MNET.2014.6963801
    J. Liu, F. Liu, and N. Ansari, " Monitoring and analyzing big traffic data of a large-scale cellular network with hadoop,” IEEE Network, vol. 28, no. 4, pp. 32–39, Jul. 2014. [Online]. Available: https://doi.org/10.1109/mnet.2014.6863129
    K. Zheng, Z. Yang, K. Zhang, P. Chatzimisios, K. Yang, and W. Xiang, " Big data-driven optimization for mobile networks toward 5G,” IEEE Network, vol. 30, no. 1, pp. 44–51, 2016. doi: 10.1109/MNET.2016.7389830
    E. Baccarelli, N. Cordeschi, A. Mei, M. Panella, M. Shojafar, and J. Stefa, " Energy-efficient dynamic traffic offloading and reconfiguration of networked data centers for big data stream mobile computing: review, challenges, and a case study,” IEEE Network, vol. 30, no. 2, pp. 54–61, 2016. doi: 10.1109/MNET.2016.7437025
    S. Jain, M. Khandelwal, A. Katkar, and J. Nygate, " Applying big data technologies to manage QoS in an sdn,” in Proc. 12th IEEE Int. Conf. Network and Service Management, 2016, pp. 302–306.
    R. I. Jony, A. Habib, N. Mohammed, and R. I. Rony, " Big data use case domains for telecom operators,” in Proc. IEEE Int. Conf. Smart City/SocialCom/SustainCom, Dec. 2015, pp. 850–855.
    A. Saad, A. R. Amran, I. W. Phillips, and A. M. Salagean, " Big data analysis on secure VoIP services,” in Proc. 11th Int. Conf. Ubiquitous Information Management and Communication. ACM, pp. 5, 2017.
    H. Park, H. Gebre-Amlak, B. Choi, S. Song, and D. Wolfinbarger, " Understanding university campus network reliability characteristics using a big data analytics tool,” in Proc. 11th Int. Conf. Design of Reliable Communication Networks, March 2015, pp. 107–110.
    M. Rathore, A. Paul, A. Ahmad, M. Imran, and M. Guizani, " Highspeed network traffic analysis: Detecting VoIP calls in secure big data streaming,” in Proc. IEEE 41st Conf. Local Computer Networks, Nov. 2016, pp. 595–598.
    C. Şenbalcı, S. Altuntaş, Z. Bozkus, and T. Arsan, " Big data platform development with a domain specific language for telecom industries,” in Proc. High Capacity Optical Networks and Emerging/Enabling Technologies, Dec. 2013, pp. 116–120.
    J.-C. Tseng, H.-C. Tseng, C.-W. Liu, C.-C. Shih, K.-Y. Tseng, C.-Y. Chou, C.-H. Yu, and F.-S. Lu, " A successful application of big data storage techniques implemented to criminal investigation for telecom,” in Proc. 15th IEEE Conf. Asia-Pacific Network Operations and Management Symposium, 2013, pp. 1–3.
    R. Van Den Dam, " Big data a sure thing for telecommunications: telecom’s future in big data,” in Proc. IEEE Int. Conf. CyberEnabled Distributed Computing and Knowledge Discovery, 2013, pp. 148–154.
    T. Yigit, M. A. Cakar, and A. S. Yuksel, " The experience of nosql database in telecommunication enterprise,” in Proc. 7th IEEE Int. Conf. Application of Information and Communication Technologies, 2013, pp. 1–4.
    R. Siddavaatam, I. Woungang, G. Carvalho, and A. Anpalagan, " An efficient method for mobile big data transfer over hetnet in emerging 5G systems,” in Proc. 21st IEEE Int. Workshop on Computer Aided Modelling and Design of Communication Links and Networks, 2016, pp. 59–64.
    R. Siddavaatam, I. Woungang, G. Carvalho, and A. Anpalagan, " Efficient ubiquitous big data storage strategy for mobile cloud computing over hetnet,” in Proc. IEEE Global Communications Conf., Dec. 2016, pp. 1–6.
    R. K. Lomotey and R. Deters, " Management of mobile data in a crop field,” in Proc. IEEE Int. Conf. Mobile Services, 2014, pp. 100–107.
    M. Jonathan and K. Tor, " Subscriber Classification Within Telecom Networks Utilizing Big Data Technologies and Machine Learning,” in Proc. 1st Int. Workshop on Big Data, Streams and Heterogeneous Source Mining: Algorithms, Systems, Programming Models and Applications, pp. 77–84, 2012.
    Ö. F. Çelebi, E. Zeydan, O. F. Kurt, O. Dedeoglu, Ö. Ileri, B. AykutSungur, A. Akan, and S. Ergüt, " On use of big data for enhancing network coverage analysis,” ICT, pp. 1–5, 2013.
    C. Costa, G. Chatzimilioudis, D. Zeinalipour-Yazti, and M. F. Mokbel, " Efficient exploration of telco big data with compression and decaying,” in Proc. IEEE 33rd Int. Conf. Data Engineering, 2017, pp. 1332–1343.
    C.-M. Chen, " Use cases and challenges in telecom big data analytics,” APSIPA Trans. Signal and Information Processing, vol. 5, pp. 12, 2016. doi: 10.1017/ATSIP.2016.12
    R. Jony, " Preprocessing solutions for telecommunication specific big data use cases,” Universidad Nacional De Educación A Distancia Facultad De Educación, 2014.
    K. Wang, J. Mi, C. Xu, Q. Zhu, L. Shu, and D.-J. Deng, " Realtime load reduction in multimedia big data for mobile internet,” ACM Trans. Multimedia Computing, Communications, and Applications, vol. 12, no. 5s, pp. 1–20, Oct. 2016. [Online]. Available: https://doi.org/10.1145/2990473
    J. van der Lande, " The future of big data analytics in the telecoms industry,” White Paper, 2014.
    N. Dalkey and O. Helmer, " An experimental application of the DELPHI method to the use of experts,” Management Science, vol. 9, no. 3, pp. 458–467, Apr. 1963. [Online]. Available: https://doi.org/10.1287/mnsc.9.3.458
    R. R. Panko, Business Data Networks and Telecommunications, 4th ed., Amazon, Ed. USA: Prentice Hall, 2002.
    D. Slama, F. Puhlmann, J. Morrish, and R. M. Bhatnagar, Enterprise IoT: Strategies and Best Practices for Connected Products and Services, 1st ed., Amazon, Ed. O’Reilly Media, 2015.
    A. Banerjee, " Big data & advanced analytics in telecom: a multibillion-dollar revenue opportunity (technical report),” New York: Heavy Reading, 2013.
    M. Kiran, P. Murphy, I. Monga, J. Dugan, and S. S. Baveja, " Lambda architecture for cost-effective batch and speed big data processing,” in Proc. IEEE Int. Conf. Big Data, 2015, pp. 2785–2792.
    C. E. Perkins and P. R. Calhoun, " Authentication, authorization, and accounting (AAA) registration keys for mobile IPV4,” RFC, vol. 3957, pp. 1–27, 2005.
    A. Anthony, Mastering AWS Security: Create and Maintain A Secure Cloud Ecosystem, 1st ed., Amazon, Ed. USA: Packt Publishing, 2017.
    D. S. Yuri Diogenes, Tom Shinder, Microsoft Azure Security Infrastructure (IT Best Practices - Microsoft Press), 1st ed., Amazon, Ed. USA: Microsoft Press, 2016.
    H. Zahid, T. Mahmood, and N. Ikram, " Enhancing dependability in big data analytics enterprise pipelines,” in Security, Privacy, and Anonymity in Computation, Communication, and Storage, G. Wang, J. Chen, and L. T. Yang, Eds. Cham: Springer International Publishing, 2018, pp. 272–281.
    G. Weiss, Data Mining in the Telecommunications Industry. GI Global, 2009.
    W. Queiroz, M. A. Capretz, and M. Dantas, " An approach for SDN traffic monitoring based on big data techniques,” J. Network and Computer Applications, vol. 131, pp. 28–39, Apr. 2019. doi: 10.1016/j.jnca.2019.01.016
    I. Haber, " Why redis beats memcached for caching,” https://www.infoworld.com/article/3063161/nosql/why-redis-beats-memcached-for-caching.html, 2018.
    Redis, " Using redis as an LRU cache,” https://redis.io/topics/lru-cache, 2018.
    J. Kreps, " Questioning the lambda architecture,” https://www.oreilly.com/ideas/questioning-the-lambda-architecture, Jul. 2, 2014.
    I. Samizadeh, " A brief introduction to two data processing architectures — lambda and kappa for big data,” https://towardsdatascience.com, 2018.
    J. Forgeat, " Data processing architectures — lambda and kappa,” https://www.ericsson.com/en/blog/2015/11/data-processing-architectures--lambda-and-kappa, 2015.
    M. Verrilli, " From lambda to kappa: a guide on real-time big data architectures,” https://www.talend.com/blog/2017/08/28/lambda-kappa-real-time-big-data-architectures/, Aug. 28, 2017.
    J. ZAGELBAUM, " Kappa architecture: a different way to process data,” https://www.blue-granite.com/blog/a-different-way-to-process-data-kappa-architecture, Jan. 25, 2019.
    L. Zhou, A. Fu, S. Yu, M. Su, and B. Kuang, " Data integrity verification of the outsourced big data in the cloud environment: a survey,” J. Network and Computer Applications, vol. 122, pp. 1–15, Nov. 2018. doi: 10.1016/j.jnca.2018.08.003
    R. Nachiappan, B. Javadi, R. N. Calheiros, and K. M. Matawie, " Cloud storage reliability for big data applications: a state of the art survey,” J. Network and Computer Applications, vol. 97, pp. 35–47, 2017. doi: 10.1016/j.jnca.2017.08.011
    W. Xu, H. Zhou, N. Cheng, F. Lyu, W. Shi, J. Chen, and X. Shen, " Internet of vehicles in big data era,” IEEE/CAA J. Autom. Sinica, vol. 5, no. 1, pp. 19–35, Jan. 2018. doi: 10.1109/JAS.2017.7510736
    Z. Sheng, S. Pfersich, A. Eldridge, J. Zhou, D. Tian, and V. C. M. Leung, " Wireless acoustic sensor networks and edge computing for rapid acoustic monitoring,” IEEE/CAA J. Autom. Sinica, vol. 6, no. 1, pp. 64–74, 2019. doi: 10.1109/JAS.2019.1911324


    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(11)  / Tables(4)

    Article Metrics

    Article views (5707) PDF downloads(204) Cited by()


    • Systematic literature review on big data analytics applications to telecom sector.
    • Only 38 research articles published in total with no industrial deployment use case.
    • Classification of articles across the standard big data analytics technology stack.
    • Propose and implement a lambda architecture for big data analytics in telecom.
    • Many research gaps in the face of the rapidly expanding big data technology stack.


    DownLoad:  Full-Size Img  PowerPoint