Advance Search

Problems in microservice systems testing: a systematic mapping study

Christian Candela Uribe
Universidad Tecnológica de Pereira - Universidad del Quindío, Colombia
Daniel Gómez Betancur
Universidad Tecnológica de Pereira, Colombia
Luis Sepúlveda Rodríguez
Universidad Tecnológica de Pereira - Universidad del Quindío, Colombia
Julio Chavarro Porras
Universidad Tecnológica de Pereira, Colombia
John Sanabria Ordoñez
Universidad del Valle, Colombia
Keywords:
Copyright and Licensing:

Copyright (c) 2021 Investigación e Innovación en Ingenierías

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

Share:

Abstract

Objective: This work aimed to build a SMS (Systematic Mapping Study) for the identification and classification of problems and causes in testing in the MSA (Micro-Service Architecture). Methodology: This SMS adopted established practices in evidence-based software engineering. The SMS considered articles published since 2016 in databases such as ACM, IEEE Xplore, Science Direct, WoS, Springer, and Scopus. In addition, the search strategy included four snowball iterations. Results: The SMS started with 842 studies and ended with 32 selected studies, identifying nine problems and eight causes related to software testing in MSA. Finally, the studies were classified by problems and causes. Conclusions: The SMS allowed deducing that High Complexity in MSA testing is the most representative problem. Likewise, the most outstanding causes are the High Number of Services, the Lack of Research, and the proliferation of each service version (Service Versioning).

PDF (Español (España)) XML (Español (España))
PDF (Español (España)) XML (Español (España))

References

  1. P. di Francesco, I. Malavolta, and P. Lago, “Research on Architecting Microservices: Trends, Focus, and Potential for Industrial Adoption,” in Proceedings - 2017 IEEE International Conference on Software Architecture, ICSA 2017, 2017, pp. 21–30. doi: https://doi.org/10.1109/ICSA.2017.24.
  2. J. Lewis, “Microservices - A definition of this new architectural term,” Martinfowler.Com, vol. 25, pp. 1–16, 2014, [Online]. Available: http://martinfowler.com/articles/microservices.html
  3. N. Dragoni, I. Lanese, S. T. Larsen, M. Mazzara, R. Mustafin, and L. Safina, “Microservices: How to make your application scale,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2018, vol. 10742 LNCS, pp. 95–104. doi: https://doi.org/10.1007/978-3-319-74313-4_8.
  4. N. Alshuqayran, N. Ali, and R. Evans, “A systematic mapping study in microservice architecture,” in Proceedings - 2016 IEEE 9th International Conference on Service-Oriented Computing and Applications, SOCA 2016, 2016, pp. 44–51. doi: https://doi.org/10.1109/SOCA.2016.15.
  5. F. H. Vera-Rivera, C. M. Gaona Cuevas, and H. Astudillo, “Desarrollo de aplicaciones basadas en microservicios: tendencias y desafíos de investigación,” Revista Ibérica de Sistemas e Tecnologias de Informação, vol. E23, no. E23, pp. 107–120, 2019.
  6. T. Engel, M. Langermeier, B. Bauer, and A. Hofmann, “Evaluation of microservice architectures: A metric and tool-based approach,” in Lecture Notes in Business Information Processing, 2018, vol. 317, pp. 74–89. doi: https://doi.org/10.1007/978-3-319-92901-9_8.
  7. B. Kitchenham et al., “Systematic literature reviews in software engineering–a tertiary study,” Information and software technology, vol. 52, no. 8, pp. 792–805, 2010, doi: https://doi.org/10.1016/j.infsof.2010.03.006.
  8. D. Budgen, M. Turner, P. Brereton, and B. A. Kitchenham, “Using Mapping Studies in Software Engineering.,” in Ppig, 2008, vol. 8, pp. 195–204. [Online]. Available: http://www.ppig.org/papers/20th-budgen.pdf
  9. L. E. Sepúlveda-Rodríguez et al., “Study-based Systematic Mapping Analysis of Cloud Technologies for Leveraging IT Resource and Service Management: The Case Study of the Science Gateway Approach,” Journal of Grid Computing, vol. 19, no. 4, p. 41, 2021, doi: https://doi.org/10.1007/s10723-021-09587-7.
  10. S. Li et al., “Understanding and addressing quality attributes of microservices architecture: A Systematic literature review,” Information and Software Technology, vol. 131, p. 106449, 2021, doi: https://doi.org/10.1016/j.infsof.2020.106449.
  11. D. Taibi, V. Lenarduzzi, and C. Pahl, “Architectural patterns for microservices: A systematic mapping study,” in CLOSER 2018 - Proceedings of the 8th International Conference on Cloud Computing and Services Science, 2018, vol. 2018-Janua, pp. 221–232. doi: https://doi.org/10.5220/0006798302210232.
  12. T. Cerny, M. J. Donahoo, and M. Trnka, “Contextual understanding of microservice architecture,” ACM SIGAPP Applied Computing Review, vol. 17, no. 4, pp. 29–45, 2018, doi: https://doi.org/10.1145/3183628.3183631.
  13. P. di Francesco, P. Lago, and I. Malavolta, “Architecting with microservices: A systematic mapping study,” Journal of Systems and Software, vol. 150, pp. 77–97, 2019, doi: https://doi.org/10.1016/j.jss.2019.01.001.
  14. I. Ghani, W. M. N. Wan-Kadir, A. Mustafa, and M. I. Babir, “Microservice testing approaches: A systematic literature review,” International Journal of Integrated Engineering, vol. 11, no. 8, pp. 65–80, 2019, doi: https://doi.org/10.30880/ijie.2019.11.08.008.
  15. L. M. Hillah et al., “Automation and intelligent scheduling of distributed system functional testing: Model-based functional testing in practice,” International Journal on Software Tools for Technology Transfer, vol. 19, no. 3, pp. 281–308, 2017, doi: https://doi.org/10.1007/s10009-016-0440-3.
  16. Y. Wang, L. Cheng, and X. Sun, “Design and Research of Microservice Application Automation Testing Framework,” in Proceedings - 2019 International Conference on Information Technology and Computer Application, ITCA 2019, 2019, pp. 257–260. doi: https://doi.org/10.1109/ITCA49981.2019.00063.
  17. M. J. Kargar and A. Hanifizade, “Automation of regression test in microservice architecture,” in 2018 4th International Conference on Web Research, ICWR 2018, 2018, pp. 133–137. doi: https://doi.org/10.1109/ICWR.2018.8387249.
  18. S. Sivanandan, “Test Automation Framework as a Service (TAFaaS) - scale test automation & devops practices with cloud, containers, and microservice.,” International Journal of Innovative Technology and Exploring Engineering, vol. 8, no. 7C2, pp. 108–111, 2019, [Online]. Available: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85070741957&partnerID=40&md5=879423b8e93c0fc922a389dddafa709e
  19. V. R. Basili and G. Caldiera, “The Goal Question Metric Paradigm,” Encyclopedia of Software Engineering - 2 Volume Set, pp. 528–532, 2000, [Online]. Available: https://www.cs.umd.edu/~basili/publications/technical/T89.pdf
  20. V. R. Basili, “Software Modeling and Measrement,” 1992.
  21. I. G. Needleman, “A guide to systematic reviews,” Journal of Clinical Periodontology, vol. 29, no. SUPPL. 3, pp. 6–9, 2002, doi: https://doi.org/10.1034/j.1600-051x.29.s3.15.x.
  22. M. Petticrew and H. Roberts, Systematic Reviews in the Social Sciences: A Practical Guide. John Wiley & Sons, 2008. doi: https://doi.org/10.1002/9780470754887.
  23. E. Almanasreh, R. Moles, and T. F. Chen, “Evaluation of methods used for estimating content validity,” Research in Social and Administrative Pharmacy, vol. 15, no. 2, pp. 214–221, 2019, doi: https://doi.org/10.1016/j.sapharm.2018.03.066.
  24. F. YAGHMAEI, “Content validity and its estimation,” Journal of Medical Education, vol. 3, no. 1, pp. 25–27, 2003, doi: https://doi.org/10.22037/jme.v3i1.870.
  25. E. Mourao, M. Kalinowski, L. Murta, E. Mendes, and C. Wohlin, “Investigating the Use of a Hybrid Search Strategy for Systematic Reviews,” in International Symposium on Empirical Software Engineering and Measurement, 2017, vol. 2017-Novem, pp. 193–198. doi: https://doi.org/10.1109/ESEM.2017.30.
  26. P. H. Nguyen, M. Kramer, J. Klein, and Y. le Traon, “An extensive systematic review on the Model-Driven Development of secure systems,” Information and Software Technology, vol. 68, pp. 62–81, 2015, doi: https://doi.org/10.1016/j.infsof.2015.08.006.
  27. A. Q. Ali, A. B. M. Sultan, A. A. A. Ghani, and H. Zulzalil, “A Systematic Mapping Study on the Customization Solutions of Software as a Service Applications,” IEEE Access, vol. 7, pp. 88196–88217, 2019, doi: https://doi.org/10.1109/ACCESS.2019.2925499.
  28. K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson, “Systematic mapping studies in software engineering,” in 12th International Conference on Evaluation and Assessment in Software Engineering, EASE 2008, 2008, pp. 1–10. doi: https://doi.org/10.14236/ewic/ease2008.8.
  29. M. Waseem, P. Liang, and M. Shahin, “A Systematic Mapping Study on Microservices Architecture in DevOps,” Journal of Systems and Software, vol. 170, p. 110798, 2020, doi: https://doi.org/10.1016/j.jss.2020.110798.
  30. S. P. Ma, C. Y. Fan, Y. Chuang, I. H. Liu, and C. W. Lan, “Graph-based and scenario-driven microservice analysis, retrieval, and testing,” Future Generation Computer Systems, vol. 100, pp. 724–735, 2019, doi: https://doi.org/10.1016/j.future.2019.05.048.
  31. N. Dragoni et al., “Microservices: Yesterday, today, and tomorrow,” in Present and Ulterior Software Engineering, M. Mazzara and B. Meyer, Eds. Cham: Springer International Publishing, 2017, pp. 195–216. doi: https://doi.org/10.1007/978-3-319-67425-4_12.
  32. M. Villamizar et al., “Cost comparison of running web applications in the cloud using monolithic, microservice, and AWS Lambda architectures,” Service Oriented Computing and Applications, vol. 11, no. 2, pp. 233–247, Jun. 2017, doi: https://doi.org/10.1007/s11761-017-0208-y.
  33. S. Baškarada, V. Nguyen, and A. Koronios, “Architecting Microservices: Practical Opportunities and Challenges,” Journal of Computer Information Systems, vol. 60, no. 5, pp. 428–436, 2020, doi: https://doi.org/10.1080/08874417.2018.1520056.
  34. D. Savchenko, G. Radchenko, T. Hynninen, and O. Taipale, “Microservice Test Process: Design and Implementation,” International Journal on Information Technologies & Security, vol. 10, no. 3, pp. 13–24, 2018.
  35. T. Cerny et al., “On Code Analysis Opportunities and Challenges for Enterprise Systems and Microservices,” IEEE Access, vol. 8, pp. 159449–159470, 2020, doi: https://doi.org/10.1109/ACCESS.2020.3019985.
  36. V. Lenarduzzi, F. Lomio, N. Saarimäki, and D. Taibi, “Does migrating a monolithic system to microservices decrease the technical debt?,” Journal of Systems and Software, vol. 169, p. 110710, 2020, doi: https://doi.org/10.1016/j.jss.2020.110710.
  37. Y. Wang, H. Kadiyala, and J. Rubin, “Promises and challenges of microservices: an exploratory study,” Empirical Software Engineering, vol. 26, no. 4. pp. 1–45, 2021. doi: https://doi.org/10.1007/s10664-020-09910-y.
  38. A. Benayache, A. Bilami, S. Barkat, P. Lorenz, and H. Taleb, “MsM: A microservice middleware for smart WSN-based IoT application,” Journal of Network and Computer Applications, vol. 144, pp. 138–154, 2019, doi: https://doi.org/10.1016/j.jnca.2019.06.015.
  39. T. Cerny, “Aspect-oriented challenges in system integration with microservices, SOA and IoT,” Enterprise Information Systems, vol. 13, no. 4, pp. 467–489, 2019, doi: https://doi.org/10.1080/17517575.2018.1462406.
  40. X. Larrucea, I. Santamaria, R. Colomo-Palacios, and C. Ebert, “Microservices,” IEEE Software, vol. 35, no. 3, pp. 96–100, 2018.
  41. A. Nebel and L. González, “Quality-based Methodology for Assessing the Applicability of Microservices Architecture,” 23rd Iberoamerican Conference on Software Engineering, CIbSE 2020, 2020.
  42. A. Arcuri, “RESTful API automated test case generation,” in Proceedings - 2017 IEEE International Conference on Software Quality, Reliability and Security, QRS 2017, 2017, pp. 9–20. doi: https://doi.org/10.1109/QRS.2017.11.
  43. A. Balalaie, A. Heydarnoori, and P. Jamshidi, “Microservices Architecture Enables DevOps: Migration to a Cloud-Native Architecture,” IEEE Software, vol. 33, no. 3, pp. 42–52, 2016, doi: https://doi.org/10.1109/MS.2016.64.
  44. A. Avritzer et al., “Scalability Assessment of Microservice Architecture Deployment Configurations: A Domain-based Approach Leveraging Operational Profiles and Load Tests,” Journal of Systems and Software, vol. 165, p. 110564, 2020, doi: https://doi.org/10.1016/j.jss.2020.110564.
  45. R. Pietrantuono, S. Russo, and A. Guerriero, “Testing microservice architectures for operational reliability,” Software Testing Verification and Reliability, vol. 30, no. 2, Mar. 2020, doi: https://doi.org/10.1002/stvr.1725.
  46. R. Chhonker, Ananya and de Lemos, “Microservices Architectures and Technical Debt: A Self-adaptation View,” pp. 1–2, 2019.
  47. C. T. Joseph and K. Chandrasekaran, “Straddling the crevasse: A review of microservice software architecture foundations and recent advancements,” Software - Practice and Experience, vol. 49, no. 10, pp. 1448–1484, 2019, doi: https://doi.org/10.1002/spe.2729.
  48. J. Soldani, D. A. Tamburri, and W. J. van den Heuvel, “The pains and gains of microservices: A Systematic grey literature review,” Journal of Systems and Software, vol. 146, pp. 215–232, 2018, doi: https://doi.org/10.1016/j.jss.2018.09.082.
  49. N. C. Mendonça, P. Jamshidi, D. Garlan, and C. Pahl, “Developing self-adaptive microservice systems: Challenges and directions,” IEEE Software, 2019.
  50. M. Viggiato, R. Terra, H. Rocha, M. T. Valente, and E. Figueiredo, “Microservices in Practice: A Survey Study,” arXiv, 2018, [Online]. Available: http://arxiv.org/abs/1808.04836
  51. N. Raychev, “Test automation in microservice architecture,” vol. 8, no. 7. IEEE-SEM, 2020.
  52. R. Boncea, A. Zamfiroiu, and I. Bacivarov, “A scalable architecture for automated monitoring of microservices,” Academy of Economic Studies. Economy Informatics, vol. 18, no. 1, pp. 13–22, 2018, [Online]. Available: https://search.proquest.com/docview/2172010739?accountid=14542%0Ahttp://dn3nh3eq7d.search.serialssolutions.com/?genre=article&sid=ProQ:&atitle=A+scalable+architecture+for+automated+monitoring+of+microservices&title=Academy+of+Economic+Studies.+Economy+Inf
  53. F. Auer, V. Lenarduzzi, M. Felderer, and D. Taibi, “From monolithic systems to Microservices: An assessment framework,” Information and Software Technology, vol. 137. 2021. doi: https://doi.org/10.1016/j.infsof.2021.106600.
  54. O. Zimmermann, “Microservices tenets: Agile approach to service development and deployment,” Computer Science - Research and Development, vol. 32, no. 3–4, pp. 301–310, Jul. 2017, doi: https://doi.org/10.1007/s00450-016-0337-0.
  55. J. Davies, D. Schorow, S. Ray, and D. Rieber, Eds., “Versioning Services BT - The Definitive Guide to SOA: Oracle® Service Bus,” Berkeley, CA: Apress, 2008, pp. 341–358. doi: https://doi.org/10.1007/978-1-4302-1058-0_14.
  56. Chris. Richardson, Microservices patterns : with examples in Java. Manning Publications, 2019. [Online]. Available: https://learning.oreilly.com/library/view/microservices-patterns/9781617294549/
How to Cite
[1]
C. A. Candela Uribe, D. Gómez-Betancur, L. Sepúlveda-Rodríguez, J. Chavarro-Porras, and J. Sanabria-Ordoñez, “Problems in microservice systems testing: a systematic mapping study”, Investigación e Innovación en Ingenierías, vol. 9, no. 3, pp. 3–17, Dec. 2021.

Send mail to Author

Send Cancel

Custom technologies based on your needs

  • MongoDB
  • ElasticSearch
  • Redis
  • Solr
  • Memcached