PARTNER CONTENT: With the continuous application of cloud computing technology in various industries, private cloud, public cloud, and hybrid cloud gradually become the core infrastructure of enterprises. The development of cloud computing reaches a new inflection point.
In cloud-native mode, the system architecture is fast, flexible, elastic, and scalable. However, it also brings challenges such as the increasing complexity of system development and delivery; as well as results in high requirements for development and O&M personnel.
DevOps is one of the four key pillars of cloud native technology. The concepts of agile development, continuous delivery, GitOps, and integrated development and O&M are gradually recognized by the industry as the best development and delivery solution in the background of cloud native technologies. Whether they are large internet companies, traditional financial or telecommunications companies, or small and medium-enterprises, the DevOps is applied at various degrees. DevOps helps enterprises to reduce IT development and O&M costs, improve service innovation delivery efficiency, and accelerate industry value monetization. However, Cloud-native DevOps is facing new challenges during its implementation journey.
With a population of more than 80 million, China Mobile Jiangsu serves more than 60 million individual users and 160 million loT users. The front desk of the IT system receives 40 million visits every day, and the number of Charging Data Records (CDRs) exceeds 15 billion. The extremely large number of users, busy service transactions, and complex cloud-based service architecture bring a series of problems to Jiangsu. It takes an average of 40 days to launch a new service and 22 days to fix a problem.
To surmount these challenges, China Mobile group decided to build the Panji PaaS platform based on Docker and Kubernetes and the Panzhou DevSecOps platform, which is an integrated R&D platform for digital intelligence transformation, and promoted the platform to 31 provincial companies in China.
Huawei’s DevOps model supports continuous innovation through transformation of organizations, processes, technologies, and tools. Under the guidance of the group’s unified planning, Jiangsu and Huawei started the cloud-native digital transformation journey in the BSS system (CRM) project.
In the CRM project of Jiangsu, Huawei has explored the following DevOps technologies:
1. Divide the CRM core service system into three domains, 14 subdomains, and 8 centers based on the domain-driven design (DDD) and microservices. Deploy the whole core service system on the Panji platform of China Mobile and containerize 100% of the core services with microservices, enabling rapid and continuous delivery.
2. The organization of “Product Management Office” in Jiangsu Branch is changed from “cross-group coordinated delivery” to “end-to-end delivery by product manager team” therefore, product managers can re-allocate organizational resources from the perspective of customers. The product manager is responsible for resource application and allocation, as well as for the end-to-end business. This changes the management mode of chain interaction between end users, business departments, IT systems, and vendors, breaks down barriers, and supervises the daily implementation of agile management methods by using agile self-organizing management.
3. Panzhou DevSecOps platform is designed to build continuous integration (CI) hierarchical pipelines; balance test feedback speed and coverage; unify product control and eliminate manual operations, and ensure that all deliverables transferred to the continuous deployment (CD) pass quality gate. By connecting all engineering capabilities through the pipeline, the platform greatly improves the launch efficiency; 86% of versions can be launch before 24:00pm, instead of staying up all night long, which greatly improves O&M personnel’s happiness.
4. Apply the canary deployment on the Internet to all BSS domains. This starts by selecting a city or channel to verify the release in advance. After the verification is complete, the solution in the specific province will be released. The canary release not only improves the quality of new services, but also enables some users to experience the convenience and benefits brought by new services in advance. The unaware traffic switchover technology and rolling release mode are used to implement upgrade and release of new service features during the day without affecting services on the live network.
5. To improve the test efficiency and effectiveness, test activities are moved leftward to requirements and development. To improve the delivery quality and make continuous improvements, test activities are moved rightward to the O&M of the production environment. At the same time, the proportion of automatic tests is continuously increased, covering the whole process of software development and delivery, thus forming continuous tests.
6. Change thinking, apply intelligent O&M capabilities, restore user tracks and detect exceptions through multi-dimensional indicator collection and monitoring, service call chains, and quickly detect, locate, and solve problems. Apply intelligent customer service to improve frontline support experience.
The practice has achieved very good results. In 2022, Jiangsu launched 1,300 new functions, with more than 14.07 million new individual users and more than 70 million new IoT users. The delivery period of new requirements is shortened from 40 days to 21 days, which is improved by 50%. The O&M fault self-healing rate reaches 20%, and the problem resolution period is shortened from 22 days to 10 days, an improvement of 49%.
Through the transformation of DevOps agile delivery and intelligent O&M, China Mobile and Huawei continuously improve the delivery quality and efficiency of IT products and services, deliver improved new service features to customers with shorter TTM, create greater business value, and have successfully entered 2023 GLOTEL Telecoms Excellence Award shortlist.