ABSTRACT

Disruptive technologies, product innovations, new distribution channels, and rising competition are redrawing the contours of the banking landscape.

To keep pace with this rapidly evolving landscape and remain competitive, banks have sharpened the focus on regulatory compliance, business processes, and customer experience. Moreover, the COVID-19 pandemic and its effects, geopolitical developments, and stringent government policies make it imperative for banks to be more agile. Banks are looking to improve data and analytics quality, reduce cycle times for new analytics, and thus increase productivity. 

Transforming business processes and regulatory compliance while keeping agility and innovation in mind are the top priorities for banks with respect to data management. DataOps can help banks be agile and overcome the complexities and challenges while delivering strategic choices for business model modernization, new revenue lines, and regulatory adherence. We discuss how DataOps-powered solutions can enable banks to optimize strategic data management and enhance efficiency while remaining competitive.

A TREASURE TROVE OF INNOVATIVE DATA INITIATIVES

Banks are data-intensive organizations where data moves through various points such as front-office, middle-office and back-office transactions, risk and settlement, and operational systems.

Most of the existing data ecosystems are built on siloed legacy infrastructure and address the specific business problems of the past. They lack agility and obstruct banks and financial institutions from achieving the full potential of digital transformation. These legacy systems rarely offer the scale needed for the increasing data inflow from diverse data sets, and create room for innovation.

Data initiatives like data commercialization and data democratization present new business avenues in the finance sector. However, enabling capabilities like fraud detection, developing risk models and benchmarking the customers on changing business scenarios, deriving pricing models for credit card holders, and so on, require multiple iterative processes. Moreover, these data initiatives involve complex data integration, data trustworthiness, and applying artificial intelligence (AI) and machine learning (ML) to produce a valued outcome. While doing so, time-to-market is one of the key aspects that determine the business value derived from these data initiatives. 

Banks face multifold challenges in capturing the right data from legacy systems or adding new data and having the right expertise to work with. The whole process undergoes multiple iterations by data scientists and data engineers to get meaningful results.

Adopting DataOps as an approach to data initiatives can help banks address these challenges. It leverages DevOps practices, along with data, for agile and automated data processing and delivery. A DataOps-backed data initiative ensures that data is turned into meaningful insights rapidly and at scale, while ensuring repeatability of the process. 

UNDERSTANDING THE DATAOPS FRAMEWORK

DataOps involves streamlining and automating the data lifecycle.

In Figure 1, we present a framework that applies a comprehensive approach taking into account the bank’s existing ecosystem as well as considering the right balance of new technology and data platforms required. For example, a single product vendor or cloud provider can adopt a tool that stitches together open-source and commercial components.

Figure 1: DataOps framework

The scope of DataOps includes building and running data pipelines for ingestion, data engineering, and the feeding of data to downstream systems for advanced analytics. The data pipeline represents a supply chain that processes, refines, and enriches data for consumption by various business users and applications. Building such a pipeline involves processes like acquisition, processing, and preparation, and governance of data, as well as building advanced analytics for authorized data consumers.

The objective is to develop new functionalities with self-organizing business analytics teams that build functional code and test it thoroughly in short sprints. Orchestration tools help automate the processes and are used to run the tests before and after every stage in the pipeline. These tools are responsible for enabling the data environment by coordinating with data and developing the code, technologies, and infrastructure. We list a few benefits of adopting a data management framework:

• Faster time-to-market and increased business agility

• Better business decisions with more business insights

• Greater collaboration and cross-functional alignment

• Improved regulatory compliance and security

• Better operational efficiency and cost reduction

• Greater business value by industrializing the data analytics pipeline and operationalizing data science

BUT, THERE ARE SOME CHALLENGES TO ADOPTING A DATAOPS FRAMEWORK

Banks and financial services firms can encounter some technical and behavioral challenges when embarking on a DataOps journey.

Technical challenges: As DataOps is not a technology innovation but rather a process innovation, establishing a formal process of using DataOps tools and software effectively can be challenging.

Behavioral challenges: People’s behavior toward changing the way they work to adopt agile development is not easy. Getting business and technical users to buy into the process can also be difficult.

Since it is an evolving concept, organizations must consider these technical and behavioral challenges when establishing DataOps practices. This will prevent failures in the longer term. 

To implement DataOps practices, organizations should start with a few use cases and scale up gradually so that ongoing processes remain undisturbed. To overcome DataOps adoption challenges, banks should focus their efforts on three areas: team structure, tools and technologies, and processes and methodologies.

A collaborative team structure always helps

Collaboration within and across various teams is an integral part of the DataOps framework. Along with the traditional roles of data modelers, data architects, and data engineers, new responsibilities must be added. These are data scientists, data analysts (for self-service and predictive analytics), and IT operations specialists, as DataOps emulates DevOps with continuous integration (CI) and continuous deployment (CD) for sprints and minimum viable product (MVP) releases. The product owner plays the critical role of being the owner of the data product. The scrum team comprises all the above-mentioned roles under the product owner to carry out the product development activities.

Tools and technologies are key

DataOps brings the entire data lifecycle into play, from data preparation to data delivery, and involves the interconnected data analytics team and data-related operations. DataOps technologies are a mix-and-match of data engineering and DevOps tools. Depending on the architectural pattern, a data pipeline can be built by utilizing various data management and engineering tools. Metadata tools for metadata-driven data management and governance play an essential role in DataOps deployment. In addition, test automation tools are required to automate the test functionality and improve the quality of the code. Hence, code repositories are essential, especially in agile development environments with stringent version control procedures and continuous integration processes. 

Don’t forget processes and methodologies

The DataOps process is a series of steps, methodologies, guidelines, and metrics that have to be monitored. People are ineffectual without practices in place to support their decisions. There must be standard processes for data requirement gathering, data pipeline development, testing and tuning, orchestration and automation, production move, and continuous iteration. Moreover, the teams must be trained as part of the DataOps framework.

Lately, several product vendors are offering tools that address some or all of the capabilities across the DataOps lifecycle. DataOps products provide solutions to orchestrate people, processes, and technology for delivering a trusted data pipeline. Several cloud providers also offer DataOps services by assembling different types of data management software into a single integrated environment.

TRANSFORMING DATA MANAGEMENT PRACTICES

Understanding the current state of DataOps dimensions – people, processes, technical landscape – and continuously improving in a particular discipline is essential for beginning the DataOps journey.

Banks must benchmark their DataOps maturity against industry best practices, identify gaps, and consider appropriate deployment options from on-premises, cloud, or hybrid environments. Along the journey, banks must move up the curve by addressing other gaps in functionalities and building capabilities for required data pipelines, orchestration of multi-layered data architecture, and team collaboration for agility. 

By adopting DataOps, banks can gain speed and quality of data-driven products and services, and foster a culture of continuous improvement. Banks must embrace DataOps and embark on data initiatives to stay competitive.