Service
Cloud
Enabling Enterprise Agility & Resilience with a Robust Cloud Strategy
Enterprise Solutions
TCS Enterprise Cloud: Unlocking Exponential Business Value
HIGHLIGHTS
- While the inherent appeal of cloud along with the advancements in AI and automation has accelerated migration of machine learning pipelines from on-premise tools to cloud platforms, this also brings to the fore the challenges involved in moving in-flight data science work.
- Re-writing legacy code, preserving ongoing experiments, and adapting to new orchestration and governance models are some of the key obstacles that can disrupt productivity and research continuity if a migration is rushed.
- A good migration strategy should start with a comprehensive asset inventory, build a shadow environment, introduce cloud-native experiment tracking, and validate results side by side to ensure seamless transition and long-term success for enterprise AI initiatives.
On this page
Introduction
Enterprises shift from legacy on-premise tools to modern cloud machine learning (ML) platforms for agility and resilience.
Enterprises are rapidly modernising their data science environments, moving from traditional on-premise tools like SAS, SPSS, and RStudio Server to cloud platforms such as AWS SageMaker, Azure Machine Learning, Google Vertex AI, or Dataiku Cloud. The promise is appealing—elastic compute, lower infrastructure costs, richer machine learning operations (MLOps) capabilities, and the ability to collaborate globally. But what often goes unnoticed is the complexity of migrating in-flight ML pipelines, where data scientists are mid-way through feature engineering, A/B testing, model iteration, or experiment tracking in addition to data prep challenges in terms of hundreds of undocumented macros and intermediate tables. Data scientists spend a lot of time in these activities. Hence, they do not want to lose track of experimentation and want to continue from the same state after migration.
Migrating such active pipelines isn’t just technical. It directly impacts productivity, continuity of research, and business-critical model delivery timelines. It poses real challenges for businesses but a structured approach will ensure seamless migration without affecting in-progress analytical work.
Why move to cloud ML platforms?
Why cloud-based ML platforms
Let us start by exploring a more fundamental question: Why are organisations moving towards cloud based ML platforms? Organisations adopt cloud ML platforms for several reasons:
- Cost and scalability pressures: SAS licensing, on-prem hardware, and scaling bottlenecks push enterprises to shift towards pay-as-you-go cloud compute.
- Artificial intelligence-machine learning (AI-ML) capability gaps: Modern frameworks, distributed training, AutoML, graphics processing unit (GPU), and tensor processing unit (TPU) options are readily available on cloud platforms.
- Operational efficiency: Built-in experiment tracking, feature stores, pipeline automation, and deployment tools accelerate the ML lifecycle.
- Global collaboration needs: Cloud notebooks and shared workspaces allow distributed teams to work on the same experiments from anywhere on the globe.
These advantages make cloud ML a natural next step with the caveat that migration should be done right.
Challenges
Migrating ML data pipelines comes with many challenges.
Some of the key challenges include:
1. Rewriting into cloud-native code
Most on-premise teams rely heavily on SAS macros, SPSS flows, and R scripts. Moving these to Python, scikit-learn, SparkML, or SageMaker, Azure, Vertex training jobs often isn’t straightforward. Many SAS statistical procedures don’t have exact cloud equivalents. Recreating years of accumulated logic becomes a lengthy, iterative project.
2. Complex feature engineering embedded in legacy systems
On-prem solutions often hide layers of preprocessing within macros, scheduled Unix scripts, or extract, transform, load (ETL) jobs that nobody fully documented. When cloud migration teams start extracting pipeline logic, they often find that intermediate feature datasets, temporary tables, or data cleaning rules are embedded deep inside legacy code.
3. Risk of losing in-progress experiments
This is the most sensitive issue. Data scientists may be in the middle of:
- hyperparameter search jobs
- exploratory data analysis
- multiple model versions
- feature engineering iterations
- partial notebook progress
If migration is not performed carefully, weeks of research can be lost and teams must restart from scratch, delaying business outcomes.
4. Shifting from on-prem schedulers to cloud orchestration
Jobs that ran using cron, SAS DI Studio, SPSS Modeler, or R shell scripts must be refactored into Step Functions, Azure ML Pipelines, Vertex Pipelines, or Dataiku scenarios. Dependencies, retries, security, and monitoring behave differently in cloud environments.
5. Governance, access, and compliance redesign
Cloud migration forces a shift from simple firewall-based control to identity-driven access policies, workspace isolation, encryption, and auditability. This impacts workflows, data access, and pipeline structure.
Solution
Empower teams with a gradual, strategic migration to maximise productivity while you modernise.
The key is to avoid a ‘big-bang cutover’. Instead, adopt a controlled, staged migration that keeps data scientists productive throughout the transition. The following steps provide a simple approach to migration while retaining the thread of continuity for in-flight data prep and modelling jobs.
1. Conduct a comprehensive asset inventory
Before touching the cloud, catalogue everything:
- SAS, SPSS, R scripts
- experiment logs
- macro dependencies
- intermediate datasets
- model versions and metrics
- notebook states
Automated scanners can extract metadata from SAS directories or R project folders to map pipeline dependencies.
2. Build a cloud ‘shadow environment’
A shadow or dual-run environment helps data scientists continue working without interruption. This includes:
- containerised R or SAS-compatible notebooks where possible
- synchronised data from on-prem to cloud storage
- mirrored folder structures and environment variables
- access to cloud compute with minimal workflow changes
For example, a shadow Azure ML workspace for an enterprise lets its data scientists open and run their existing R notebooks via cloud compute while engineering teams rebuild pipelines behind the scenes.
3. Introduce cloud-native experiment tracking
Experiment tracking is essential to ensure that no work in progress is lost. Tools like MLflow, SageMaker Experiments, Azure ML Experiments, Vertex AI Experiments or Dataiku’s built-in versioning can be used to track experiments, models, metrics, feature versions, intermediate outputs, and parameters. After migration, data scientists can resume exactly where they left off.
4. Migrate feature pipelines with versioning
The following cloud-native tools can be used to version feature outputs and capture mid-cycle feature engineering works.
- AWS Feature Store, Azure Feature Store, Vertex Feature Store
- Delta Lake or Iceberg for dataset versioning
- Cloud-based orchestrators for scheduled data processing
5. Validate through side-by-side runs
Run cloud pipelines in parallel with on-prem systems, comparing model metrics, performance, statistical equivalence, runtime and cost. Once outputs converge, on-prem workloads can safely be retired.
Conclusion
Migrating ML workloads, especially in-flight pipelines, requires a structured strategy.
The strategy should preserve experimental continuity, data lineage, and data scientist productivity. When done well, cloud platforms unlock faster experimentation, richer ML capabilities, and long-term sustainability for enterprise AI programmes.