With the rise of Microsoft Fabric, I’m often asked: “What does that mean for Synapse and Data Factory?” I’ve pondered this question and researched what Microsoft are saying publicly and where the shifts might be. With the lack of a crystal ball, I can’t predict the future but it’s important to prepare for when (and if) Microsoft outline their roadmap.
How did we get here?
Azure Data Factory (ADF)
GA: 2018
Features: 90+ connectors, 39 activity types, 3 integration runtime types, 72 templates
Summary: Fully fledged enterprise scale ETL and orchestration cloud hybrid product
Portal: https://adf.azure.com/
Active development: Small changes only
The author tab offers CI/CD Git integration, drag & drop development, and parameterization to allow metadata lead orchestration and customisable ETL processes to meet nearly every business need.
The monitoring tab offers a one-stop shop for all pipeline and trigger runs, providing clear visuals of each run’s start, run, duration, and parameters in either list or Gantt view.
The manage tab allows factory-level settings configuration, from linked services connectors, git config, ARM template, global parameters and airflow management.
The learning centre offers tutorials, videos, templates, SAP knowledge centre, and links to all ADF related blogs and feedback.
Azure Synapse Analytics
GA: 2022
Features: 90+ connectors, 39 activity types, 3 integration runtime types, 72 templates, SQL Datawarehouse (newly named dedicated SQL pool), Apache Spark pools
Summary: With (almost) all the same features as ADF, Synapse adds the power of SQL pools, and Spark pools so instead of having 2 or 3 different Azure resources (SQL pool, Databricks and ADF), Synapse offers a single resource.
Portal: https://web.azuresynapse.net/
Active development: Small changes only
On top of the author (renamed Integrate), monitor, and management tabs offered in ADF, we gain two additional tabs, but lose the learning centre:
The data tab offers a location to mount SQL databases, Lake databases or external data (blob storage, Cosmos or Data Explorer).
The Develop tab offers the ability to write or import SQL, KQL, Spark Notebooks or dataflows, allowing queries and transformations to be written within the ETL engine.
Microsoft Fabric
GA: 2023
Features:
Summary: Fabrics is the first SaaS AI-powered, highly scalable ETL and report platform.
Active development: A huge number of new connectors, features and tools every month.
Portal: app.fabric.microsoft.com
The Fabric Portal introduces workspaces to add and write, pipelines, dataflows, eventstreams, experiments, KQL databases, Lakehouses, ML models, notebooks, reports and warehouses.
The OneLake data hub tab shows all the data in the workspaces, from databases, to Lakehouse and beyond. This serves as an analogue to the Data tab in Synapse.
A completely redesigned monitoring hub allows us to track runs of pipelines, dataflows, Lakehouses, notebooks and semantic models, allowing you to monitor the orchestration, analytics engine and report refreshes all in one place. Unlike ADF and Synapse, this new hub allows monitoring across all workspaces, streamlining the management of multiple workspaces.
How do they compare?
ADF and Synapse share a similar feel, right down to their colour palettes, often making it easy to mistake one portal for the other. Fabric, however, feels fresh and different, though you can still spot traces of ADF and Synapse if you squint hard enough. It’s as if the brilliant minds behind Power Platform had created ADF or Synapse first.
Every time you open Fabric, there’s more to explore – more buttons, more connectors, more activities, more everything. The speed of these changes is honestly breathtaking. As Fabric is SaaS rather than PaaS, you do lose some of the management ability, which is either on purpose (farewell, linked services and datasets) or still in development, only time will tell.
Where are they going?
I don’t have a crystal ball, and Microsoft certainly won’t let me peek inside theirs. They play their cards close to their chest, but I can share some insight that might shed some light on the situation.
Historically, when one Microsoft product team support multiple products, they tend to phase out older ones where possible. From what I can gather, there’s only one team handling these orchestration products – the Fabric product group.
Azure Data Factory
Microsoft deployed an ‘Upgrade from ADF’ into Fabric in Q2 FY2024. This suggests they aim to make Fabric equal to or better than ADF in most, if not all regards. The goal seems to be making migration or mounting a simple and quick decision. Microsoft has already published a comprehensive list of features and how they align.
Does this mean ADF will be sunset? Not yet! There is no official word on ADF’s potential sunset. If Microsoft ever decides to do so, they’d likely provide suitable notice and clear migration steps.
Unlike the transition from ADFv1 to ADFv2, where the writing was on the wall from the name alone, it’s harder to judge Microsoft’s direction with these three products.
Azure Synapse Analytics
As soon as Fabric was announced, engineers began writing off Synapse. This reaction was so strong that the CVP of Azure Data Analytics wrote a blog stating “Microsoft has no current plans to retire Azure Synapse Analytics. Customers can continue to deploy, operate, and expand the PaaS offering of Azure Synapse Analytics. Rest assured, should these plans change, Microsoft will provide you with advanced notice and will adhere to the support commitments in our Modern Lifecycle Policy in order to ensure our customers’ needs are met.”
There’s a comprehensive blog on migrating from Synapse dedicated SQL pools to Fabric Warehouse. Spoiler Alert: it’s not as simple as ADF.
Does this mean Synapse will be sunset? Not yet again! While it is likely to happen one day, Microsoft will again give plenty of notice. They need to build out all the capabilities in Fabric before pushing customers over, otherwise they’ll abort and look elsewhere.
Should I develop in Fabric now?
It depends! Do you want to use the latest and greatest and not have to worry about migrating for years to come? Are you happy with the limitations, known issues, and constant changes? If so then, go for it! Fabric is extremely powerful and scalable. The numerous talks on fabric at SQLBits 2024 and Fabric in Vegas aren’t happening for nothing – it’s truly impressive and really shaking up the analytics space.
Should I develop in ADF or Synapse?
Yes! Whilst there is no sunset date for either, developing in both is still worthwhile. They’re production ready and ideal for different use cases. Given their large customer bases, if Microsoft ever sunsets either, they’ll likely have great migration support. They don’t want a reputation for abandoning products after a few months without support.
Should I migrate now?
Fabric offers many benefits, and migrations don’t have to happen all at once. It could be done gradually, piece by piece and if you are not sure where to start we can assist with planning and executing that migration.
Will I need to migrate in the future?
We can be certain that it will happen eventually but for now, ADF and Synapse are still widely used, so they won’t be sunset anytime soon.
Microsoft wants people to leverage Fabric to its full capacity. They’re rapidly removing adoption blockers and smoothing the path for potential migrations. With heavy investments in AI and Fabric, and all the buzz surrounding them, there’s no sign of slowing down.
If you would like support migrating from ADF or Synapse to Microsoft Fabric reach out to our experienced consultants to find the right roadmap for you at info@coeo.com.