Last week I attended DATA:Scotland in Glasgow, which I’ve been looking forward to since I signed up and it certainly didn’t disappoint. The day started with a commute to Glasgow in glorious Scottish sunshine, where I joined over 1,000 other attendees at the University of Strathclyde‘s Technology & Innovation Centre (TIC). This is a fantastic venue and the campus has changed a lot since I was a student at Strathclyde Uni in the 90’s! The auditorium was packed for the welcome session and there was a buzz of excitement around the exhibition hall as I topped up the caffeine levels for a full day of learning.
I had already picked out some Microsoft Fabric related sessions kicking off the day with Kamil Nowinski talking about Delta Lake Tables 101, showing the benefits of Delta tables over parquet data lakes. Improved performance, schema evolution and the time travel feature of Delta tables really stood out. You can follow Kamil for more great content here.
Next up was the inimitable Rob Sewell and a session describing the features of the new sqlcmd (go-mssqldb) a cross-platform utility compiled in GO. Rob showed how to use this to quickly create a SQL development environment. He also showed how to use the Data API Builder to create REST and GraphQL endpoints on your SQL databases and the Azure Developer CLI (AZD) to easily take you from a development environment into Azure. The session was definitely outside my comfort zone but I learned some new tools like Swagger that allow you to easily build API calls, which is what I love about these events.
A quick break then straight into Your First Fabric Lakehouse Implementation presented by Bas Land where he showed how he has transitioned from building SQL data warehouses to lakehouses in Microsoft Fabric. Bas stepped through the concepts of tenant design, setting up a Fabric Capacity, creating domains and workspaces and setting up and customising your own environments. He also described a case study where he implemented a medallion architecture on Fabric, bringing this to life with demos and examples. This was a great session for those embarking on their first Fabric Lakehouse project.
After that we had a break for lunch and a chance to wander down memory lane through the university campus, amazingly the sun was still shining as the pictures prove!
Straight after lunch is a tough gig at any conference, but I really enjoyed Barry Smart‘s session on Microsoft Fabric and Data Mesh. This session showed that Microsoft Fabric is heavily influenced by Data Mesh theory, and it’s the result of 20 years of data platform evolution. A Data Mesh aims to build a healthy sociotechnical environment using 4 key principles: Data as a Product, Domain-orientated ownership, self-service data platform and federated governance. Barry explained that as there’s less work to maintain the data platform, data teams should adopt a product mindset and create innovative data products to deliver higher value to the business. I’ll be adding Zhamak Dehghani’s book Data Mesh: Delivering Data-Driven Value at Scale to my reading list after this.
A quick wander around the exhibitor’s hall to pick up some highland coo stickers was followed by Toni Baron’s session talking about the Importance of Change Control in Report Development. Data is the backbone of decision-making, so change control is essential to maintaining trust in your data, which is very hard to regain once lost. Implementing change control increases the quality of your reports and data and should not be just the domain of engineers. Toni explained that using Power BI Deployment pipelines is a great start to implementing change control and a way to track changes as you move through your Dev, UAT and Prod environments.
Another quick break to refuel with a Tunnocks Teacake and onto Governing Your Data on Fabric presented by Sarath Sasidharan and Gabi Münster from the Microsoft Fabric CAT team. This session discussed the objective of data governance; data management and protection, data discovery and data confidence (trust), and how you can use Microsoft Purview to encourage data discovery, trust and use through endorsements and lineage. The session showed how to use Purview’s Data Map to define technical, business and data domains. We also saw how to grant control plane and data plane access, define sensitivity labels and assign policies to these. We also learned that Fabric can integrate with 3rd party governance products, so you can incorporate existing solutions into your Fabric environment.
The final session I attended was a look at the Second Visualisation Tool from Microsoft presented by Brian Bonk and Devang Shah. This second tool is Real-Time Dashboards, announced this year at MS Build in May. These reports are built on top of Kusto data to provide time series reporting and are available in Azure Data Explorer or as a native Microsoft Fabric service. These reports were developed by Microsoft to visualise telemetry data and provide almost zero latency on real-time data. The visualisations are rendered by the KQL query language and provide lightning-fast query times on huge amounts of data. The session concluded with a light-hearted (!) comparison to Power BI and the features and costs of each solution. Real-time dashboards are great for filtering on dynamic time-range data, and you’re only charged for the Eventhouse costs but the downside is the limited gallery of visualisations compared to Power BI.
I also learned that I had been using real-time dashboards in Database Watcher for Azure SQL without actually realising, so came away with some great ideas for extending these and creating my own dashboards in Microsoft Fabric!
At this point my head was full of new-found knowledge and great ideas, so there was only one thing left to do: discuss the day’s sessions over a beer or two while soaking up the early evening sunshine in Glasgow’s fabulous West-end (yes seriously, it was still shining).
In conclusion, Data:Scotland is a fantastic event and serves a large, vibrant data community in Scotland. Although I was focused more on Microsoft Fabric content, the range of sessions and the depth and quality of the speakers meant there was something for everyone to take away. As with most conferences of this size, there were sessions I couldn’t attend due to scheduling clashes, but I’ve subscribed to the Data:Scotland YouTube channel and will be keeping an eye out for these when they become available.
Finally, a massive thank you to the organisers, volunteers, and sponsors for the hard work making this event happen and ensuring it ran smoothly on the day, but also keeping the entrance cost free so that it’s accessible to all. I will definitely be recommending that my friends and colleagues attend Data:Scotland next year and beyond.