Hi everyone. Welcome to figuring out fabric where I ask dumb questions and you get smart answers. And I've got on the show, uh, Fatima, who is a data analyst and consultant in Stockholm. She has experience with ETL data architecture and feels comfortable both with fabric and data bricks. And so I'm really excited to have her on because we're going to get to talk about data pipelines, which is something I've had to learn in the past four months and have no experience with. So I appreciate you coming on Fatima. Thank you so much. Thanks for having me. Yeah, so um why don't you explain what a pipeline is for the audience for people who don't know? Because I think a lot of people have probably heard of it, or I had heard of Azure Data Factory before, but it may not be clear what it actually is. Yeah, yeah, so data factory. If you had any sort of past experience the way in Microsoft technology working on other data platforms that I think of it as the fabric offering like SSIS or Azure Data Factory or even Azure Synapse pipelines. If you have no experience and you don't know what that is, think of it as a low code ETL tool. E for extract and then sort of transform load or even Elt where you have activities in a graphical interface that you want to chain together or group together in a logical way and you want to be able to set that on a schedule so you can automate it and have it run let's say once every night or something like that. So yeah that's how I would explain it. Yeah, that makes sense. And I guess, um, one of the things that's kind of challenging with fabric is we have, depending on how you count like three or four or five or six different ways to move data around. the three ETL tools that come to mind for me immediately is. Gen two data flows, fabric pipelines and notebooks. And it's not, I don't think it's entirely clear off the right off the bat or immediately. which one you should use and when so in your opinion, when does it make sense to use a pipeline? What are they good at? So I think for me, uh any tool that you could use is good. Start by thinking what you actually know how to use and that you feel comfortable with. So you can, of course, do many weird and wonderful things if you uh are very comfortable writing a lot of code. For me, and the kind of teams that I work with, or if I do something for a client, I really like pipelines because it's low code. can see it with your eyes and understand what's happening. If I compare it to data flows and the difference in how I have worked with these two different type of pipelines, then a data flow for me, keep in mind it was some time ago that I used to work a lot in Power BI, but Then the use case for me was, okay, so I wanted to do some low code transformations in Power Query. And then maybe sometimes I wanted to do transformations that I wanted to use in many semantic models or many Power BI reports even. Perhaps transforming some sort of dimension from a data warehouse that wasn't up to scratch. And I would use Power Query to transform it to do all of those steps really easy. But if I want to reuse that, then it was really, really good to do that in a data flow. and then you could use that as a data source to many power query or semantic models for the reusability. I wouldn't necessarily uh do transformations in a fabric pipeline. This would be orchestration on a higher level, like your whole data pipeline that would normally involve something like ingesting data, connecting to something through a pipeline connector, and then you wanna ingest something from let's say uh an Azure SQL database and then you might want to run a note. Once you have ingested data, you might want to do something with that data. You want to automate that a notebook should run where you have some transformations and then maybe you want to load that into something else where your Power BI report would use the data as its source. um I think of it as more as orchestration and I know that was the the best practice also for their sort of predecessor studies. If we are talking about Azure Data Factory, then we would do transformations maybe code based or in the low code offering of that, we would have the transformation as an activity. But that can be quite error prone in the past. It's something that has driven quite a lot of costs. So for me, my go-to is to just use it as orchestration and then do transformations elsewhere. Yeah, I think that's an important highlight because I think, you you mentioned ETL and if you're a new person coming to this, you might think ETL is ETL is ETL. But for me, um, as I've started learning how to use fabric, I think, you know, when we talk about data flows, it's really heavily focused on the transformation part. Like they are working on expanding the E's and the L's. Mm. It used to be that it only had one L is just Power BI. um But it's all about transformations. Sometimes I joke that if you can pay a summer intern to do it manually in Excel, you can automate it with Power Query. And that feels like the list of things you can do. Whereas what I'm learning with pipelines is, in my opinion, you may have a different opinion, it's good at two things. It's really good at two things. One is data movement. The E and the L, it's got... a lot of options for loading the data from different sources and putting it places. And if you're a small business, you don't care about that because you probably have a handful of sources at most. Most of your data is either in CSV, Excel or SQL and you're loading it to one place. But if you are an enterprise or you're in Azure, Well, your data is scattered in all these places and you've got all this software you're pulling from. And so pipeline start to make a lot more sense there. think, um, the other thing is like you mentioned orchestration and I think that's the big selling point because, um, data flows are very linear. Um, whereas pipelines you can say, all right, if it fails, do this, if it succeeds, do this, I can send an email of something screwed up. I can, you know, schedule all these parts. I can do a for loop. and wait until everything's done in that loop before I move on to the next stage. You don't have any of that control with data flows. So for me personally, I mean, that's where I see a lot of the value. Yeah, for sure. I agree 100%. And ah I say it was some time ago that I used data flows because I really, really like pipelines. And although I could achieve all of those things in code, just I find them really, really, I find them a really, really like strong thing to have in your architecture because it's easy to onboard people, you have so many options for, as you say, like monitoring and just routing. Okay, so if this uh the is successful, can root it and something should happen. If it's not successful, can root it and something else should happen. It's really good. Yeah, you can look at the, it's super easy to look at the logs and see what step failed in a pipeline. If you have a notebook, like good luck, like you can look at the session history for a notebook running and go cell by cell, but it's not the same as just seeing that table and being like, okay, this one failed. Here's the error message. Here's why all that kind of stuff. um I think if we're also talking about data movement, then we have an activity called copy activity, and that's a really, really like quality assured performance way to ingest data. So you don't have to be this code whiz that's really, really clever with things. So you just have like your source tab that's really easy to configure where you set up like, okay, all of the details about these, for example, SQL database where you want to get data from. And then you have your destination tab and you could just like easy to understand input, whatever you wanted to understand. And you can see that the data is landed there. And it would also give you when you run that activity, you would be able to see how many rows did I write, sorry, read from my source and how many rows did I write in my target. So it gives us a lot of things for free that would require some code or maybe logging in code in order for us to get all of that information. Yeah. And true, true to form with fabric, there's three different ways that you can use that copy activity and they don't make it easy to talk about it because you can just add it normally or you can use the, I think it's the copy wizard or something when you're setting up a new pipeline, which is super convenient. And then you can use the copy job, which is different than a copy activity, but it uses it behind the scenes, which is super, super convenient. And I love the UI, but it makes it very difficult. to talk about all this stuff sometimes. Because there's always like five different ways to do the same thing, unfortunately. Yeah, that's very, very true. But I'm talking about the copy activity. That's my favorite. Yeah. um how did you get into this space? Because one of the things I'm learning is that there's all these different roads that lead to Fabric or lead to Databricks, and people have these very different backgrounds. So how did you get into data? Or how did you end up working with pipelines and Fabric and a little bit of Databricks and all that? ah Well, I'm one of these people that made a career change when I was like 30 plus. So I had my own career that was on the business side and I was working as an executive assistant and just being involved in different things around like running companies and supporting people that run companies and entrepreneurs and things like that. And I started thinking about it. Okay, so what is it actually that I enjoy doing? uh This data space seems fairly interesting and I had a keen interest in using data to make business decisions, not guesswork, and without realizing it I was actually working a lot with business intelligence because I was producing different reports and I was a super user in Excel and all of these things. um So I needed some new inspiration and a new direction in life. So actually I started studying some C-sharp coding and then I actually took a career break, started studying and that was like, the studies were covering most things in business intelligence. So it was like a structured two-year program, but it was a lot of like old school data warehousing, Kimball modeling, like SSIS packages and then this old school world. And then when I started working, although the training was full stack, actually started working as a Power BI specialist because Power BI was quite new. So if you knew some Power BI, you were like so high in demand. So I became this Power BI specialist for some time. And then I quit that, joined the consultancy. And when you're a consultant, then, you know, every project will throw you into different things. So that's why I've been working on. anything from like Power BI to building Synapse platforms to Databricks and now Fabric. And I think for me, because I'm curious by nature and I also have this past career, then joining together technology and the interest of business and being able to talk to different stakeholders in the organization has sort of led me more. towards the architecture side of things nowadays. I'm very organized. So I really, really like to invent things from nothing and plan out how everything should work together and so on and so forth. Yeah. Yeah, that's honestly really impressive because I think a lot of people try to maybe pivot mid-career and it can be challenging. uh And I think the path you took makes a lot of sense because a lot of people are, a lot of techies look down on business users. And what I try to remind myself is those people are the domain experts and they can tell you whether your model is useful at all. or not, just because they maybe don't know C sharp doesn't mean they aren't experts in their own kind of regard. Um, but yeah, I get the consulting thing. If you want to supercharge your career or work for a consulting firm, cause you'll get thrown at stuff that you have no experience in and you'll be expected to figure it out. Um, it's, it's a little rough, but it's a great way to learn. I'm an independent consultant and it's different beast because you don't want to sell yourself on stuff you don't feel like you're an expert in and so Learning stuff is hard, but I'm lucky that this current project I Told them I don't know ETL. I'm not experienced in it they're like well You're already in our system. You already have credentials and you seem to know what you're talking about So I'm it's very rare to find a customer that'll pay for on-the-job learning. So I'm grateful for that Press your heart. um So what are some challenges you've run into with implementing pipelines or using pipelines? So in this current project that I'm working on, we tried to set up more sort of an enterprise pattern in fabric using only fabric components as far as possible. So we have succeeded in that apart from Key Vault in Azure that we are still using. At the foundation of this platform was actually fabric pipelines because what we wanted to do was to implement a metadata driven architecture. So what that means is that you try to create reusable pipelines. So instead of having pipelines that are custom made for every single table or even every single SQL database, then you try to work with metadata that you have somewhere and then you have a reusable pipeline that is, example, you could use it for any SQL database that you might want to ingest. So once you have built that pipeline, you could use it for 100 different SQL databases. But how do you make it reusable? Well, you make it reusable by not hard coding anything. Instead, you put in a parameter where you, for example, have your database name or your server address. That's just a parameter. And those parameters, parameter values you could have in, it could just be a JSON. file or something really simple. In our case, we set up a small Fabric SQL database and these two uh artifacts sort of work together so you could have very few pipelines and in just a lot of data sources. In doing this, I relied quite heavily on some of the things that I was used to being able to use in uh Azure Data Factory that wasn't quite uh ready in Fabric. So I noticed that some of the parameters that I needed to use, like some of the fields, they weren't prepared yet for dynamic content. So that made things a little bit uh interesting, should we say. And something that was new to me as well, it was the fact that fabric is very centered around me as an individual. Like because I've worked a lot in Power BI, I understand this whole thing about me as an individual owning different artifacts, but that's not the way it would work if I'm working elsewhere in some other type of platform. So understanding security context for all of these different components in pipelines and also when I wanted to set up in a different environment like we have our dev environment, test environment and production environment. That took quite some thinking and designing and planning to get that working. Also because the support for service principle was limited um in things like notebooks when we started out. So that was another challenge I would say. What else? I really wanted to use schedules, like multiple schedules on the same pipeline. That wasn't possible when we started out. and also using parameters in these schedules because it was an architecture, like a setup that I was used to building in Synapse pipelines and also Azure Data Factory. But things are improving every day, so who knows? I've managed to find some work around. It's gone. It's gone. The first year was rough. think I think anyone who tells you otherwise is lying to you. The first year was rough, but like you know you were there early on with power bi power bi in 2016 was very rough. And so it's a similar kind of kind of thing and it has gotten a lot better. I definitely agree about the frustration about. Okay, this is an Azure data factory, but on fabric pipelines. think to use an analogy, I mean, my understanding is that if you talk in Swedish to someone from Norway who speaks Norwegian, you're mostly mutually intelligible, but not all the words are the same, not all the stuff is the same. And, know, occasionally you can get a faux pas. And so a lot of stuff in fabric, whether it's gen two data flows or pipelines or some of the stuff, maybe with spark or whatever. Mm. If you're coming from Databricks, um, most of it's the same, but there may be stuff that's missing. Maybe some of the different that can be frustrating. Um, I'm, curious about the metadata pipeline stuff. So I've had to implement metadata driven pipelines and it's so nice that I think you, don't know if you saw, but SQL databases went GA at, I think at night. So now we have a GA way, uh, to store that metadata, which is great. Cause like you said, That wasn't there the first year. And so you didn't have that as an option to store the stuff. But the way that I'm doing it is fairly simplistic because we have notebooks to pull data from some APIs and do some work. So I'm just sending that notebook. Here's the table name. Here's the query. Here's some other details. Go do the work. I think something that I've always been confused about the kind of metadata driven pipelines you're talking about. is how do you go fully abstract? Because you talked before about how pipelines are great because they're a low code solution. And I'm used with pipelines to be like, okay, I have that UI where it's like, here's every column, and here's what I'm mapping it to. So what does it look like whenever you've abstracted away the table names, the column names, all of that metadata? How do you work with that? And how do you test that? You get what saying? Yeah, so I think this is where it becomes quite important to have like a good naming convention and work even like when you are naming parameters it's so important to have like names that actually mean something and that you have a clear convention for that. Also in my case we have tried to keep the database fairly small and fairly simple so In the database, for example, I would have a table that's called artifacts to ingest and that will list artifacts to ingest. It will have table names or maybe what else could it be like, is this incremental load? Yes, no, or something like that. So I would have that database table with all of that metadata. And then in my pipeline, I would have parameters. So in my start page of my pipeline, I would have parameters like table name and then I could have a default value for just ease of understanding what's actually happening when I am developing. And I would have, for example, have a parameter is incremental that I would then be able to use in my dynamic expressions. But when I am developing the way I sort of train, like for example, the team I'm handing over to now, the in-house team, I train them to work. in the pipeline and not involve the metadata database until the very end because it is confusing. So you could just hard code. You could have your parameter called table name in your pipeline that you want to use and just hard code the real value with exactly what the name is called in your source database. And then once you understand the pipeline and everything is working as it should, then just make that into like something that is actually sent from the control database. And it's also, yeah. for the column mappings though, are you storing that data metadata is like Jason? Cause I've done the copy wizard and it seems like it's doing that. Cause I'm just curious how, like how you store that mapping information. If you don't have the GUI to manually specify everyone. Do you know what I'm saying? Or do you not get into that level of detail? Okay. working in the GUI and then we have the control database. So we wouldn't be doing any mapping outside of that. It's the control database where we would manually input all of the metadata, like all of the information. So when you say mapping table, I think that's what we have in the control database. So no other artifacts involved but the control database and the fabric pipeline. Yeah. So like just to give an example, I was just wondering how like you might handle that. Okay. If you, because when you were describing it, it sounds like you might have pipelines that are so abstracted or so generic. can point at any table. And so then the question is like, how does it know what column name to map to and from what data type to map to and from, are you having to store that somewhere? Or is it just dynamic enough that it's keeping the same name and data types? Yeah, yeah, yeah. So we are actually, it might be important to mention here that we are working in medallion architecture, like that's our operating model for what we are doing. So that means that we uh have a bronze layer, that's our landing zone, that's where we dump all of the data that we are ingesting from everywhere. And one of the core like foundational things with that is that we do not change the data. So I really don't care what data type I'm getting or if something is missing. I just pull everything into my landing zone. And then, and that's the work that I'm doing with my copy activity. So I write it into my landing zone. And then as I move on in my architecture, I do schema validation and things like that. But I'm not doing that in pipelines. I'm doing that with notebooks because then you have really, really handy things with Delta where it can check any schema conflict for you and that's quite easy to handle in code if you needed to do something. Yes, so that's an important, yeah, now I understand what you mean. Yeah, yeah. So it really is a lot of just the E and L and very minor transformation piece. Okay. no transformation is unless something would break. It's really just orchestration and then the transformation would be in Spark Notebooks. Okay, yeah, that helps clarify and that's maybe a future episode talking about why you made that kind of choice, but we're kind of running on the end now, but no, that explains things perfectly. Yeah, so let's wrap up with a few questions. So I don't know if you got a chance to look them over, but one of the things we ask everyone is what is your most favorite and least favorite fabric features? oh One of the things that I was really impressed about with Fabric, two platforms ago I was working in Azure Synapse, is this high concurrency setting for pipelines that you could use in conjunction with session tags. When we were working in Azure Synapse, we used to have this old time issue that if you wanted to have a for each loop and you wanted to loop through, let's say a list of notebooks and you wanted to execute them, you have this startup time for Spark every single time. So it was like, no, no, no, don't do that. Find a code way to resolve this issue. With the high concurrency setting enabled, you could actually spin up a Spark session and you could run multiple notebooks. So you don't have this Spark tax that you need to restart it over and over again. So I've used that and I really, really like it. And if you combine it with session tags, you can sort of decide what spark session you actually map these notebook runs to. So that's really, really handy. Yeah, it's worth clarifying on that too, is that it's not just the speed and environment thing, but it's a capacity thing. Because um I forget off the top of my head, but the default spark session, I wanna say it either has eight or 12 cores, which is pretty beefy. But if you're on an F64, then I think you get 128 spark cores. And so I forget the math, but basically whenever I was learning this the hard way, If you do a four each loop and you just let it parallelize as much as you want, once it's running like 12 notebooks concurrently, it takes down your whole capacity, unless you know about this feature. So, and you have to turn on per workspace. Yeah, no, that makes sense. I don't know, I think the limit is like five notebooks or something, but I still think it's a step in the right direction there. So I really like that. Least, least, what was it? My least liked feature, least favorite. still a little rough or has some pain points or needs to bake a little bit more, or you're just like, I don't even know why I'd use this. Yeah. well, I think what I know least about is everything to do with like real time analytics and all of that offering. So that's like uh something that's like hanging over me all the time that I need to find the time and head space to go into that because I really want to understand more. I think my main challenge around working in Fabric is just to share the volume of information, everything that's changing. We're all working full time. We have our day-to-day stuff that we need to do. Just keeping up with Fabric blog or if you're on Reddit or wherever you are to get information. That is my main challenge and that's very different to working with uh more mature platforms, I would say, just to share information load and trying to understand So what does it mean for my current project? Should we stop everything now and start handling this new release or this new feature or should we ignore it and take it at a later stage? mean, there's some discussion and thinking around all of these things that need to happen as well. So if you're working in a live project with a deadline, it can become a little bit challenging, I would say. Yeah, it's, it's rough. Like if you want to have an existential crisis, just like watch Ignite and see all the blog posts come out at the same time or whatever. I, I pay for a feed reader called feedly and I, I use, you know, I still read blogs and I do a lot of that, but you know, I think for a couple of years I felt like I wasn't keeping up to speed and, now I feel like I am, but it feels like, uh, water skiing, I feel like at any moment I'm gonna hit a rough patch and just fall off and fall behind and all that kind of stuff. So it's stressful, it really is. um So then the other question is, there's 70 bajillion social media networks, um where can people find you and is there anything you wanna pitch or promote um that maybe you're involved with or anything? Yeah, so I'm not a lot on the different socials, but you can find me on LinkedIn. It's Fatima Jallof, J-A-L-L-O-W. I work for a company called Vass, if you want to find the right person. uh In terms of promoting stuff, my company and I, we co-host or help sponsor the Swedish Fabric and Power BI user groups. So if you are... in Sweden then roughly once a quarter you can come to our office and listen to some peers some interesting stuff about fabric and and or Power BI. Yeah, and we were talking about before the episode started, but if you have a chance to go to the Data Saturday in Stockholm, it's great. Daniel Hutmacher is a great guy. And if you can get him to give you a tour of the city after the event, I highly recommend it from personal experience. uh You can also see one of the largest capsized ships in history, uh which is quite interesting. good promotion for Stockholm right there. you might even find me out and about. I tend to go to FabCon and the sequel bits and things like that. So if you see me out and about, then do come and say hello. oh your time, Fedema. This has been awesome. Thank you. Take care.