Hello and welcome to this video on how to monitor activities in Microsoft Fabric.

To start off with, here we have a monitoring workspace. We're going to add a lakehouse and then a dataflow Gen2 to move data into the data warehouse. Select new item, then search for lakehouse and select that.

This is going to be a lakehouse for athletes, so we can name it Athletes. Then select Create and let it load.

Next, we're going to select Files. Then choose New subfolder. We'll call this subfolder Athletes and then select Create.

Within the folder, we select Upload, then Upload files.

In My Files, we've got the athletes from the games. We're going to select the three files there and select Open, then select upload and wait for them to load.

Now we can go back into the Athletes table and we can see the individual CSV files. We'll select the first one and load the information.

There are no headers on these ones, so we have C1, C2, etc. going down. So, what we could do is now take that and load it into the lakehouse or into the data warehouse. It's our choice of whether to do that or not.

We haven't got a dataflow Gen2 yet, so we're going to exit the reports and the lakehouse in the bottom left-hand corner.

We can see here we've got a data warehouse. We have one for the staging and one here for the monitoring. We're going to add our own dataflow Gen2. So, select New item and search for Gen2, which we can see in the results.

Select that, name it load Athletes and select Create.

So, with that, we can do whatever we need to load that data. We can choose to import from Excel, add default destination, and more.

Select New Source and under the options here we can go through and select the dataflows.

We're going to select more and then type in lakehouse and select the Microsoft Fabric option.

Here we have the connection, so we can select next, select the monitoring file, and then Athletes.

Within that we have various items here going through, so we've got all the files there.

We open up Files and then Athletes. There are the three files, so we can select each one. We could bring the three through, or we could bring just one of them in. It's up to us. Let's bring just one in.

So, select the top one and then the Create button. That'll bring us into the Gen2.

So, we've got the different options coming through here, and we've got lots of different items here going across, so things like which country they're a part of; we'll rename this column to country.

Then the IOC code, so which committee they're a part of, which we'll rename as IOC, and we've also got the year, which we'll rename to year.

Then whether it was summer or winter, so we'll rename to session.

We've got the sport, which we'll rename as sport.

And then the event column.

We finish off with gold, silver, bronze, so we'll rename that to medal.

Here at the start is their name, which we'll also rename.

We can go through and select the columns that we don't need.

So, we've got that first column, the gender column, and the age column, which can be removed.

Also, in the final column, we only want medallists. We don't want the blank ones.

Let's select bronze, silver, and gold in the medal column. So only those with medals will now be left here.

Now we've done that, we're going to tell it where to place this data. We select data destination and then warehouse as the destination.

Once we have a connection to the local warehouse, we can then select the next button.

We'll tell it which one we're going to go to, so select monitoring.

Now we've got various options in here. We'll select monitoring to send it to.

We can see there's some information in there at the moment. We're going to send it to this table, but remove the CSV from the end. Then select the next button.

We're going to select Replace to replace the table, and we can see we've got the different items here.

Select Save Settings and, once it's running, we'll select the arrow next to Publish, then select Publish Now.

Now we've published it into the dataflow, so we'll see that it is refreshing and publishing it. We'll then get a refresh option.

While that's being published, I'm going to go across into Monitor on the left-hand side and see which objects are there.

There we go. There's the loading that's in progress at the moment. We can see how long it takes for each one.

Let's refresh that.

OK, so that's gone through. It's got a duration of 48 seconds. We can go in and have a look at some other items. So, select workspaces, then select Monitoring and then go back into the lakehouse titled Athletes.

We could set up our own notebooks by selecting Open Notebook, then Create a new notebook.

Within the Athletes item, we can see tables and files. Select Athletes. We can see the files here.

We could select tables, but there are no tables at the moment.

If we select Athletes and, look at the dropdown, there's an option to upload if we wanted to. We can also look at the individual files, select the first file, then select Load Data, followed by Spark.

The Spark command appears and that's coming in using the items.

We can now select Run, and that will load the headers. We'll get some headers, which is a bit strange because we've got no headers on this dataset, but it will read the information through.

Andthere we go. It's here. So, the first row isn't really the first row, it's the first row of the data file and the headers. So, we change it to false in the code and select run.

So, that's now going through and looking at that data frame. Select the monitor tab again and then select the refresh button.

We've got the notebook at the top. We can now select the Notebook 1 tab.

That notebook is running really quickly. If we wanted to do that from DF, as we can see in row 3 of the code, we select Run Cell and then go back into the Monitor tab.

We now have an interactive notebook in Job type. If we wanted to, we could go and set up a pipeline, select the Notebook 1 tab, and then the Exit cross that pops up next to the tab. Choose Stop and Close.

Then select the New Item button and search for a pipeline. We'll name this new pipeline, Data Running, and select Create.

Within this, we can add a dataflow from the options at the top.

Then, in Settings, choose Load Athletes from the Dataflow dropdown menu.

Then, we'll select Notebook from the buttons at the top here.

In Settings, choose Notebook 1, which is what we've just set up. We'll connect Dataflow to the notebook and select Save.

Once Save is complete, select the Run option to run the pipeline.

We then go back into the Monitor tab. The pipeline hasn't started at the moment, but it will come in eventually. Let's refresh a couple of times.

So, the status of the load athletes is now in progress. The pipeline will run this one and then run the notebook to get the result. At any point in time, we can go in and add new elements to this by selecting column options.

At the moment, we've got the status, submitted by, location, duration, capacity, job type, and refresh type ticked.

We could also look at something like the run kind if we select that option. Now, the run kind will be added to that list. We'll select the refresh button again. The pipeline is running. The dataflow is also running.

Once the dataflow finishes, the pipeline will continue to run the notebook. Then both the notebook and the data running pipeline will finish.

We want to see succeeded within the status column for the items.

There we go. All items have now succeeded.

Monitor activities in Microsoft Fabric

About