Microsoft Fabric: Difference between revisions
Marek Rodak (talk | contribs) |
Marek Rodak (talk | contribs) |
||
| Line 15: | Line 15: | ||
===How to import questionnaire data=== | ===How to import questionnaire data=== | ||
#To import data, we have to create a Lakehouse (collection of files, folders, and tables that represent a database). Go to [https://app.fabric.microsoft.com/ Microsoft Fabric] and select '''Synapse Data Engineering''' experience. | #To import data, we have to create a Lakehouse (collection of files, folders, and tables that represent a database). Go to [https://app.fabric.microsoft.com/ Microsoft Fabric] and select '''Synapse Data Engineering''' experience. | ||
#Create a Lakehouse from the landing page or go to '''Create''' in the left panel and select Lakehouse there. To this lakehouse, we import raw questionnaire data; it's a bronze data layer. | #Create a Lakehouse from the landing page or go to '''Create''' in the left panel and select Lakehouse there. To this lakehouse, we import raw questionnaire data; it's a bronze data layer.<br>[[File:Homepage lakehouse.png|600px]] | ||
#In the lakehouse, click '''New Dataflow Gen2'''.<br>[[File:getdatain lakehouse.png|600px]] | #In the lakehouse, click '''New Dataflow Gen2'''.<br>[[File:getdatain lakehouse.png|600px]] | ||
#When Dataflow loads, click Get Data in the top panel and select Dataverse as a new source.<br>[[File:getdata toppanel dataflow.png|600px]] | #When Dataflow loads, click Get Data in the top panel and select Dataverse as a new source.<br>[[File:getdata toppanel dataflow.png|600px]] | ||
Fill out the required information to connect to your Dataverse. Once connected, select resco_ questionnaire and resco_question (plus tables you need for reporting later). Import the data after initial data cleaning. | #Fill out the required information to connect to your Dataverse. Once connected, select resco_ questionnaire and resco_question (plus tables you need for reporting later). Import the data after initial data cleaning. | ||
Revision as of 08:16, 9 July 2024
| Warning | Work in progress! We are in the process of updating the information on this page. Subject to change. |
Microsoft Fabric is a unified data analytics platform that provides tools for data movement, processing, ingestion, transformation, real-time event routing, and report generation. It incorporates OneLake, a unified data storage solution, eliminating the need to duplicate data for each data manipulation step. Fabric functions as Software as a Service (SaaS). It combines new and existing components from Power BI, Azure Synapse Analytics, Azure Data Factory, and other services into a unified environment. These components are then tailored to customized user experiences.
Questionnaire data in Fabric
Questionnaires are digital forms, usually running in Resco mobile apps, that allow you to collect data in the field. Microsoft Fabric is a tool designed for data management, and there are two main reasons to consider its integration:
- The size of the collected data
- The size of questionnaire data can grow substantially, growing data storage expenses. Microsoft Fabric offers a cheap alternative for storing large amounts of data.
- Need for structured questionnaire data (AI and BI ready)
- To save storage, questionnaire answers are stored in a serializedanswer column in a JSON format. First, we have to transform this format to create structured data. Here, we can utilize the Fabrics Notebook. Notebook is a multi-language interactive programming tool that executes Spark jobs to transform, process, and visualize data.
How to import questionnaire data
- To import data, we have to create a Lakehouse (collection of files, folders, and tables that represent a database). Go to Microsoft Fabric and select Synapse Data Engineering experience.
- Create a Lakehouse from the landing page or go to Create in the left panel and select Lakehouse there. To this lakehouse, we import raw questionnaire data; it's a bronze data layer.
- In the lakehouse, click New Dataflow Gen2.
- When Dataflow loads, click Get Data in the top panel and select Dataverse as a new source.
- Fill out the required information to connect to your Dataverse. Once connected, select resco_ questionnaire and resco_question (plus tables you need for reporting later). Import the data after initial data cleaning.
