The Developer's Cantina

Consuming an authenticated MCP server with a custom engine agent in Copilot Studio

Mon, 16 Feb 2026 00:00:00 +0000

In the previous posts of this series, we built an MCP server protected with Microsoft Entra authentication using Azure API Management, and then we consumed it with a declarative agent in Microsoft 365 Copilot. However, declarative agents aren’t the only way to leverage MCP servers in the Microsoft 365 ecosystem. Copilot Studio offers a powerful alternative for building custom engine agents that can connect to MCP servers.

In this post, we’re going to explore how to connect a custom engine agent built with Copilot Studio to our Entra-authenticated MCP server. If you’ve been following the series, you’ll find that the experience is actually quite straightforward, especially after working through the gotchas we encountered with declarative agents. As we’ll see, Copilot Studio provides a streamlined wizard that handles most of the OAuth configuration for us.

This post assumes you’re already familiar with the basics of creating agents in Copilot Studio. We won’t dive deep into how to build custom engine agents from scratch; instead, we’ll focus specifically on the MCP integration and authentication flow.

Let’s get started!

Prerequisites

Before we begin, make sure you have the following in place:

An Entra-authenticated MCP server: You can use the Flights MCP server we built earlier in this series, or any other MCP server that supports OAuth authentication through Microsoft Entra. The server should be deployed and accessible via a public URL.
Access to Copilot Studio: You’ll need a valid license to create and test agents, like the Microsoft 365 Copilot or the Copilot Studio User licenses.
An app registration in Microsoft Entra: This is the same app registration we created to consume our MCP server in GitHub Copilot or in a declarative agent, which includes the client ID, client secret, and the access_as_user scope we configured.

Creating the custom engine agent

The first step is to create a new custom engine agent in Copilot Studio. Since we’re building a flight search agent that will connect to the Flights MCP server, let’s set it up with appropriate instructions.

Open Copilot Studio and navigate to your environment.
Click on Agents, then Create blank agent.
Edit the name and the description by using something meaningful, such as “Flight Search Agent”.

Now we need to configure the agent instructions. These instructions define the persona and behavior of the agent. For our flight search assistant, I’m using a professional tone with clear guidelines about how to use the available tools:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


You are a professional flight search assistant. Your role is to help users find flight information using the available flight search tools.

Capabilities:
- Search for flights by origin, destination, and departure date
- Retrieve detailed information about a specific flight by flight number
- Provide a list of available departure airports
- Provide a list of available destination airports

Guidelines:
- When a user asks about flights, use the SearchFlights tool. Origin, destination, and date are all optional — only ask for clarification if the query is too vague to return useful results.
- If the user provides a specific flight number, use GetFlightByNumber to retrieve details.
- If the user asks what airports or routes are available, use GetAvailableOrigins or GetAvailableDestinations as appropriate.
- Present flight results in a clear, organized format including flight number, airline, departure time, duration, price, and availability.
- If no flights match the criteria, inform the user and suggest broadening their search (e.g., different dates or nearby airports).
- Be concise and factual. Do not invent flight information.

These instructions tell the agent exactly how to behave and which tools to use in different scenarios. The key is being specific about the capabilities without being overly rigid.

Adding the MCP server

Now comes the interesting part: connecting the MCP server to our agent. Copilot Studio provides native support for MCP servers through a dedicated wizard in the Tools section.

In your agent, navigate to the Tools section.
Click on Add tool.
In the “Create new” area, select Model Context Protocol.

The wizard will ask you for the following information:

Server name: Give it a descriptive name, such as “Flights MCP Server”.
Server description: Provide a brief description, like “MCP server for searching flight information”.
Server URL: Enter the URL of your MCP server. If you followed the previous posts, this will be something like https://flightsapismcp-apim.azure-api.net/mcp.

Configuring OAuth authentication

Since our MCP server is protected with Microsoft Entra, we need to configure OAuth 2.0 authentication. In the wizard:

Under Authentication, select OAuth 2.0.
For the authentication type, choose Manual. This allows us to provide the specific Entra endpoints and credentials.

Now you’ll need to fill in the OAuth configuration details:

Field	Value
Client ID	The Application (client) ID from your app registration
Client secret	The secret you created for your app registration
Authorization URL	`https://login.microsoftonline.com/{tenant-id}/oauth2/v2.0/authorize`
Token URL	`https://login.microsoftonline.com/{tenant-id}/oauth2/v2.0/token`
Refresh URL	`https://login.microsoftonline.com/{tenant-id}/oauth2/v2.0/token`
Scopes	`api://{client-id}/access_as_user`

Replace {tenant-id} with your Microsoft Entra tenant ID and {client-id} with the Application ID of your MCP server’s app registration in the scopes field. If you’re using a multi-tenant app registration, you can use common instead of the tenant-id.

Note: For Microsoft Entra, the Refresh URL is the same as the Token URL. This is standard for Entra’s OAuth 2.0 implementation.

Click Create to save the configuration.

Adding the redirect URL to your app registration

This is a critical step that you must not skip. After clicking Create, Copilot Studio will display a success message along with a Redirect URL. This URL is unique to your Copilot Studio connection and must be added to your app registration before you can proceed.

Copy this redirect URL and add it to your app registration in Microsoft Entra:

Go to the Azure portal and navigate to your app registration.
Click on Authentication in the left menu.
Under Platform configurations, click on Add a platform if you haven’t already, and choose Web.
Add the redirect URL you copied from Copilot Studio.
Click Save.

Important: You must complete this step before clicking Next in the Copilot Studio wizard. If you skip this, the authentication flow will fail because Microsoft Entra won’t recognize Copilot Studio as a valid redirect destination.

Creating the connection

Once you’ve added the redirect URL to your app registration, go back to Copilot Studio and click Next. You’ll be prompted to create a connection by signing in with a valid Microsoft Entra account.

Click the sign-in button and complete the authentication flow using your organizational account. This step establishes the connection between Copilot Studio and your protected MCP server.

After successful authentication, click Add and configure to finalize the setup. Copilot Studio will redirect you to the tool configuration page.

Verifying the tools

If everything was configured correctly, you should now see all the tools exposed by your Flights MCP server listed in the Tools section of your agent. For our Flights MCP server, you should see the following four tools:

get_airport_origins: List all available departure airports
get_airport_destinations: List all available destination airports
get_flight_details: Get detailed information about a specific flight
search_flights: Search for flights by origin, destination, and departure date.

These tools are automatically imported based on the tool definitions in your MCP server. The descriptions you provided using the [Description] attributes in your .NET code are used here to help the agent understand when and how to use each tool.

Testing the agent

Now we’re ready to test our flight search agent! Use the test panel built into Copilot Studio to interact with the agent.

Try asking a question like:

Find me flights from New York to London

The agent should recognize that it needs to use the search_flights tool from the MCP server. The first time you use it, you’ll see the agent requesting to create a connection. You just need to click on Open connection manager and then click Connect near the name of the MCP server. After doing that, you can click on Retry and then the agent will call the MCP server and return the results.

We did it! The custom engine agent is now successfully consuming our Entra-authenticated MCP server.

What happens behind the scenes?

One interesting aspect of how Copilot Studio handles MCP servers is worth calling out. When you add an MCP server to your agent, Copilot Studio creates a custom Power Platform connector behind the scenes. This connector acts as the bridge between your agent and the MCP server.

This design choice has important implications for enterprise scenarios:

Governance: Administrators can manage and monitor MCP connections using the same tools they use for standard Power Platform connectors.
Data Loss Prevention (DLP): Connectors can be classified and controlled through DLP policies.
On-premises support: If your MCP server is hosted on-premises, you can potentially use the on-premises data gateway to establish connectivity, just like you would with any other Power Platform connector.

You can verify that by opening the Power Platform portal, clicking More in the left sidebar and looking for the section named Custom connectors. You will see the custom connector which is mapped to your MCP server in the list.

Additional resources

If you want a more hands-on experience with MCP and OAuth in Copilot Studio, I recommend checking out the Copilot Developer Camp. The lab provides a step-by-step walkthrough of connecting Copilot Studio to an OAuth-protected MCP server.

You can also find the complete source code of the Flights MCP server on GitHub.

Wrapping up

In this post, we’ve seen how to connect a custom engine agent built with Copilot Studio to an MCP server protected with Microsoft Entra authentication. Compared to the declarative agent approach we explored in the previous post, the Copilot Studio experience is notably smoother:

Streamlined wizard: Copilot Studio provides a guided experience for configuring OAuth, eliminating the need to manually edit YAML files or fix parameter types.
Automatic tool discovery: Once the connection is established, the tools are imported automatically without requiring additional fetch operations.
Power Platform integration: The connector-based architecture brings MCP servers into the familiar Power Platform governance model.

The main thing to remember is the redirect URL step: make sure to copy the URL from Copilot Studio and add it to your app registration before completing the wizard. Other than that, the process is quite straightforward.

Happy coding!

Consuming an authenticated MCP server with a declarative agent in Copilot Chat

Fri, 30 Jan 2026 00:00:00 +0000

In the previous post, we built an MCP server protected with Microsoft Entra authentication using Azure API Management. We tested it using Visual Studio Code with GitHub Copilot, proving that the authentication flow works correctly. However, one of the most exciting scenarios for MCP servers is integrating them with Microsoft 365 Copilot through declarative agents.

In this post, we’re going to explore how to create a declarative agent using the Microsoft 365 Agents Toolkit that consumes our Entra-authenticated MCP server. This is a relatively new feature, and there are some gotchas along the way that I’ll help you navigate. By the end of this tutorial, you’ll have a working declarative agent that can search for flights using our protected MCP server.

This is the first post in a series exploring authenticated MCP in the Microsoft 365 Copilot ecosystem. In the upcoming posts, we’ll cover how to consume authenticated MCP servers from Microsoft Foundry and Copilot Studio as well.

Let’s get started!

Prerequisites

Before we begin, make sure you have the following in place:

An Entra-authenticated MCP server: You can use the one we built in the previous post, or any other MCP server that supports OAuth authentication through Microsoft Entra. The server should be deployed and accessible via a public URL.
Microsoft 365 Agents Toolkit: You need version 6.4.3 or higher installed as an extension in Visual Studio Code. You can install it from the VS Code Marketplace.
Microsoft 365 Copilot license or free Copilot Chat: You’ll need access to Copilot Chat to test the declarative agent.

Creating the declarative agent project

The Microsoft 365 Agents Toolkit provides a streamlined way to create declarative agents that can connect to MCP servers. Let’s create our project:

Open Visual Studio Code and select the Microsoft 365 Agents Toolkit icon from the sidebar.
Click on Create a new Agent / App.
Choose Declarative Agent.
Select Add an action.
Choose Start with an MCP server (preview).
Provide the URL of your MCP server. In my case, it’s https://flightsapismcp-apim.azure-api.net/mcp.
Choose a folder where to store the project.
Give it a meaningful name (for example, “Flights Agent”).

The toolkit will create a declarative agent project with the mcp.json file already configured to connect to the MCP server you provided.

Connecting to the MCP server

Now that we have our project set up, we need to establish a connection to the MCP server. Open the mcp.json file and you’ll see a CodeLens toolbar above the server entry. Click on Start to initiate the connection.

Since our MCP server is protected with Entra authentication, Visual Studio Code will prompt you to provide:

The client ID of your app registration
The client secret

This is the same flow we covered in the previous post when testing the MCP server. After providing the credentials, you’ll be redirected to the Microsoft Entra login page to authenticate with your account. Complete the authentication flow, and the connection should be established.

Fetching the MCP tools

Here’s where the first gotcha comes in. After connecting, you might need to restart Visual Studio Code because the Microsoft 365 Agents Toolkit options in the CodeLens sometimes don’t show up immediately after the initial connection.

Once you’ve restarted, open the mcp.json file again and look for the ATK Fetch action from MCP option in the CodeLens toolbar. Click on it to import the available tools from your MCP server.

Issue #1: The GET endpoint requirement

If you followed my previous blog post exactly, the toolkit will fail to recognize that the MCP server requires authentication. The option to fetch the tools will work, but you will end up with a configuration that treats the MCP server as unauthenticated.

This happens because the Microsoft 365 Agents Toolkit performs a GET request to the MCP server to discover authentication requirements. It expects the response to fail with a 401 Unauthorized status and include the WWW-Authenticate header containing the authorization endpoint URL. We support the WWW-Authenticate header in our APIM configuration, but the MCP server we built only configured this authentication challenge for POST requests, not GET. As such, the toolkit will get in response a 404 error, leading to treat the MCP server as unauthenticated.

To fix this, we need to add a GET endpoint in Azure API Management:

Go to the Azure portal and navigate to your API Management instance.
Go to APIs → Flight APIs.
Click on the three dots (…) next to the MCP POST operation and choose Clone.
Click on the pencil icon on the Frontend section.
Change the HTTP method from POST to GET.
Set the endpoint to /mcp.
Save the changes.

Now the toolkit will be able to properly discover the authentication requirements of your MCP server.

Importing the tools

After fixing the GET endpoint issue, click on ATK Fetch action from MCP. This time, the process should work correctly:

Visual Studio Code will ask which action manifest you want to update. Select the ai-plugin.json file in your project.
You’ll see a list of all available tools from your MCP server. For the Flights API, you should see 4 tools: search_flights, get_flight_details, get_airports_origins, and get_airports_destinations. Select all of them.
Click OK to import the tools.

Since the toolkit detected that the MCP server is protected, it will ask you to choose the authentication method. Select OAuth (with static registration).

The ai-plugin.json file will be updated with the metadata of all the MCP tools. You’ll notice that the auth property looks like this:

1
2
3
4


"auth": {
 "type": "OAuthPluginVault",
 "reference_id": "${{MCP_DA_AUTH_ID_FLIGHTSAPI}}"
}

The reference_id points to an OAuth registration in the Teams Developer Portal. The good news is that you don’t need to create this manually. The toolkit will take care of it when you provision the agent.

Provisioning the agent

Now we’re ready to provision our declarative agent. In the Microsoft 365 Agents Toolkit sidebar, navigate to the Lifecycle panel and click on Provision.

The provisioning process will start, and VS Code will prompt you for additional information to create the OAuth registration:

Client ID: The client ID of your app registration
Client secret: The client secret
Scopes: The OAuth scopes required by your API, which should be api://your-client-id/access_as_user

Issue #2: Missing OAuth properties in m365agents.yml

At this point, you might encounter an error like this:

1
2
3


[Error] - code:oauthRegister.InvalidActionInputError, message: The 'oauth/register' action cannot be completed as the following parameter(s):
authorizationUrl,tokenUrl,apiSpecPath, are either missing or have an invalid value in the provided yaml file:
d:\src\flights-agent\m365agents.yml. Ensure that the required parameters are provided and have valid values and try again.

This happens because the m365agents.yml file in your project root is missing some required OAuth properties. To fix this, open the file and find the step that contains uses: oauth/register. Add the following properties:

1
2
3


authorizationUrl: https://login.microsoftonline.com/common/oauth2/v2.0/authorize
tokenUrl: https://login.microsoftonline.com/common/oauth2/v2.0/token
apiSpecPath: https://flightsapismcp-apim.azure-api.net/mcp

The first two URLs are the standard Microsoft Entra OAuth endpoints (they’re the same for all Entra-authenticated APIs). The apiSpecPath should be the endpoint of your MCP server.

Issue #3: Tool parameter type incompatibilities

After fixing the OAuth properties, you might encounter additional errors related to the tool definitions. This happens because the toolkit imports MCP tool parameters with some types and formats that aren’t supported by the declarative agent runtime.

Here are the the fixes I had to apply:

in the search_flights tool, the property maxResults is set as an integer, but it should be actually treated as a string. You must change it from this:

1
2
3
4
5


"maxResults": {
 "description": "Maximum number of flights to return",
 "type": "integer",
 "default": 10
}

to this:

1
2
3
4
5


"maxResults": {
 "description": "Maximum number of flights to return",
 "type": "string",
 "default": "10"
}

The departureDate parameter has an invalid property (format) and the type should be string, not an array. You must change it from:

1
2
3
4
5
6
7
8
9


"departureDate": {
 "description": "Departure date in YYYY-MM-DD format (e.g., '2025-01-20')",
 "type": [
 "string",
 "null"
 ],
 "format": "date",
 "default": null
}

to:

1
2
3
4
5


"departureDate": {
 "description": "Departure date in YYYY-MM-DD format (e.g., '2025-01-20')",
 "type": "string",
 "default": null
}

Here’s how the corrected search_flights tool should look in its entirety:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30


{
 "name": "search_flights",
 "description": "Search for flights by origin, destination, and departure date. Date is optional, if the user doesn't provide it, don't ask for it. Returns a list of available flights matching the criteria.",
 "parameters": {
 "type": "object",
 "properties": {
 "origin": {
 "description": "Departure airport or city (e.g., 'New York', 'JFK', 'Los Angeles')",
 "type": "string",
 "default": null
 },
 "destination": {
 "description": "Arrival airport or city (e.g., 'London', 'LHR', 'Tokyo')",
 "type": "string",
 "default": null
 },
 "departureDate": {
 "description": "Departure date in YYYY-MM-DD format (e.g., '2025-01-20')",
 "type": "string",
 "default": null
 },
 "maxResults": {
 "description": "Maximum number of flights to return",
 "type": "string",
 "default": "10"
 }
 },
 "required": []
 }
}

After making these corrections, run the Provision command again. This time, it should complete successfully, providing you are logged into the Microsoft 365 Agents Toolkit with your Microsoft 365 account.

Testing the agent

Now that our declarative agent is provisioned, we can test it in Copilot Chat. Open Copilot Chat and select your agent from the list of available agents.

Try asking a question like:

Find me flights from New York to London

The agent will go through the following flow:

Permission request: The agent will ask for your permission to connect to the MCP server. Approve this request.
Authentication: You’ll see a Sign in button. Click it to authenticate with your Microsoft Entra credentials.
Results: After authentication, the agent will call the MCP server and return a list of available flights.

We did it! The declarative agent is now successfully consuming our Entra-authenticated MCP server.

What happened behind the scenes?

When we imported the MCP server using the Microsoft 365 Agents Toolkit, I called out how the following entry was added to manage the authentication:

1
2
3
4


"auth": {
 "type": "OAuthPluginVault",
 "reference_id": "${{MCP_DA_AUTH_ID_FLIGHTSAPI}}"
}

The value of the ${{MCP_DA_AUTH_ID_FLIGHTSAPI}} placeholder will be saved inside the .env.dev file.

This entry references an entry in the Teams Developer Portal, which is created behind the scenes for you. However, if you want to better understand where it comes from, you can check it by logging in with your Entra Id on the portal at https://dev.teams.microsoft.com/. Move to Tools -> OAuth Client Registration. You will find the entry created by the Microsoft 365 Agents Toolkit and, if you click on it, you will see all the information required for the authentication, like the client id or the authorization URL. Notice that, at the top, there’s a box titled OAuth client registration ID, with a value inside it. This is the id that gets stored in the .env.dev file and assigned to the the ${{MCP_DA_AUTH_ID_FLIGHTSAPI}} placeholder.

Additional resources

If you want a more hands-on experience with MCP in declarative agents, I recommend checking out the Copilot Developer Camp. The Microsoft 365 Dev Advocacy team created an amazing lab which will guide you though the step-by-step experience of building your own MCP server with the authentication flow managed in code rather than through Azure API Management.

Wrapping up

In this post, we’ve seen how to create a declarative agent using the Microsoft 365 Agents Toolkit that consumes an Entra-authenticated MCP server. Along the way, we navigated several gotchas:

The GET endpoint requirement: The toolkit needs a GET endpoint (not just POST) to discover authentication requirements.
Missing OAuth properties: The m365agents.yml file needs authorizationUrl, tokenUrl, and apiSpecPath properties added manually.
Tool parameter type incompatibilities: Integer and array types must be converted to strings, and unsupported format properties must be removed.

The one that took me the longest to figure out was the GET endpoint requirement. It’s not immediately obvious from the error messages, and the MCP specification primarily focuses on POST requests. But GitHub Copilot came to the rescue, by analyzing the repository of the Microsoft 365 Agents Toolkit to analyze how the operation was performed and matching it with my configuration.

As I mentioned at the beginning, this is the first post in a series exploring authenticated MCP in the Copilot ecosystem. In the next posts, we’ll see how to consume the same authenticated MCP server from Azure AI Foundry and Copilot Studio. Stay tuned!

Happy coding!

Enabling multi-tenant authentication for MCP servers with Microsoft Entra and APIM

Wed, 28 Jan 2026 00:00:00 +0000

In my previous post, we explored how to build an authenticated MCP server using Microsoft Entra for identity management and Azure API Management to protect our APIs. That implementation worked great, but it had one important limitation: it only supported single-tenant authentication. This means that only users from the same organization where the app registration was created could access the MCP server.

While single-tenant authentication is perfectly fine for internal applications, there are many scenarios where you need to enable access to users from different organizations. For example, imagine you’re building an MCP server as an ISV that you want to offer to multiple customers. In these cases, requiring each user to create their own app registration becomes impractical and creates unnecessary friction.

In this post, we’re going to explore how to enable multi-tenant authentication for our MCP server. The good news is that the changes we need to make are surprisingly straightforward—they’re all in the Azure configuration, with no code changes required. However, knowing exactly which settings to change isn’t always obvious, so let’s walk through the process step by step.

Understanding the challenge

Before we dive into the configuration changes, it’s important to understand what we’re trying to solve. In the single-tenant setup from the previous post, both our app registrations (the API and the client) were configured to accept “Accounts in this organizational directory only.” This means that when Microsoft Entra validates an access token, it checks that the user belongs to the same tenant where the app was registered.

For multi-tenant scenarios, we need to make a few key changes:

Configure both app registrations to accept accounts from any organizational directory
Update the APIM policies to use the common endpoint instead of a specific tenant ID
Add explicit client application IDs to the token validation policy
Update the OAuth metadata endpoint to point to the multi-tenant endpoints

The beauty of this approach is that all these changes happen at the infrastructure level. Our .NET MCP server code doesn’t need to change at all, since authentication is handled entirely by APIM.

Let’s get started!

Updating the app registrations

The first step is to update both app registrations to support multi-tenant authentication. We need to modify both the API app registration (Flights MCP Server) and the client app registration (Flights MCP Client).

Updating the Flights MCP Server API app registration

Let’s start with the API app registration. Navigate to the Azure portal and go to the App registrations section. Find the Flights MCP Server app registration we created in the previous post.

Click on the app registration to open its details
In the left menu, click on Authentication and move to the Settings tab
Under Supported account types, change the selection from Accounts in this organizational directory only to Accounts in any organizational directory (Any Microsoft Entra directory - Multitenant)
Click Save to apply the changes

That’s it for the API app registration! Notice that we didn’t need to change anything else. The Application ID URI (api://your-client-id), the access_as_user scope, and all other settings remain exactly the same.

Updating the Flights MCP Client app registration

Now we need to make the same change to the client app registration. Still in the App registrations section, find the Flights MCP Client app registration.

Click on the app registration to open its details
In the left menu, click on Authentication and move to the Settings tab
Under Supported account types, change the selection to Accounts in any organizational directory (Any Microsoft Entra directory - Multitenant)
Click Save to apply the changes

Again, the redirect URIs (https://vscode.dev/redirect and http://127.0.0.1:33418) and the API permissions (access_as_user from the Flights MCP Server API) remain unchanged.

Now that both app registrations are configured for multi-tenant support, we can move to the APIM configuration changes.

Updating the APIM token validation policy

The next step is to update the APIM policy that validates access tokens. This is where we need to make two important changes: switching to the common tenant ID and adding explicit client application IDs.

Navigate to your APIM instance in the Azure portal, then:

Click on APIs in the left menu
Select the API that was created for your MCP server
Click on the Design tab
Select All operations
In the Inbound processing section, click on the policy editor icon (the </> symbol)

You should see the existing policy that we created in the previous post. Now we need to modify the <validate-azure-ad-token> element. Here’s what the updated policy looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


<inbound>
 <base />
 <set-backend-service id="apim-generated-policy" backend-id="WebApp_flightsapismcp" />
 <validate-azure-ad-token tenant-id="common" header-name="Authorization">
 <client-application-ids>
 <!-- Multi-tenant client app registration (shared with attendees) -->
 <application-id>your-client-app-id</application-id>
 </client-application-ids>
 <audiences>
 <!-- Your existing API app registration -->
 <audience>api://your-api-app-id</audience>
 <audience>your-api-app-id</audience>
 </audiences>
 </validate-azure-ad-token>
</inbound>

Let’s break down the two key changes:

Using the `common` tenant ID

The first change is replacing the specific tenant ID with common:

1

<validate-azure-ad-token tenant-id="common" ...>

The common endpoint is a special Microsoft Entra endpoint that accepts users from any organizational directory. When a user authenticates, Microsoft Entra will validate their credentials against their home tenant, but the token validation will work for any tenant.

Adding client application IDs

The second change is the addition of the <client-application-ids> section:

1
2
3


<client-application-ids>
 <application-id>your-client-app-id</application-id>
</client-application-ids>

This is a critical piece that isn’t immediately obvious. When using multi-tenant authentication, you need to explicitly specify which client applications are allowed to call your API. Replace your-client-app-id with the Application (client) ID of your Flights MCP Client app registration.

This is what enables the ISV scenario I mentioned earlier: you create one multi-tenant client app registration, and then you can share its client ID with all workshop attendees. They don’t need to create their own app registrations, they simply use the one you provide, and they authenticate with their own credentials from their own tenant.

Remember to replace your-api-app-id with the actual Application (client) ID of your Flights MCP Server app registration.

Updating the OAuth metadata endpoint

You might remember that, in the previous post, we have added to our MCP server a special endpoint, /.well-known/oauth-authorization-server. This endpoint is used by clients (like Visual Studio Code) to discover the authorization and token endpoints they need to use for authentication. Now we need to update the OAuth metadata endpoint that we configured in the previous post to support a multi-tenant scenario.

Still in the APIM Design tab:

Find the operation called OAuth 2.0 Authorization Server Metadata (the one at /.well-known/oauth-authorization-server)
Click on it to select it
In the Inbound processing section, click on the policy editor icon

You’ll see the policy that returns the OAuth metadata. We need to update all the URLs to use common instead of the specific tenant ID. Here’s the updated policy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


<inbound>
 <return-response>
 <set-status code="200" reason="OK" />
 <set-header name="Content-Type" exists-action="override">
 <value>application/json</value>
 </set-header>
 <set-body>{
 "issuer": "https://login.microsoftonline.com/common/v2.0",
 "authorization_endpoint": "https://login.microsoftonline.com/common/oauth2/v2.0/authorize",
 "token_endpoint": "https://login.microsoftonline.com/common/oauth2/v2.0/token",
 "scopes_supported": ["api://your-api-app-id/.default"],
 "response_types_supported": ["code"],
 "code_challenge_methods_supported": ["S256"]
 }</set-body>
 </return-response>
</inbound>

Notice how all three endpoints (issuer, authorization_endpoint, and token_endpoint) now use common instead of a specific tenant ID. The scopes_supported array remains unchanged, it still references your API app registration.

Remember to replace your-api-app-id with the actual Application (client) ID of your Flights MCP Server app registration.

That’s it! Your APIM configuration is now ready for multi-tenant authentication.

Testing the multi-tenant setup

Now comes the moment of truth: testing that our multi-tenant configuration actually works. The best way to do this is to test with a user from a different tenant than the one where you created the app registrations.

Testing with the original tenant

Before testing with a different tenant, I recommend first verifying that your original setup still works. This is an important regression test! You want to make sure that users from your own tenant can still authenticate successfully.

Open Visual Studio Code and configure the mcp.json file to point to your APIM endpoint, just like you did in the previous post. When you start the MCP server connection, you should be prompted to authenticate. Log in with your original credentials (from the tenant where the app registrations were created), and verify that everything still works as expected. You should be able to still see, from the toolbar, that the MCP server is running and that there 4 tools available. You could also ask GitHub Copilot to find flights just like before.

Testing with a different tenant

Now for the real test: authenticating with a user from a different tenant. If you don’t have one and you are eligible, you can create a free Microsoft 365 developer tenant for testing purposes.

Here’s an important tip: Visual Studio Code caches authentication credentials, and I didn’t find an easy way to clear them without completely resetting the storage. For testing, I recommend using Visual Studio Code Insiders as a separate installation or a separate PC with another Visual Studio Code installation. This way, you can test the multi-tenant flow without interfering with your existing VS Code setup. If, by any chance, you want to clear the cached credentials in your existing VS Code installation, you can delete the files states.vscbd and states.vscdb.backup located in:

For VS Code: the %APPDATA%\Code\User\globalStorage\ folder on Windows
For VS Code Insiders: the %APPDATA%\Code - Insiders\User\globalStorage\ folder on Windows

Deleting these files will completely reset the VS Code configuration, including extensions, customizations, and settings. This is why I suggest using a separate installation for testing.

When you try to connect to the MCP server with a user from the new tenant:

Visual Studio Code will prompt you for the client ID. Provide the Application (client) ID of your Flights MCP Client app registration
When asked for a secret, just leave it empty and press Enter (since we’re using a public client)
You’ll be redirected to the Microsoft Entra login page. Login with an account from your different tenant.
After logging in, you’ll see a consent screen asking for permission to access the Flights MCP Server API
Click Accept to grant consent

This consent step happens only the first time a user from a new tenant authenticates. Because we configured the access_as_user scope to allow user consent (not just admin consent), regular users can grant this permission themselves, with no admin involvement needed.

After consenting, you should see the MCP server connection established successfully. Try asking GitHub Copilot to find flights from New York to London, just like in the previous post. If everything works, you’ve successfully enabled multi-tenant authentication!

Troubleshooting tips

As I was implementing this multi-tenant setup, I ran into a few issues that might help you if you encounter similar problems.

Token validation errors

The most common error I encountered was a 404 (content not found) when trying to call the API. This wasn’t very helpful, as it didn’t clearly indicate that the problem was with token validation.

What really helped me troubleshoot was using the Azure CLI to generate a valid bearer token and then testing it directly in APIM. Here’s the command I used:

1

az account get-access-token --resource "api://your-api-app-id" --query accessToken -o tsv

Replace your-api-app-id with the actual Application (client) ID of your Flights MCP Server app registration. This command will return a bearer token that you can use for testing.

Once you have the token, go to your APIM instance in the Azure portal:

Navigate to APIs and select your MCP server API
Click on the Test tab
Select the /mcp endpoint
In the HTTP Request section, add an Authorization header with the value Bearer YOUR_TOKEN_HERE (replace YOUR_TOKEN_HERE with the token from the CLI command)
Execute the test by pressing the Trace button

The operation will fail anyway because we aren’t supplying valid MCP parameters, but the Trace output will show you detailed information about the token validation process, so you can diagnose what’s going wrong. For example, during one of my tests, I discovered that the audience claim in the token wasn’t matching what the policy expected. Without the trace output, I would have been completely lost.

Security considerations

Before you implement multi-tenant authentication, it’s important to understand the security implications. With the configuration we just implemented, any user with a valid Microsoft Entra ID from any organization can authenticate and access your MCP server.

For my scenario (delivering a workshop where attendees are required to connect my protected MCP server from their own tenant), this is exactly what I wanted. I don’t know in advance which tenants my attendees will come from, so I need to accept any organizational account. However, this might not be appropriate for all scenarios.

I recommend reviewing the Azure Architecture Center’s guide on multi-tenancy with API Management for more detailed security considerations and best practices.

Wrapping up

In this post, we’ve learned how to enable multi-tenant authentication for MCP servers using Microsoft Entra and Azure API Management. The changes we made were surprisingly simple—just a few configuration updates in the app registrations and APIM policies—but they unlock powerful scenarios for sharing your MCP servers with users from different organizations.

To summarize, here are the four key changes needed to move from single-tenant to multi-tenant:

Update both app registrations (API and client) to support “Accounts in any organizational directory”
Modify the APIM token validation policy to use tenant-id="common" and add the <client-application-ids> section
Update the OAuth metadata endpoint to use common in all URLs
No code changes required in your .NET MCP server

This multi-tenant setup is perfect for scenarios like workshops, where you want to minimize friction for attendees, or for ISVs who want to offer their MCP servers as a service to multiple customers. Users from different organizations can all use the same shared client ID, and they authenticate with their own credentials from their own tenants.

The complete source code for the MCP server is still available in the same GitHub repository we used in the previous post—remember, the code itself didn’t change at all!

Happy coding!

Building an authenticated MCP server with Microsoft Entra and .NET

Mon, 26 Jan 2026 00:00:00 +0000

The post has been updated on 29th January 2026 to include additional APIM endpoints which are required to be fully compliant with the MCP Streamable HTTP protocol specifications.

We have talked multiple times about Model-Context-Protocol (MCP) in this blog. It’s been a while since the last time I wrote about it, but in the meantime the interest and growth around MCP has been steadily increasing. More and more companies are adopting MCP as a standard for exposing their services to AI applications.

As MCP grew, one of the aspects that has become increasingly important is authentication and authorization. Over the past months, we built a few sample MCP servers but, for simplicity, we didn’t include any authentication mechanism. However, in a production scenario, it’s crucial to ensure that only authorized clients can access the MCP services. When our MCP server is exposing sensitive data or tools, we need to ensure only authorized users and agents can access them.

In this post, we’re going to explore how to build an authenticated MCP server using Microsoft Entra for identity management. Microsoft Entra provides a robust and scalable solution for managing identities and access, making it an excellent choice for securing our MCP server. Then, later in the series, we’ll look at how we can leverage this setup in Microsoft 365 Copilot, both with declarative agents (which recently added support for MCP servers) and Copilot Studio.

We’re going to use .NET to implement our MCP server, Azure App Service to host it and Azure API Management to easily enable authentication using Microsoft Entra. While we could add authentication directly to our .NET application, using Azure API Management provides several advantages: centralized authentication across multiple APIs, built-in rate limiting, and the ability to manage authentication policies without redeploying code

For the sake of this demo, we’re going to build a Flights MCP server, which exposes flight information to clients. Agents can use this MCP server to get information about available flights, routes, airports, etc. For the sake of simplicity, however, we won’t return real-time flight information, but just mock data that will be embedded in the server itself.

Let’s get started!

Setting up the MCP server project

I won’t spend too much time on the basic setup of the .NET project, since we already covered it when we talked about building a MCP server for Copilot Studio. Even if that post is a bit old, the steps are still valid and the SDK still works in the same way when it comes to build a MCP server which uses HTTP as a transport protocol. Since that post, the default MCP protocol for remote servers changed from SSE to Streamable HTTP, but the .NET SDK abstracts this detail: now, when you set up the MCP server using the HTTP transport, it will use Streamable HTTP by default.Streamable HTTP allows the server to send responses incrementally using chunked transfer encoding, which is ideal for long-running agent operations that need to stream results back to the client

The first step is to create a new ASP.NET Core Web API project. You can do this using the .NET CLI:qdvbv

1

dotnet new webapi -n FlightsMcpServer

Next, we need to add the necessary NuGet packages for MCP. You can do this by running the following commands:

1
2


dotnet add package ModelContextProtocol
dotnet add package ModelContextProtocol.Http

Now we can start building the code needed to search and retrieve flight information

FlightService.cs

This class implements the logic that is used by the MCP service to return flight information. The service offers 4 methods:

1
2
3
4


IEnumerable<Flight> SearchFlights(FlightSearchRequest request);
Flight? GetFlightByNumber(string flightNumber);
IEnumerable<string> GetAvailableOrigins();
IEnumerable<string> GetAvailableDestinations();

For the sake of simplicity, the flight data is loaded from a JSON file which is included in the project. Here is how a sample flight looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


 {
 "flightNumber": "BA112",
 "airline": "British Airways",
 "origin": "New York (JFK)",
 "destination": "London (LHR)",
 "durationHours": 7.5,
 "aircraft": "Boeing 777",
 "price": 850,
 "availableSeats": 25,
 "status": "OnTime",
 "dayOffset": 0,
 "hourOffset": 6
 }

This JSON data is mapped with an equivalent C# class called FlightData:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


private class FlightData
{
 public string FlightNumber { get; set; } = string.Empty;
 public string Airline { get; set; } = string.Empty;
 public string Origin { get; set; } = string.Empty;
 public string Destination { get; set; } = string.Empty;
 public double DurationHours { get; set; }
 public string Aircraft { get; set; } = string.Empty;
 public decimal Price { get; set; }
 public int AvailableSeats { get; set; }
 public string Status { get; set; } = string.Empty;
 public int DayOffset { get; set; }
 public double HourOffset { get; set; }
}

The four methods of the service are implemented simply by filtering and searching the in-memory list of flights. For example, this is how the SearchFlights method is implemented:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


public IEnumerable<Flight> SearchFlights(FlightSearchRequest request)
{
 _logger.LogInformation(
 "SearchFlights called - Origin: {Origin}, Destination: {Destination}, DepartureDate: {DepartureDate}, MaxResults: {MaxResults}",
 request.Origin ?? "(not specified)",
 request.Destination ?? "(not specified)",
 request.DepartureDate?.ToString() ?? "(not specified)",
 request.MaxResults);

 var query = _flights.AsEnumerable();

 if (!string.IsNullOrEmpty(request.Origin))
 {
 query = query.Where(f => f.Origin.Contains(request.Origin, StringComparison.OrdinalIgnoreCase));
 }

 if (!string.IsNullOrEmpty(request.Destination))
 {
 query = query.Where(f => f.Destination.Contains(request.Destination, StringComparison.OrdinalIgnoreCase));
 }

 if (request.DepartureDate.HasValue)
 {
 query = query.Where(f => DateOnly.FromDateTime(f.DepartureTime) == request.DepartureDate.Value);
 }

 var results = query.Take(request.MaxResults).ToList();


 return results;
}

This method is taking a FlightSearchRequest object as input, which contains the search criteria (origin, destination, date, etc.), and returns a list of Flight objects that match the criteria. The other methods follows the same pattern, so I won’t include them here. You can find the full code of the FlightService class in the GitHub repository.

FlightsTool.cs

Now that we have the FlightService class implemented, we can take advantage of the .NET MCP SDK to expose the methods we built as MCP tools. Since we have implemented all the flight search logic in the FlightService class, our MCP tools will simply need to call the corresponding methods. However, we need to wrap them with some special attributes provided by the .NET SDK to explain, using natural language, the role of the tool and its parameters. This is a critical step, since these information will be used by the LLMs to understand how to use the tools.

The first attribute is the [McpServerToolType] one, which we need to apply to the entire class, as shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


[McpServerToolType]
public class FlightTools
{
 private readonly FlightService _flightService;

 public FlightTools(FlightService flightService)
 {
 _flightService = flightService;
 }

 // Tool methods go here
}

To use the FlightService inside the MCP tools, we inject it through the constructor. The .NET dependency injection system will take care of providing an instance of the FlightService when the MCP server is started.

Now we need to start adding the tool methods. Each method needs to be decorated with the [McpServerTool] attribute, then we’re going to use the [Description] attribute to provide a natural language description of the tool and the input parameters. For example, this is how we expose the SearchFlights() method that we implemented as an MCP tool:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


[McpServerTool, Description("Search for flights by origin, destination, and departure date. Date is optional, if the user doesn't provide it, don't ask for it. Returns a list of available flights matching the criteria.")]
public IEnumerable<Flight> SearchFlights(
 [Description("Departure airport or city (e.g., 'New York', 'JFK', 'Los Angeles')")] string? origin = null,
 [Description("Arrival airport or city (e.g., 'London', 'LHR', 'Tokyo')")] string? destination = null,
 [Description("Departure date in YYYY-MM-DD format (e.g., '2025-01-20')")] DateOnly? departureDate = null,
 [Description("Maximum number of flights to return")] int maxResults = 10)
{

 var request = new FlightSearchRequest(origin, destination, departureDate, maxResults);
 var results = _flightService.SearchFlights(request);

 var resultList = results.ToList();

 return resultList;
}

As you can see, we are simply creating a new FlightSearchRequest object using the input parameters, then we call the SearchFlights() method of the FlightService class and return the results. However, the attributes we have added describe in details the role of the tool and its parameters: it’s used to search flights given a series of parameters; some of them are optional; the origin parameter can be an airport code or a city name; the departure date should be in a specific format, etc.

Again, like for the FlightService class, the other tool methods follow the same pattern, so I won’t include them here. You can find the full code of the FlightsTool class in the GitHub repository.

Program.cs

Finally, to turn our ASP.NET Web API project into an MCP server, we need to modify the Program.cs file to set up the MCP server and register our tools. The first step is to register in the dependency injection system the FlightService class, so that it can be injected into the FlightsTool class:

1

builder.Services.AddSingleton<FlightService>();

Next, we need to add the MCP server to the ASP.NET pipeline and register our FlightsTool class as an MCP tool:

1
2
3


builder.Services.AddMcpServer()
 .WithHttpTransport()
 .WithTools<FlightTools>();

Please notice an important part of the registration: we must specify the transport protocol, in this HTTP, using the WithHttpTransport() method. This is crucial, especially when we want to enable authentication, since the authentication middleware will be applied to HTTP requests. It’s important for us also because, at some point, our goal will be to use this MCP server with platforms like Microsoft 365 Copilot and Copilot Studio, which supports only HTTP transport for remote MCP servers.

Finally, we need to expose the MCP server endpoint in the ASP.NET pipeline:

1

app.MapMcp("/mcp");

This means that our MCP server will be available at the /mcp endpoint of our ASP.NET Web API.

That’s it! Now we have a basic implementation of the MCP server. If you want, you can easily test it with Visual Studio Code with GitHub Copilot:

Press F5 in Visual Studio or Visual Studio Code to run the server locally.
In Visual Studio Code, open the GitHub Copilot panel and, in the chat interface, set the conversation type to be “Agent”.
Click on the Tools icon in the same box.
When the command palette opens up, you will find three icons in the Configure Tools toolbar. Click on the first one, with the MCP logo, which tooltip is Add MCP Server.
Choose HTTP as the transport protocol and enter http://localhost:5213/mcp as the MCP server URL. This step will create a mcp.json file with the following entry:
1 2 3 4

"flights-mcp": { "url": "http://localhost:5213/mcp", "type": "http" }
You should see a toolbar displayed over the name of the MCP server. Click Start, after a few seconds you should see that the server is running and that there are 4 tools available.
You can test that the MCP server is really working as expecting by ask to GitHub Copilot a question like:
1

Find me flights from New York to London
If everything works correctly, the agent will realize that it needs to use the Flights MCP server to complete the task, so it will ask the permission to connect to it. Once you approve, you will see a list of flights which matches what we have in the mock data of our MCP server.

Adding OpenAPI support

The next step is to add an OpenAPI definition for our MCP server. This is needed because Azure API Management requires an OpenAPI definition to understand our API surface. Since we’ll use APIM to enable authentication, having this definition is crucial. .NET has a few packages and helper that greatly simplifies the generation of OpenAPI definitions, so we can take advantage of them, but we’ll need to supply anyway a custom implementation. This is because the .NET helpers are able to automatically generate OpenAPI definitions only for standard HTTP operations (GET, POST, etc.), but MCP servers use a single endpoint that exposes multiple tools, so we need to manually describe the tools in the OpenAPI definition.

First, let’s add the necessary NuGet packages:

1
2


dotnet add package Microsoft.AspNetCore.OpenApi
dotnet add package Microsoft.OpenApi

Then, inside the Program.cs file, we need to add the following code before builder.Build() is called:

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146


builder.Services.AddOpenApi(options =>
{
 // Add schema transformer to fix Azure API Management compatibility
 // Azure APIM doesn't support OpenAPI 3.1 type arrays like ["integer", "string"]
 options.AddSchemaTransformer((schema, context, cancellationToken) =>
 {
 // Fix type arrays - Azure APIM expects single type, not array
 // JsonSchemaType is a flags enum, so we check if it has multiple types set
 if (schema.Type is not null)
 {
 var type = schema.Type.Value;

 // Check if both Integer and String flags are set (common for query params)
 if (type.HasFlag(JsonSchemaType.Integer) && type.HasFlag(JsonSchemaType.String))
 {
 // Keep only Integer for numeric types
 schema.Type = JsonSchemaType.Integer;
 // Remove the pattern that was added for string parsing
 schema.Pattern = null;
 }
 else if (type.HasFlag(JsonSchemaType.Number) && type.HasFlag(JsonSchemaType.String))
 {
 // Keep only Number for numeric types
 schema.Type = JsonSchemaType.Number;
 schema.Pattern = null;
 }
 }

 return Task.CompletedTask;
 });

 options.AddDocumentTransformer((document, context, cancellationToken) =>
 {
 document.Info = new OpenApiInfo
 {
 Title = "Flights API",
 Version = "v1",
 Description = "Flight search and booking API with MCP (Model Context Protocol) support.\n\n" +
 "## REST API Endpoints\n" +
 "Standard REST endpoints for flight search and retrieval.\n\n" +
 "## MCP Endpoints\n" +
 "This API also exposes MCP tools at `/mcp` for AI agent integration:\n\n" +
 "- **SearchFlights**: Search for flights by origin, destination, and departure date\n" +
 "- **GetFlightDetails**: Get detailed information about a specific flight\n" +
 "- **GetAirportsOrigins**: Get list of all available departure airports\n" +
 "- **GetAirportsDestinations**: Get list of all available destination airports\n\n" +
 "MCP endpoints use the Model Context Protocol for AI tool calling.",
 Contact = new OpenApiContact
 {
 Name = "Flights API Support"
 }
 };

 // Add tags to the document
 if (document.Tags is not null)
 {
 document.Tags.Add(new OpenApiTag
 {
 Name = "Flights",
 Description = "Flight search and retrieval operations"
 });
 document.Tags.Add(new OpenApiTag
 {
 Name = "MCP",
 Description = "Model Context Protocol endpoints for AI agent integration. Connect to /mcp using HTTP streaming transport."
 });
 }

 // Add MCP endpoint to paths (MCP uses Streamable HTTP transport, not auto-documented by ASP.NET)
 var mcpTag = new OpenApiTagReference("MCP");

 var mcpPathItem = new OpenApiPathItem
 {
 Operations = new Dictionary<HttpMethod, OpenApiOperation>
 {
 [HttpMethod.Post] = new OpenApiOperation
 {
 Tags = new HashSet<OpenApiTagReference> { mcpTag },
 Summary = "MCP Streamable HTTP endpoint",
 Description = "Single endpoint for MCP communication using the Streamable HTTP transport. " +
 "Clients send JSON-RPC 2.0 messages via POST and receive responses. " +
 "Supports streaming responses via chunked transfer encoding.\n\n" +
 "Available tools:\n\n" +
 "- **SearchFlights**: Search flights by origin, destination, and date\n" +
 "- **GetFlightDetails**: Get details for a specific flight number\n" +
 "- **GetAirportsOrigins**: List all departure airports\n" +
 "- **GetAirportsDestinations**: List all destination airports",
 OperationId = "McpStreamableHttp",
 RequestBody = new OpenApiRequestBody
 {
 Description = "JSON-RPC 2.0 message for MCP protocol",
 Required = true,
 Content = new Dictionary<string, OpenApiMediaType>
 {
 ["application/json"] = new OpenApiMediaType
 {
 Schema = new OpenApiSchema
 {
 Type = JsonSchemaType.Object,
 Properties = new Dictionary<string, IOpenApiSchema>
 {
 ["jsonrpc"] = new OpenApiSchema
 {
 Type = JsonSchemaType.String,
 Description = "JSON-RPC version, must be \"2.0\""
 },
 ["id"] = new OpenApiSchema
 {
 Type = JsonSchemaType.String,
 Description = "Request identifier"
 },
 ["method"] = new OpenApiSchema
 {
 Type = JsonSchemaType.String,
 Description = "MCP method name (e.g., \"tools/call\", \"tools/list\", \"initialize\")"
 },
 ["params"] = new OpenApiSchema
 {
 Type = JsonSchemaType.Object,
 Description = "Method parameters"
 }
 }
 }
 }
 }
 },
 Responses = new OpenApiResponses
 {
 ["200"] = new OpenApiResponse
 {
 Description = "JSON-RPC response. May use chunked transfer encoding for streaming responses."
 },
 ["401"] = new OpenApiResponse
 {
 Description = "Unauthorized - Valid authentication token required"
 }
 }
 }
 }
 };

 document.Paths["/mcp"] = mcpPathItem;

 return Task.CompletedTask;
 });
});

This code is a bit verbose, but it simply adds an OpenAPI document transformer that customizes the OpenAPI definition for our MCP server. It adds a new path for the /mcp endpoint and describes the available tools and their parameters. The code addresses also a compatibility issue. Azure API Management doesn’t support OpenAPI 3.1 type arrays (like ["integer", "string"]), which the .NET OpenAPI generator sometimes creates for query parameters. We need to add a schema transformer to convert these type arrays to single types Now, if we run our MCP server locally, we’ll be able to see the OpenAPI definition at http://localhost:5213/openapi/v1.json.

Publish the MCP server to Azure App Service

Let’s publish now our MCP server to Azure App Service. This is the first deployment step before we can enable authentication using Azure API Management. Open the Azure portal and create a new Azure App Service instance:

Click on Create a resource and search for Web App, then click on Create.
Choose your subscription and create a dedicated resource group (for example, rg-flights-mcp)
Give it a meaningful name
Choose Code as publish target
Pick .NET 10 (LTS) as runtime stack
Choose Linux as Operating System
Choose a region close to you
Under Pricing plans, pick an existing App Service plan you might already have or create a new one. For testing purposes, the free tier (F1) should be enough.

Once the App Service is created, we can publish our MCP server to it. You can do this directly from Visual Studio Code:

Right-click on an empty space in the Explorer panel and select Deploy to Web App…
Log-in with your Azure account if prompted
Select the subscription where you created the App Service
Select the App Service you just created
Confirm that you want to deploy the project (this will overwrite any existing content in the App Service)
Wait for the deployment to complete

Now that the project is deployed, we can test it by navigating to https://<your-app-service-name>.azurewebsites.net/mcp in the browser. You can find the URL in the main page of your Azure App Service in the Azure portal.

You should see the following message:

1
2
3
4
5
6
7
8


{
 "error": {
 "code": -32000,
 "message": "Bad Request: Mcp-Session-Id header is required"
 },
 "id": "",
 "jsonrpc": "2.0"
}

This is good news! It means that our MCP server is up and running. The error message is expected because the /mcp endpoint requires a valid MCP session to be established through a POST request with the appropriate headers. We need a valid MCP client to test it properly.

The easiest way to test it is to use, again, Visual Studio Code with the GitHub Copilot extension. Go back to the mcp.json file that was previously created and replace the localhost URL with the URL of your Azure App Service, so that it looks like this:

1
2
3
4


"flights-mcp": {
 "url": "https://<your-app-service-name>.azurewebsites.net/mcp",
 "type": "http"
}

If you press the Start link which is displayed over the MCP server entry, you should see after a few seconds that the server is properly running and that there are 4 tools available. If you click on the tools icon in GitHub Copilot chat box, you’ll find the flights-mcp server in the list and, if you expand it, you’ll see the 4 tools, which are matching the 4 methods we exposed in the FlightsTool class.

Finally, if you really want to be sure that the MCP server is working correctly, you can try to ask the same question:

1

Find me flights from New York to London

GitHub Copilot will realize that it needs to use the Flights MCP server to complete the task, so it will ask the permission to connect to use the search_flight tool.

After pressing Allow, you will get a list of flights matching the criteria.

Securing the MCP server with Microsoft Entra

It’s now time to secure the MPC server using Microsoft Entra. The first step is to register our MCP server as an application in Microsoft Entra. This will allow us to manage the authentication and authorization for our MCP server.

Go to the Azure portal and navigate to the App registrations section.
Click on New registration to create a new application.
Give it a meaningful name, like Flights MCP Server.
Choose the appropriate supported account types. For this demo, we’ll choose Accounts in this organizational directory only, but you make the application also multi-tenant if you want to allow users from other organizations to access it.
Click on Register to create the application.

Now we have to tweak a few things.

Expose an API

Next, we need to expose an API for our MCP server application. This will allow clients to request access tokens to call our MCP server. Click on the Expose an API section in the left menu, then click on Add a scope. In the dialog that opens up, we need to set the following values:

Scope name: access_as_user
Who can consent?: choose Admins and users
Admin consent display name: set Access Flights MCP Server
Admin consent description: set Allows the app to access the Flights MCP Server on behalf of the signed-in user.
User consent display name: set Access Flights MCP Server
User consent description: set Allows the app to access the Flights MCP Server on your behalf.
State: Enabled

Then click on Add scope to create the scope.

Now, under Application ID URI, click on Add a URI and set it to api://<client-id>, where <client-id> is the Application (client) ID of your MCP server application. This will be used as the audience (aud) claim in the access tokens. By default, it will be automatically set with the proper client id of this application.

Protecting the MCP server with Azure API Management

We’re going to leverage Azure API Management (APIM) as a way to easily enable authentication using Microsoft Entra. APIM provides a simple way to add an authentication layer in front of our MCP server without having to modify the server code itself. Additionally, we can easily expose the endpoints we need to manage the full Entra authentication flow.

As the first step, go to your Azure App Service and head to the section API -> API definition. In the API definition location field, you need to provide the URL of the OpenAPI definition we created earlier. This is available at https://<your-app-service-name>.azurewebsites.net/openapi/v1.json.

Now go to the section API -> API Management. Under API Management, click on Create new and move forward with the creation of a new API Management instance. You can choose the free tier (Developer) for testing purposes.

Once the APIM instance is created, select it under the API Management section of your Azure App Service. Under API, leave the Create New option and make sure to check the option Use OpenAPI specification.

The Azure portal will create a new APIM configuration with the APIs described in the OpenAPI definition. However, we need to tweak a few things to enable authentication.

To double check that you did everything correctly, go to the APIM instance you just created and click on the APIs section in the left menu. Click on the API that was created and move to the Design tab. You should see the /mcp endpoint listed there.

In the screenshot above, you can see other endpoints than the /mcp one. This is because the sample server published on GitHub exposes the same MCP functionality both through the MCP protocol and through standard REST endpoints. In your case, you should see only the /mcp endpoint.

Validating the Entra tokens

The first step is to add a policy to validate the access tokens issued by Microsoft Entra. This will make sure that, if the request doesn’t have a valid token, it will be rejected. To do this, go to the APIs section of your APIM instance, then select the API that was created for your MCP server. Click on the Design tab, then select the All operations option.

APIM policies are applied in four stages: inbound (before forwarding to backend), backend (when calling the backend service), outbound (before returning to client), and on-error. For authentication, we only need to modify the inbound stage to validate tokens before any requests reach our MCP server

Click on the Policy editor icon in the Inbound processing box to edit the policies, which are represented in XML.

APIM simplifies Entra authentication through a dedicated policy designed for this purpose. This is how it looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


<inbound>
 <base />
 <set-backend-service id="apim-generated-policy" backend-id="WebApp_flightsapismcp" />
 <validate-azure-ad-token tenant-id="your-tenant-id" header-name="Authorization">
 <audiences>
 <!-- Your existing API app registration -->
 <audience>api://your-client-id</audience>
 <audience>your-client-id</audience>
 </audiences>
 </validate-azure-ad-token>
</inbound>

The <base /> and <set-backend-service /> tags are automatically added by APIM:

The <base /> tag indicates that the default inbound policies that we applied to all the operations should be applied.
The <set-backend-service /> tag configures the backend service where APIM will forward the request to. In our case, it’s the MCP server hosted on Azure App Service.

We need to add the <validate-azure-ad-token> policy, which will validate the access tokens presented in the Authorization header of incoming requests.

You need to replace your-tenant-id with the tenant ID of your Microsoft Entra instance, and your-client-id with the Application (client) ID of your MCP server application. You can find both of them in App registration under Microsoft Entra ID in the Azure portal.

This policy will validate the access tokens presented in the Authorization header of incoming requests. If the token is valid, the request will be forwarded to the MCP server; otherwise, a 401 Unauthorized response will be returned.

As next step, in order to be compliant with the MCP specification, we need to make sure that, if the authentication fails, we return a header named WWW-Authenticate with the URL where it can find all the information to access the protected resource. We can do this by adding a policy in the <on-error> section of the policy editor:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


<on-error>
 <base />
 <choose>
 <when condition="@(context.LastError.Source == "validate-azure-ad-token")">
 <return-response>
 <set-status code="401" reason="Unauthorized" />
 <set-header name="WWW-Authenticate" exists-action="override">
 <value>Bearer realm="https://flightsapismcp-apim.azure-api.net", resource_metadata="https://flightsapismcp-apim.azure-api.net/.well-known/oauth-protected-resource"</value>
 </set-header>
 <set-header name="Content-Type" exists-action="override">
 <value>application/json</value>
 </set-header>
 <set-body>@{
 return "{\"statusCode\": 401, \"message\": \"Azure AD JWT not present\"}";
 }</set-body>
 </return-response>
 </when>
 </choose>
</on-error>

Now we need to enable the endpoints which are needed to support the OAuth 2.0 authorization code flow and that tell to clients how to properly authenticate.

Supporting the oAuth 2.0 authorization code flow

The other step is to expose the endpoints needed to support the OAuth 2.0 authorization code flow. When a client application (like an agent) needs to connect to a service protected with OAuth 2.0 protocol, it looks for the necessary endpoints to start the authentication flow under the /.well-known/oauth-authorization-server path of the API. When the API is called, it returns a JSON payload like this:

1
2
3
4
5
6
7
8


{
 "issuer": "https://login.microsoftonline.com/your-tenant-id/v2.0",
 "authorization_endpoint": "https://login.microsoftonline.com/your-tenant-id/oauth2/v2.0/authorize",
 "token_endpoint": "https://login.microsoftonline.com/your-tenant-id/oauth2/v2.0/token",
 "scopes_supported": ["api://your-client-id/.default"],
 "response_types_supported": ["code", "token", "id_token", "code id_token"],
 "code_challenge_methods_supported": ["S256"]
}

Thanks to these information, a client who wants to authenticate against our MCP server will be able to discover the authorization and token endpoints to use.

One of the best features of APIM is that we can add new endpoints to our APIs without having to code them. To add the /.well-known/oauth-authorization-server endpoint, go to the APIs section of your APIM instance, then select the API that was created for your MCP server. Click on the Design tab, then click on the + Add operation button.

Set the following properties:

Display name: OAuth 2.0 Authorization Server Metadata
Name: oauth-authorization-server
URL: GET - /.well-known/oauth-authorization-server

After the operation has been created, click on it and open the policy editor for the Inbound processing section. Here, we need to add a policy that returns the JSON metadata we saw earlier. This is how the inbound policy looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


<inbound>
 <return-response>
 <set-status code="200" reason="OK" />
 <set-header name="Content-Type" exists-action="override">
 <value>application/json</value>
 </set-header>
 <set-body>{
 "issuer": "https://login.microsoftonline.com/your-tenant-id/v2.0",
 "authorization_endpoint": "https://login.microsoftonline.com/your-tenant-id/oauth2/v2.0/authorize",
 "token_endpoint": "https://login.microsoftonline.com/your-tenant-id/oauth2/v2.0/token",
 "scopes_supported": ["api://your-client-id/.default"],
 "response_types_supported": ["code", "token", "id_token", "code id token"],
 "code_challenge_methods_supported": ["S256"]
 }</set-body>
 </return-response>
</inbound>

Remember, also in this case, to replace your-tenant-id and your-client-id with the actual values of your MCP server application registration.

Notice how we have removed the <base /> tag from the policy, which means that the inbound policy we have assigned to all the operations won’t be applied here. This is important, since the endpoint that provides the OAuth 2.0 metadata must be publicly accessible without authentication.

The second endpoint we need to add is the one we have included in the WWW-Authenticate header we created before. This endpoint provides metadata about the protected resource and is located at /.well-known/oauth-protected-resource. To add it, repeat the same steps we did before to create a new operation with the following properties:

Display name: OAuth 2.0 Protected Resource Metadata
Name: oauth-protected-resource
URL: GET - /.well-known/oauth-protected-resource

Then, open the policy editor for the Inbound processing section and add the following policy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


<inbound>
 <choose>
 <!-- Handle MCP-specific OAuth metadata endpoint -->
 <when condition="@(context.Request.Url.Path.EndsWith("/.well-known/oauth-protected-resource"))">
 <return-response>
 <set-status code="200" />
 <set-header name="Content-Type" exists-action="override">
 <value>application/json</value>
 </set-header>
 <set-body>@{
 return "{\"resource\":\"https://flightsapismcp-apim.azure-api.net/mcp\",\"authorization_servers\":[\"https://login.microsoftonline.com/common/v2.0\"]}";
 }</set-body>
 </return-response>
 </when>
 <!-- Your existing policies continue below -->
 </choose>
 <!-- Rest of your inbound policies -->
</inbound>

Thanks to this policy, when a client calls the /.well-known/oauth-protected-resource endpoint, it will receive a JSON payload that describes the protected resource and the authorization servers that can be used to access it.

Securing the App Service

There’s one last step to do. Right now our APIM is properly protecting the MCP server, but the Azure App Service is still publicly accessible. To avoid that, we need to configure the Azure App Service to accept requests only from the APIM instance. To do this, go to the Settings -> Networking section of your Azure App Service. Here, we need to add a new rule that allows traffic only from Azure services.

Under Inbound traffic configuration, click on Enabled from all networks near Public network access.
Change it to Enabled from selected virtual networks and IP addresses
Under Main site, set Unmatched rule action to Deny
Click Add to add a new rule
Configure it in the following way:
1. Name: Allow APIM
2. Priority: 100
3. Action: Allow
4. Type: Service Tag
5. Service Tag: look for AzureCloud
Click on Add rule to create the rule
Click on Save to apply the changes

Now, to verify that everything is working correctly, try to access the Azure App Service URL directly in the browser: https://<your-app-service-name>.azurewebsites.net/mcp. You should see a 403 Forbidden error, which means that the Azure App Service is properly blocking direct access. However, thanks to the rule we have created, Azure services (like our APIM instance) are still allowed to access it.

Testing the job

We are ready to test the job we did. The easiest way to do that is by using Visual Studio Code with the GitHub Copilot extension, just like we did before. However, Visual Studio Code (like any other application which wants to access our MCP server) behaves like a client and, as such, it needs a dedicated application registration in Microsoft Entra to be able to request access tokens. We need to register a new application in Microsoft Entra with the proper permissions to access to the MCP server we have just protected.

Go to the Azure portal and navigate to the App registrations section.
Click on New registration to create a new application.
Give it a meaningful name, like Flights MCP Client.
Choose the appropriate supported account types. For this demo, we’ll choose Accounts in this organizational directory only, but you can make the application also multi-tenant if you want to allow users from other organizations to access it.
Click on Register to create the application.

Now move to the Authentication section and click on Add Redirect Uri. We must add the redirect URI that Visual Studio Code will use to receive the authentication response. Choose Web and add the following Redirect URI:

1

https://vscode.dev/redirect

There’s a second one we need to add, but need to add it manually in the manifest because it’s based on the HTTP protocol (instead of HTTPS) and, as such, the UI will block us. Under Manage, click on Manifest and look for the section web - > redirectUris. Add the following URI to the list:

1

http://127.0.0.1:33418

This is how it should look like:

1
2
3
4
5
6


web: {
 "redirectUris": [
 "https://vscode.dev/redirect",
 "http://127.0.0.1:33418"
 ]
}

Now we need to make is to authorize this client to call the MCP server. Under Manage, move under API permissions, then:

Click on Add a permission
Choose APIs my organization uses
Search for Flights MCP Server (which is the name of the first app registration we have created)
Under What type of permission does your application require?, choose Delegated permissions.
Under Permissions at the bottom, check access_as_user and click Add permissions.

The last step is to create a client secret that we’ll have to provide to VS Code:

Go to Certificates & Secrets.
Click on New client secret.
Give it a meaningful name and choose an expiration.
You’ll be redirected to the page, where you’ll see the client secret in clear under the Value column. Make sure to note it down because, after you’ll move away from this page, the secret will be hidden forever.

Now go back to the Overview page and take note also of the Application (client) ID; we’ll need it in a moment.

Setting up Visual Studio Code

Head back to the mcp.json file we have created before, click on Stop above the MCP server entry to stop it, then restart Visual Studio Code.

Now open it again on the mcp.json file and start the server. At first, Visual Studio Code will show a message to communicate that the MCP server doesn’t support Dynamic Client Registration. This is expected. Just click on Copy URIs & Proceed.

Visual Studio Code will ask for the client id of the application to use to access the MCP server. Here, paste the Application (client) ID of the Flights MCP Client application registration we have just created.

Make sure to not confuse the two app registration. The one you need to use now is the Flights MCP Client app we have created just now, not the Flights MCP Server app we created at the beginning.

Then, it will ask for a secret. Provide the one you have just created for the Flights MCP Client app.

After that, VS Code will ask you to authenticate. Click on Allow. It will open a browser on the authorization endpoint of Microsoft Entra, which has been retrieved by the APIM instance using the /.well-known/oauth-authorization-server endpoint we created earlier.

Log-in with your Microsoft Entra credentials and then, once you see the success message, authorize the browser to go back to Visual Studio Code. You might be prompted to perform a second login. Complete the authentication flow when prompted—this second login verifies your identity for the MCP server access.

If everything works correctly, now you should see the MCP server running and with 4 tools available. This time, however, the server is protected with Microsoft Entra authentication.

Again, if you want to be sure that everything is working correctly, you can try to ask a question like:

1

Find me me flights from New York to London

GitHub Copilot should prompt you to call the MCP server and, after approval, it will return a list of flights matching the criteria.

We did it!

Wrapping up

In this article, we have seen how to create a simple MCP server using .NET and ASP.NET Web API. We have also seen how to publish it to Azure App Service and how to protect it using Microsoft Entra and Azure API Management. In this post, we tested the agent using Visual Studio Code with the GitHub Copilot extension, but you can use any other MCP-compatible client to interact with the server. In the next posts, we’re going to take a look on how we can consume it from the Microsoft 365 Copilot ecosystem. Stay tuned!

You can find the complete source code of the MCP server on GitHub.

Using Model Context Protocol in agents - Copilot Studio and authentication with API key

Thu, 19 Jun 2025 00:00:00 +0000

In all the examples we have seen so far when implementing the Model Context Protocol (MCP), we have assumed that our backend didn’t require any authentication. This is a common scenario when we are just experimenting with the protocol, but in real-world applications, we often need to connect to APIs that require some form of authentication.

Let’s say we want to build an MCP server that we can connect to an agent to get information about parks in the US. The National Park Service (NPS) provides a public API that you can use to retrieve information about pretty much everything related to US national parks: generic data, activities, places. This API is well documented here: https://www.nps.gov/subjects/developer/api-documentation.htm. The site includes a page created with Swagger, which you can use to test the API without writing any code. However, the first thing you will notice is that all the endpoints have a lock icon next to them. At the top of the page, you will see an Authorize button. If you click on it, you will be asked to provide an API key, which will be passed as a query parameter called api_key.

How can we manage this requirement if we want to build an MCP server from these APIs and use it with an agent created in Copilot Studio? Let’s see that! Before we start writing code, however, make sure to request an API key from the NPS website. It’s free!

Creating our MCP server

This time, we’ll learn something new and build the server using TypeScript and the corresponding MCP SDK.

Let’s start by creating a new TypeScript project. To perform the following tasks, you will need to have Node.js installed on your machine. If you don’t have it yet, you can download it from the official website.

Open a terminal and create a new directory for your project:

1
2


mkdir parks-http-typescript
cd parks-http-typescript

Initialize a new Node.js project:
1

npm init -y
Install TypeScript and ts-node as development dependencies. ts-node allows you to run TypeScript files directly without compiling them first:
1 2

npm install typescript --save-dev npm install ts-node --save-dev
Since we’re going to use the Streamable HTTP transport layer, we also need to install the express framework, which will help us expose the MCP server through an HTTP endpoint:
1

npm install express
Initialize a TypeScript configuration file:
1

npx tsc --init

Open the tsconfig.json file with an editor (like Visual Studio Code) and set the following properties:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


{
"compilerOptions": {
 "target": "ES2020",
 "module": "NodeNext",
 "moduleResolution": "nodenext",
 "outDir": "./dist",
 "rootDir": "./src",
 "strict": true,
 "esModuleInterop": true,
 "skipLibCheck": true,
 "forceConsistentCasingInFileNames": true,
 "sourceMap": true
},
"include": ["src/**/*"],
"exclude": ["node_modules"]
}

Adjust the package.json file to add a script for running the application using ts-node:
1 2 3 4

"scripts": { "build": "tsc", "start": "ts-node src/app.ts" },
As the final step, let’s install the MCP SDK for TypeScript:
1

npm install @modelcontextprotocol/sdk

Now we can open our solution in Visual Studio Code and start writing the actual implementation. Let’s start by creating a new app.ts file in the src folder. This will be the primary entry point of our application.

First, let’s see how we can set up the MCP server using the TypeScript SDK:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


const server = new McpServer({
 name: "mcp-streamable-http",
 version: "1.0.0",
});

const transport: StreamableHTTPServerTransport =
 new StreamableHTTPServerTransport({
 sessionIdGenerator: undefined, // set to undefined for stateless servers
 });

const setupServer = async () => {
 await server.connect(transport);
};

Similar to what we have seen with the C# SDK, the TypeScript SDK greatly simplifies setting up the MCP server:

First, we need to create a new instance of the McpServer class, passing the name and version of our MCP server.
Next, we need to define the transport layer we want to use. In this case, we are using the StreamableHTTPServerTransport. For simplicity, we’re going to use a stateless server, so we set the sessionIdGenerator to undefined.
Finally, we call the connect() method on the server instance, passing the transport layer we just created.

Since we’re using the Streamable HTTP transport layer, we need a way to expose it through an HTTP endpoint. In C# and .NET we were using the built-in ASP.NET Core middleware to map the MCP server to an HTTP endpoint. In Node.js, we can use the express framework to achieve the same result.

1
2


const app = express();
app.use(express.json());

Now we can implement the MCP endpoint:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


app.post("/mcp", async (req: Request, res: Response) => {
 try {
 await transport.handleRequest(req, res, req.body);
 } catch (error) {
 console.error("Error handling MCP request:", error);
 if (!res.headersSent) {
 res.status(500).json({
 jsonrpc: "2.0",
 error: {
 code: -32603,
 message: "Internal server error",
 },
 id: null,
 });
 }
 }
});

We’re simply implementing the specs of the MCP protocol. We’re setting up an endpoint (/mcp) that will handle POST requests. When we receive a request, we call the handleRequest() method of the transport layer, passing the request and response objects. This process will automatically manage the communication between the MCP server and the MCP client.

If an error occurs while handling the request, we log it to the console and return a 500 Internal Server Error response.

Just to make sure that our endpoint responds properly, let’s also implement the two other HTTP verbs, GET and DELETE. In this case, however, we’ll simply return an error response, since the MCP protocol only uses POST requests for communication:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


app.get("/mcp", async (req: Request, res: Response) => {
 console.log("Received GET MCP request");
 res.writeHead(405).end(
 JSON.stringify({
 jsonrpc: "2.0",
 error: {
 code: -32000,
 message: "Method not allowed.",
 },
 id: null,
 })
 );
});

app.delete("/mcp", async (req: Request, res: Response) => {
 console.log("Received DELETE MCP request");
 res.writeHead(405).end(
 JSON.stringify({
 jsonrpc: "2.0",
 error: {
 code: -32000,
 message: "Method not allowed.",
 },
 id: null,
 })
 );
});

Finally, we can spin up the server by listening to a specific port, in this case 3000:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


const PORT = process.env.PORT || 3000;
setupServer()
 .then(() => {
 app.listen(PORT, () => {
 console.log(`MCP Streamable HTTP Server listening on port ${PORT}`);
 });
 })
 .catch((error) => {
 console.error("Failed to set up the server:", error);
 process.exit(1);
 });

First we call the setupServer() function we defined earlier to start the MCP server. Once it’s up & running, we call the listen() method on the Express app to start listening for incoming requests on the specified port.

Exposing the APIs

For the purpose of this demo, we’re just going to expose one of the many endpoints provided by the NPS API — specifically, the one that returns a list of parks. First, let’s create a new file called parksService.ts in the src folder. In this file, we will implement the logic to call the NPS API and return the list of parks.

Let’s start with the definition of the interfaces that represent the data we expect to receive from the NPS API:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40


export interface Park {
 id: string;
 url: string;
 fullName: string;
 parkCode: string;
 description: string;
 latitude: string;
 longitude: string;
 latLong: string;
 activities: any[];
 topics: any[];
 states: string;
 contacts: any;
 entranceFees: any[];
 entrancePasses: any[];
 fees: any[];
 directionsInfo: string;
 directionsUrl: string;
 operatingHours: any[];
 addresses: any[];
 images: any[];
 weatherInfo: string;
 name: string;
 designation: string;
}

export interface ParksResponse {
 total: string;
 data: Park[];
 limit: string;
 start: string;
}

export interface GetParksParams {
 parkCode?: string;
 stateCode?: string;
 limit?: number;
 start?: number;
 q?: string;
}

We’re going to use the Axios library to simplify making HTTP requests to the NPS API, so let’s install it as a dependency in the terminal:

1

npm install axios

Now let’s add to our parksService.ts file the definition of a class that we can use to interact with the NPS API:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


export class ParksService {
 private axiosInstance: AxiosInstance;
 private readonly baseUrl = 'https://developer.nps.gov/api/v1';

 constructor() {
 this.axiosInstance = axios.create({
 baseURL: this.baseUrl,
 headers: {
 'Accept': 'application/json',
 },
 });
 }
}

We’re setting up an axiosInstance object using the base URL of the NPS API and setting the Accept header to application/json to indicate that we expect a JSON response. Next, let’s implement a method to retrieve the list of parks:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


async getParks(apiKey: string, params: GetParksParams = {}): Promise<ParksResponse> {
const response = await this.axiosInstance.get<ParksResponse>('/parks', {
 params: {
 ...params,
 api_key: apiKey,
 },
});
// Return only the first 3 items of the collection
const data = response.data;
return {
 ...data,
 data: data.data.slice(0, 3),
};

We use the axiosInstance we have set up to make a GET request to the /parks endpoint, which will return the ParksResponse object that we have previously defined.

Notice also that we’re passing an apiKey parameter to the getParks() functio, which we’re including in the HTTP request to the API as a query parameter named api_key. This means that the request that Axios performs will look like https://developer.nps.gov/api/v1/parks?api_key=abcdef

Please note: to simplify the demo, we’re returning only the first 3 items of the collection, otherwise the response might be too big to be processed by Copilot Studio.

Now that we have our service ready, we can implement the MCP tool that will expose this functionality to agents. But, first of all, we must manage the authentication process.

Managing authentication with an API key

When you’re working with custom connectors with Copilot Studio, you can set up the authentication with different methods. One of them is API key authentication, and it allows to specify in which way the custom connector will pass the key to the API backend. It could be a header in the HTTP request, a query parameter, etc.

In our case, we’ll go with the header option: we’ll read the API key from a header in the HTTP request named x-api-key. This is a common practice when working with APIs that require an API key for authentication.

This is how we need to change the /mcp endpoint in the app.ts file to support this scenario:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


var myApiKey = '';

app.post("/mcp", async (req: Request, res: Response) => {
 const apiKey = req.header("x-api-key");
 if (apiKey && typeof apiKey === 'string' && apiKey.trim() !== '') {
 myApiKey = apiKey;
 }
 console.log("x-api-key header:", apiKey);
 try {
 await transport.handleRequest(req, res, req.body);
 } catch (error) {
 console.error("Error handling MCP request:", error);
 if (!res.headersSent) {
 res.status(500).json({
 jsonrpc: "2.0",
 error: {
 code: -32603,
 message: "Internal server error",
 },
 id: null,
 });
 }
 }
});

First, we’re setting a variable called myApiKey to store the API key. The reason for this is that Copilot Studio doesn’t pass the API key every time it calls the MCP server, but only the first time. So, we need to store it in a variable to use it later when calling the NPS APIs.

Then, we change the app.post() method to read the x-api-key header from the request. If the header is present and is a non-empty string, we store it in the myApiKey variable.

Important! This is a simplified approach to storing the API key, which works fine when you’re running the MCP server locally. However, once you deploy it on Azure or any other cloud provider, you should use a more reliable way to store the API key, since the MCP server might be stateless and unable to persist the value of the myApiKey variable across different requests.

Implementing the MCP tool

Now that we have the API key stored in a variable, we can implement the MCP tool that will expose the functionality to retrieve the list of parks. The TypeScript SDK for MCP makes this very simple. Let’s add the following code to the app.ts file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


import { ParksService } from './services/parksService';

const parkService = new ParksService();

server.tool(
 "getParks",
 "Get the list of parks",
 {
 stateCode: z.string().describe("The US state to filter parks by, e.g., 'CA' for California"),
 },
 async ({stateCode}) => {
 const response = await parkService.getParks(myApiKey, { stateCode: stateCode });
 return {
 content: [{ type: "text", text: JSON.stringify(response) }]
 };
 }
);

Tools are defined by calling the tool() function of the McpServer instance. The first parameter is the name of the tool, the second is a description, which is very important to help the LLM to understand its purpose. In this case, the tool supports an input parameter, which is the US state we want to filter parks by. We use the Zod library to define an input parameter called stateCode, which is a string. Also in this case, we must provide a description of the parameter, which will be used by the LLM to understand how to properly fill it.

Then, we can write the code that we want to execute when the tool is called. In this case, we call the getParks() method of the parkService instance, passing the API key and the state code as parameters. The response is then returned as a content object, which contains a text field with the JSON-stringified response.

Running the MCP server

Now that we have implemented the MCP server and the tool to retrieve the list of parks, we can run the server. In the terminal, run the following command:

1

npm run start

If everything goes well, you will see the following output:

1

MCP Streamable HTTP Server listening on port 3000

If you open the browser and navigate to http://localhost:3000/mcp, you will see the following response:

1

{"jsonrpc":"2.0","error":{"code":-32000,"message":"Method not allowed."},"id":null}

This is the expected response, since the MCP protocol only supports POST requests. When you hit the URL with the browser, instead, you’re just performing a GET.

Since you’re still in Visual Studio Code, use the port forwarding feature we explained in one of the previous posts to expose the MCP server to the Internet. We’ll need to expose the port 3000.

Setting up the custom connector in Copilot Studio

Now we can move to Copilot Studio and set up the custom connector to connect to our MCP server. We can do that in the Custom Connector section from the Power Platform portal. The direct URL is https://make.powerapps.com/environments/<guid>/customconnectors, where <guid> is the ID of your environment.

In the previous post we have seen how to create a custom connector for a MCP server using the new Copilot Studio capabilities, so in this post we’re going to focus on the authentication part. After you have set the proper URL of your MCP server in the Host field (make sure also to set the Base URL to /mcp, which is the endpoint we have defined in the MCP server), you can click on the Security tab and select API Key as the authentication type. You will be asked to configure how the custom connector will provide the API key to your backend. Set it as follows:

Parameter label: API key
Parameter name: x-api-key
Parameter location: Header

Now you can move on and click on the Create connector button to create the custom connector.

Using the connector in an agent

The way you’re going to use this new MCP server in an agent in Copilot Studio is no different from the way we used it in the other posts. You’ll need to go to the Tools section of your agent and click on Add tool. Then, select the Model Context Protocol label and you will see the new MCP connector you’ve just created. Click on it to add it to your agent.

The only difference is that, when you are asked to create a new connection, you will need to provide the API key you previously obtained from the NPS website:

Once you have added the tool to your agent, you should be able to click on it in the Tools section and see the list of available tools:

And now, you can test the tool by writing a prompt that requires the use of the getParks tool, such as:

1

Give me the list of parks in the Washington state

In the screenshot below, you can see how the agent used the getParks tool to retrieve the list of parks, automatically providing the state code WA as the input parameter.

Wrapping up

In this post, we have seen how to implement a Model Context Protocol (MCP) server using TypeScript and how to use it to expose an API which requires authentication with an API key, in this case the NPS APIs, which you can use to retrieve information about US national parks.

We have also seen how to tweak our custom connector in Copilot Studio so that we can connect to the MCP server and automatically pass the API key to the backend.

You can find the working sample of the solution we’ve built in this post in the official Copilot Studio MCP repository on GitHub.

Happy coding!

Using Model Context Protocol in agents - Streamable HTTP support in Copilot Studio

Wed, 18 Jun 2025 00:00:00 +0000

In a post I published a while ago, we explored the first preliminary support for the Model Context Protocol (MCP) in Copilot Studio. Thanks to this integration, we can empower agents to use tools and services exposed through the MCP protocol. Since then, however, Copilot Studio has evolved and, during Microsoft Build, the team officially launched the feature in Public Preview by adding a set of new capabilities that simplify the integration of MCP tools.

Let’s explore these changes!

Goodbye SSE, welcome Streamable HTTP

In previous posts, we explored how MCP supports different transport layers to manage communication between the client and server. Copilot Studio, being a platform running in the cloud, supported only SSE, which is based on HTTP. The other transport layer, called stdio, is based on a standard input/output stream and, as such, is suitable only for scenarios in which both the client and server are running in a local environment.

However, SSE is being deprecated for security reasons and has been replaced by another transport layer, still based on HTTP but much more efficient and secure: Streamable HTTP. Copilot Studio now supports this new transport layer, which has an extra advantage: we no longer need to apply the workaround we used to make SSE work with Copilot Studio, which was providing a custom MCP mapping to inject the full URL into the layer implementation.

Let’s see how we can update our MCP server to support the new Streamable HTTP transport layer.

Using Streamable HTTP in C#

The good news is that, if you’re using C# and .NET to implement your MCP server, you just need to upgrade the MCP NuGet package to the latest version which, at the time of writing, is 0.2.0-preview.3. Thanks to this updated package, the code we have already written to implement our Employee Vacations MCP server will automatically support both SSE and Streamable HTTP, thanks to the WithHttpTransport() extension method::

1
2
3
4


builder.Services
 .AddMcpServer()
 .WithHttpTransport()
 .WithToolsFromAssembly();

The second step is to delete the MapAbsoluteEndpointUriMcp() method we implemented to customize the MCP mapping. We no longer need it; we can just use the standard mapping by calling the app.MapMcp() method before app.Run():

1
2
3


app.MapMcp();

app.Run();

Now, if you run your MCP server after these changes and open your browser to the server URL (which, by default, is http://localhost:5248), you will see two things:

The old /sse endpoint (http://localhost:5248/sse) is still working to maintain backward compatibility with existing clients and it will return a message like this:

1
2


event: endpoint
data: /message?sessionId=DzQJIVL6sa65HiL3u1kx4Q

The Streamable HTTP layer is available at the root endpoint (so just calling the base URL http://localhost:5248) and it will return a JSON response like this:

1

{"error":{"code":-32001,"message":"Session not found"},"id":"","jsonrpc":"2.0"}

Now our MCP server officially supports the new Streamable HTTP transport layer, and we can use it with Copilot Studio.

Using Streamable HTTP in Copilot Studio

To use the new Streamable HTTP transport layer in Copilot Studio, we need to use again the custom connector strategy we have learned in the previous post. However, this time, we won’t have to manually provide the OpenAPI specification because the Power Platform offers a built-in template we can use.

First, let’s go to the Power Platform portal and, following the same instructions we have seen in the previous post, open the Custom Connectors section. The direct URL is https://make.powerapps.com/environments/<guid>/customconnectors, where <guid> is the ID of your environment.

Now click on New custom connector at the top right corner and, this time, select Import from GitHub. Now select the following options from the pop-up dialog:

Connector Type: Custom
Branch: Dev
Connector: MCP-Streamable-HTTP

Click Continue to start the configuration of the new connector. This time, you won’t have to change anything, except the Host, which must point to the URL of your MCP server. If you’re running it locally, you will need to use the port forwarding feature of Visual Studio Code to expose the server to the Internet, like we have explained in the previous post.

Another small change to make is in the Base URL field, which by default is set to /mcp. The C# MCP SDK actually exposes it at the root, so you must replace it with / (just a single slash).

That’s it! Click on Create connector to create the new custom connector.

Using the new MCP connector in Copilot Studio

Now we can use the new MCP connector in Copilot Studio. The process didn’t change from the previous post, so we can follow the same steps. Pick up your agent and, under the Tools section, click on Add tool. Click on the Model Context Protocol label and you will see the new MCP connector you’ve just created. Click on it to add it to your agent.

Copilot Studio has added a useful new feature that lets you check if the connection with the MCP server is working properly even without performing any queries. Once the tool has been added to your agent, click on it. You will land on the detail page, with a section called Tools that queries the MCP server to get the list of available tools. If everything is working properly, you will see the list of tools exposed by your MCP server, as shown in the following screenshot:

To further test the connection, you can also use the Test your agent panel on the right and write a prompt that requires the use of one of the available tools to get a response, such as the following::

1

Give me the list of employees and their vacation days left

Wrapping up

In this post, we have seen how to switch our MCP server from the old SSE transport layer to the new Streamable HTTP one, which provides better security and performance and is now officially supported by Copilot Studio. This change greatly simplifies the integration of MCP tools in Copilot Studio, as we no longer need to provide a custom mapping implementation or manually provide the OpenAPI specification for our custom connector, since we can use a new template offered by the Power Platform.

You can find the updated MCP sample code on GitHub.

Using Model Context Protocol in agents - Pro-code agents with Semantic Kernel

Fri, 02 May 2025 00:00:00 +0000

Welcome back to another post in our series on Model Context Protocol (MCP) and its applications in agents! In the first post we explored the basic concepts and we implemented a simple MCP server using the stdio transport layer. In the second post, we reimplemented our server using the HTTP Streaming / SSE transport layer and we used the exposed tools in a low code agent created with Copilot Studio. Today, we’ll move into the pro-code world and we’ll see how to use the same MCP server with Semantic Kernel, the AI orchestration SDK from Microsoft which you should be already familiar with if you are following my blog.

Connecting to an MCP server with C#

The starting point of our sample client doesn’t even require using Semantic Kernel. The capabilities to connect to an MCP server and access to the available tools, in fact, is provided directly by the MCP SDK for C#, the same one we have used in the previous posts to implement the server. We’ll start with a very simple console application:

1

dotnet new console -n MCP.Client

Then, we must install the MCP SDK for C# using NuGet:

1

dotnet add package ModelContextProtocol --prerelease

Now we can head over to the Program.cs file and start writing some code to connect to our client. The SDK supports both transportation layers we have seen so far. Let’s start tby seeing how to connect to the MCP server using the stdio transport layer. The code is very simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


var projectPath = Path.GetFullPath(Path.Combine(AppContext.BaseDirectory, "..", "..", "..", "..", "MCPServer", "Stdio", "MCPServer.Server.Stdio.csproj"));

await using var mcpClient = await McpClientFactory.CreateAsync(

 new StdioClientTransport(new () {
 Name = "MyFirstMCP",
 Command = "dotnet",
 Arguments = ["run", "--project", projectPath],
 })
);

We create a new connection using the CreateAsync() method of the McpClientFactory class and we pass a new instance of the StdioClientTransport class. The constructor of this class takes a configuration object with the same properties that we have seen when we have configured Visual Studio Code to connect to our MCP server:

Name: the name of the connection.
Command: the command to run the server. In this case, since our server is based on a .NET console application, we are simply using dotnet to leverage the dotnet CLI.
Arguments: the arguments to pass to the command. In this case, we are passing run and --project followed by the path of the project we want to run. This is the same command we would use to run the console application from the command line.

If we want to use the HTTP Streaming / SSE transport layer, we can do it by using the same initialization code, but passing this time a SseClientTransport object as parameter to the CreateAsync() method:

1
2
3
4
5
6


await using var mcpClient = await McpClientFactory.CreateAsync(
 new SseClientTransport(new () {
 Name = "MyFirstMCP",
 Endpoint = "http://localhost:5248"
 })
);

In this case, we simply pass the name of the connection and the endpoint of the MCP server as parameters. In the implementation we did in the previous post, we used ASP.NET Core to host the MCP server, so the default endpoint is http://localhost:5248.

Once the connection is established, we can use the mcpClient to get access to the tools that our MCP server is exposing:

1
2
3
4
5


var tools = await mcpClient.ListToolsAsync();
foreach (var tool in tools)
{
 Console.WriteLine($"Tool: {tool.Name} - {tool.Description}");
}

In this case, we’re simply listing the tools that are available in the MCP server. If we run the code we have written so far, we should see the following output:

1
2
3


Tool: GetAllEmployees - Get the list of employees with their number of vacation days left
Tool: ChargeVacationDays - Charge vacation days for a given employee.
Tool: GetVacationDaysLeft - Get the vacation days left for a given employee.

These are the three tools we have implemented in the MCP server in the first post to manage the vacation days of our employees. Now it’s time to see how these tools can be used in a Semantic Kernel agent.

Using MCP in a Semantic Kernel agent

As first step, we need to install Semantic Kernel in our project. We can do it by running the following command:

1
2


dotnet add package Microsoft.SemanticKernel
dotnet add package Microsoft.SemanticKernel.Agents.Core

Now we must create a new instance of the Kernel class, which is required to interact with the AI services. In my case, I’m going to use Azure OpenAI, so I’m going to use the following code to initialize the chat completion service:

1
2
3
4
5
6
7


var endpoint = "AzureOpenAI:Endpoint";
var apiKey = "AzureOpenAI:ApiKey";
var deploymentName = "AzureOpenAI:DeploymentName";

var kernel = Kernel.CreateBuilder()
.AddAzureOpenAIChatCompletion(deploymentName, endpoint, apiKey)
.Build();

The values of endpoint, apiKey and deploymentName are just placeholders - you will need to replace them with the actual values of your Azure OpenAI service.

For the sake of simplicity, I’m setting the configuration values directly in the code. In a real application, you should use a configuration file, environment variables or a service like Azure Key Vault to store these values.

Now that we have the kernel, we can use one of the new features added in the latest version of Semantic Kernel, which is the ability to convert MCP tools into Semantic Kernel functions, that can be loaded into a plugin:

1

kernel.Plugins.AddFromFunctions("MyFirstMCP", tools.Select(x => x.AsKernelFunction()));s

We assign a name to the plugin - in this case MyFirstMCP - and we pass the list of tools we have retrieved from the MCP server. The AsKernelFunction() method is an extension method that converts the MCP tool into a Semantic Kernel function. By default, the code will give raise an error. The reason is that this method is marked as experimental and, as such, you must suppress the warning to be able to compile the code. You can do this in code or in the project’s configuration. We’ll go with the second option, since later we’re going to write another piece of code that is also marked as experimental. To do this, we need to edit the .csproj file and add the following property inside the PropertyGroup section:

1

 <NoWarn>$(NoWarn);SKEXP0001</NoWarn>

Now that the MCP tools are loaded into a Semantic Kernel plugin, we can use them in our agents in the same way we would use any other Semantic Kernel plugin. First, we need to define the execution settings so the agent will automatically call the functions in the plugin when needed, based on the prompt:

1
2
3
4


OpenAIPromptExecutionSettings executionSettings = new()
{
 FunctionChoiceBehavior = FunctionChoiceBehavior.Auto(options: new() { RetainArgumentTypes = true })
};

This is the same settings we would have used in a normal Semantic Kernel agent, except for the RetainArgumentTypes property, which is set to true. This is required to make sure that the agent will pass the arguments to the functions in the plugin in the same way they are defined in the MCP server. This is important because the MCP server is expecting the arguments to be passed in a specific format, and if we don’t set this property to true, the agent will pass the arguments in a different format, which will cause the MCP server to fail. This is a property that was added specifically to support the MCP integration, so it’s marked as experimental as well.

Now we can define the agent, using the ChatCompletionAgent class, which is the default agent in Semantic Kernel:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


ChatCompletionAgent agent = new ChatCompletionAgent()
{
 Name = "LeaveRequestAgent",
 Instructions = @"You are an agent that can help employees with their tasks about leave requests. You can answer to
questions like:
- How many days of leave do I have left?
- Charge 3 days to my leave balance.",
 Kernel = kernel,
 Arguments = new(executionSettings)
};

The agent is configured with a name and a set of instructions, plus the kernel and the execution settings we have just defined. In the instructions, we define the tasks that the agent can help with, which are aligned with the tools we have defined in the MCP server. The agent will use these instructions to understand what the user is asking for and to decide which tool to use to fulfill the request.

Finally, we can execute the agent providing a prompt as input:

1
2
3
4


string prompt = "Give me the list of the employees with their vacation days left";

var response = await agent.InvokeAsync(prompt).FirstAsync();
Console.WriteLine(response.Message.Content);

We’re calling the InvokeAsync() method of the agent, passing the prompt as input, which will return a collection of responses. In this case, since it’s a single agent, there will be only a single one, so we’re simply taking it and printing the content of the message.

If we run the code, we should see the following output, which is aligned to what we have seen in the first post (when we called the MCP server from Visual Studio Code) or in the second post (when we called it from the Copilot Studio agent):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


Here is the list of employees along with their vacation days left:

1. **Alice Johnson** - 15 days
2. **Bob Smith** - 7 days
3. **Charlie Lee** - 14 days
4. **Diana Evans** - 19 days
5. **Ethan Brown** - 24 days
6. **Fiona Clark** - 22 days
7. **George Miller** - 12 days
8. **Hannah Davis** - 8 days
9. **Ian Wilson** - 18 days
10. **Julia Adams** - 23 days

Now you have a fully fledged Semantic Kernel agent, which could be used as a standalone or, as we have seen in other posts, it could be added into a group chat with other agents to tackle more complex scenarios that requires multi-agent collaboration.

Wrapping up

In this post, we have seen the final MCP implementation of our journey: using an MCP server from a pro-code agent created with Semantic Kernel. Now we have a complete picture of the MCP ecosystem and we can use it in different scenarios, from low code to pro code. On GitHub you’ll find the complete code of the solution we have built so far, with a slightly different implementation: the pro-code client isn’t a console application, but a web application built with Blazor. The code is exactly the same, however the agent invocation is a little bit more interactive: the user can type the prompt in a text box and then click on a button to get a response. When the button is clicked, the Blazor application executes the same exact code we have seen in the console application to bootstrap the agent, connect it to the MCP server and process the prompt.

Additionally, the whole solution is wrapped in a .NET Aspire project, which is an amazing platform by Microsoft to simplify the orchestration of complex .NET applications during the development experience. Thanks to .NET Aspire, we can connect multiple projects together and easily connect them. .NET Aspire supports the most common tools and libraries that are adopted by client-server scenarios: we can easily add a Docker image, a Redis cache, a Service Bus instance, etc. And when we connect them together, a lot of magical things happens automatically under the hood, like:

Service discovery: the services can discover each other automatically, without the need to configure any connection string or endpoint. .NET Aspire automatically injects the connections strings as environment variables, that can be picked up by the other projects.
Health checks: .NET Aspire automatically configures the health checks for the services, so we can easily monitor their status and see if they are up and running.
Analytics: .NET Aspire automatically configures the logging, monitoring and tracing for the services.

Thanks to .NET Aspire, I was able to combine the MCP SSE server and the Blazor client in a single solution, which is very easy to run and debug. And, as a bonus point, you get also logging, metrics and tracing out of the box 😊

You can refer to the instructions in the repository to see the steps required to run the solution.

Happy coding!

Using Model Context Protocol in agents - Copilot Studio

Mon, 28 Apr 2025 00:00:00 +0000

In the previous post, we introduced the Model Context Protocol (MCP) and we’ve learned how platforms can use it to expose services to AI applications. You can think of it like an API platform, but optimized for AI applications: data output is optimized for LLM consumption; every service and its input are carefully described in natural language; etc. We have also built a simple MCP server that simulates a service to track vacation requests from employees. In the post, we were able to test our implementation thanks to Visual Studio Code and the recent MCP support that was added to GitHub Copilot. However, it wasn’t a very realistic scenario: I don’t think many people are really interested in using GitHub Copilot to request days off to their employer 😊. In this post, we’ll make the MCP server a bit more useful by enabling an agent to use it. Today, we will focus on the low code experience with Copilot Studio, while in the next post we’ll explore the pro-code experience with Semantic Kernel.

Implementing the SSE transport protocol

Copilot Studio has recently added preview support for MCP using the SSE transport protocol, which is a way to stream data over HTTP. In such a scenario, in fact, we can’t use the stdio approach we used in the previous post, since the client doesn’t run locally (like in the case of Visual Studio Code), but it’s an agent in the cloud. However, SSE is scheduled for deprecation in the latest MCP specifications and replaced by HTTP Streaming. The good news is that this new protocol is backward compatible with SSE, so the code we’re going to write today will work with both protocols, enabling Copilot Studio to use it as well.

Let’s move forward and reimplement the MCP server we built to use the HTTP Streaming / SSE transport protocol. The boilerplate code to manage and track vacation days request will be the same, and also the implementation of the class that exposes the functions as MCP tools. However, we’ll need to slightly change the underline architecture, since HTTP Streaming / SSE work in a different way than stdio. Being based on HTTP, we need a real server which is up & running and capable of managing the requests from the agents. As such, the MCP SDK for C# includes a few helpers that you can use to turn an ASP.NET Web application into an MCP server.

Let’s start by creating a new ASP.NET Web application. You can do this using the command line or Visual Studio. In both cases, we can use the ASP.NET Core Empty template, since we won’t need any of the default features that come with the web or API template. If you’re using the command line, you can run the following command:

1

dotnet new web -n MCP.SSE.Server

As the first step, we must also add to this project the MCP SDK for C# and the Azure Table SDK, which are available as a NuGet packages. You can do this using the command line or Visual Studio. If you’re using the command line, you can run the following commands:

1
2


dotnet add package ModelContextProtocol --prerelease
dotnet add package Azure.Data.Tables

Now make sure to copy all the classes we created in the previous post to manage the MCP tools and the vacation requests into this new project:

Employee.cs and EmployeeVacationEntity.cs inside the Entities folder. These classes map the employees and the vacation requests tracked in the Azure Table Storage.
EmployeeVacationService.cs and IEmployeeVacationService.cs inside the Services folder. These classes expose the methods to manage the vacation requests.
EmployeeVacationTool.cs. This class exposes the methods to manage the vacation requests as MCP tools.

Now we’re ready to implement the MCP server using the SSE transport protocol. This operation is really easy to do thanks to some helpers included in the MCP SDK for C#. Just open the Program.cs file and, before builder.Build() is called, add the following code:

1
2
3
4


builder.Services
 .AddMcpServer()
 .WithHttpTransport()
 .WithTools<EmployeeVacationTool>();

The three methods are used, in order, to:

Add the MCP server to the ASP.NET Core pipeline.
Configure the server to use the HTTP transport protocol.
Register the tools that we want to expose to the agents. In this case, we are exposing the EmployeeVacationTool class, which contains the methods to manage the vacation requests.

Then, before calling the app.Run() method, add the following code to register the MCP endpoints:

1
2
3


app.MapMcp();

app.Run();

That’s it! Now, we’re still missing some code, which is the same one we wrote in the previous post to register the EmployeeVacationService class in the dependency injection container and to initialize the database with some test data in case it’s empty.This is how the final Program.cs file should look like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


using MCP.SSE.Server.Services;
using Microsoft.Extensions.Options;
using ModelContextProtocol.Protocol.Messages;
using ModelContextProtocol.Protocol.Transport;
using ModelContextProtocol.Server;
using ModelContextProtocol.Utils.Json;

var builder = WebApplication.CreateBuilder(args);

builder.Services
 .AddSingleton<IEmployeeVacationService>(sp =>
 {
 var connectionString = builder.Configuration.GetValue<string>("ConnectionStrings:tableServiceConnectionName");
 var tableName = "VacationsTable";
 return new EmployeeVacationService(connectionString, tableName, sp.GetRequiredService<ILogger<EmployeeVacationService>>());
 })
 .AddMcpServer()
 .WithHttpTransport()
 .WithTools<EmployeeVacationTool>();


var app = builder.Build();

// Seed fake employees if table is empty
using (var scope = app.Services.CreateScope())
{
 var vacationService = scope.ServiceProvider.GetRequiredService<IEmployeeVacationService>() as EmployeeVacationService;
 if (vacationService != null && await vacationService.IsTableEmptyAsync())
 {
 await vacationService.SeedFakeEmployeesAsync();
 }
}

app.MapMcp();

app.Run();

Before testing the server, we need to add the connection string for Azure Table to the appsettings.json file. You can do this by adding the following snippet:

1
2
3
4
5


{
 "ConnectionStrings": {
 "tableServiceConnectionName": "UseDevelopmentStorage=true"
 }
}

This is the connection string to use Azurite, the local emulator for Azure Storage.

Now we’re ready to run the server. Before doing that, make sure to run the Azurite Table Service emulator, like we did in the previous post:

You can click on the [Azurite Table Service] label in the Visual Studio Code application bar.
You can open the command palette (Ctrl+Shift+P) and run the Azurite: Start Table Service command.

Now press F5 to launch the debugging experience. If everything goes well, you should see the server starting up and a new browser opening up on the URL http://localhost:5248/. You will just see a message like the following one:

1
2


event: endpoint
data: message?sessionId=-JFYOh9dO5GvqMxhhTXfMg

This means that the MCP server is up and running. In case the client connects using the HTTP Streaming protocol, all the GET and POST operations will be managed by the same endpoint, which is the root of the server. To support SSE, instead, the server must expose two different endpoints:

/sse, which supports a GET request to establish the connection
/message, which supports a POST request to send messages to the server

You can see that the MCP SDK for C# provides backward compatibility with the SSE protocol by invoking the URL http://localhost:5248/sse in your browser. You should see the same exact response as before, meaning that the SSE transport protocol is working as expected.

Using the MCP server with Copilot Studio

To expose the MCP server to Copilot Studio, we need first to create a dev tunnel between our local machine and the cloud. Copilot Studio runs in the cloud and cannot directly connect to localhost. Thanks to a dev tunnel, we can test our MCP server implementation without deploying it to the cloud. In Visual Studio Code, you can create a dev tunnel by clicking on the Ports tab in the bottom panel:

Click on the Forward a Port button and enter 5248 as port number. Visual Studio Code will create a public URL that points to http://localhost:5248. Before using it, however, we need to make the URL public, since by default Visual Studio Code creates private tunnels. Right click on the URL, choose Port Visibility and select Public. You will see a message asking to confirm, click Continue and notice how the Visibility column should be set to Public now.

Now that we have a public URL, we are ready to move to Copilot Studio. As the first step, open the Copilot Studio portal. Copilot Studio enables MCP support through custom connectors. Power Platform has a catalog of 1000+ connectors, which enables makers to build agents that can connects to a wide range of platforms and services from Microsoft and 3rd parties, like Outlook, Excel, SharePoint, ServiceNow, SAP, etc. In case you need to connect to a service that is not available in the catalog (like an internal LOB application), you can create a custom connector. This is exactly what we need to do in this case, since Copilot Studio offers MCP support through a special custom connector.

Custom connectors aren’t immediately visible in the portal. The easiest way to find them is to open the Power Apps portal and, in the left panel, click on More, then press Discover all.

You will land on a page titled Discover all you can do with Power Apps. Scroll down until you see a box titled Data and click on the Custom connectors option on it.

Tip: you can click on the Pin icon on the left to pin the custom connectors option to the left panel, so that you can easily access it in the future.

In this page, you will see a list of all the custom connectors you have created (or an empty page, in case you have none). Click on the + New custom connector button at the top right and select Create from blank. Give it a meaningful name, like MCP Vacation Requests and click on Continue. You will be redirected to the wizard to create the custom connector. However, in our case we’re going to ignore it, since we’re going to leverage the Swagger editor to prove an OpenAPI definition of the connector. As such, click on the Swagger editor switch in the top bar.

OpenAPI is a standard format (based in JSON or YAML) that you can use to describe an API. For our own purpose, we can use a standard definition, since the SSE transport protocol doesn’t have a dependency on the tools we expose. As we have learned in the beginning of the post, an MCP server which uses the SSE transport protocol always exposes two endpoints: /sse and /message. It will be up to client to call them in the proper way to get the list of tools and invoke them when needed. As such, you can just copy and paste the following OpenAPI definition in the Swagger editor:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49


swagger: '2.0'
info:
 title: MCP Vacation Requests
 description: MCP Test Specification, showing the MVP spec to work in Copilot Studio
 version: 1.0.0
host: <your-dev-tunnel>
basePath: /
schemes:
 - https
definitions:
 QueryResponse:
 type: object
 properties:
 jsonrpc:
 type: string
 id:
 type: string
 method:
 type: string
 params:
 type: object
 result:
 type: object
 error:
 type: object
paths:
 /sse:
 get:
 summary: MCP Server Actions
 parameters:
 - in: query
 name: sessionId
 type: string
 required: false
 produces:
 - application/json
 responses:
 '200':
 description: Immediate Response
 schema:
 $ref: '#/definitions/QueryResponse'
 '201':
 description: Created and will follow callback
 operationId: InvokeMCP
 tags:
 - Agentic
 - McpSse
securityDefinitions: {}
security: []

The only thing you must change is the <your-dev-tunnel> placeholder, which should be replaced with the public URL you got from Visual Studio Code. For example, if your public URL is https://qfpn28w9-5248.euw.devtunnels.ms, you should replace the placeholder with qfpn28w9-5248.euw.devtunnels.ms (make sure to remove the HTTPS prefix). The key feature of this OpenAPI definition are the Agentic and McpSse tags, which are used by Copilot Studio to identify the custom connector as an MCP server.

Now you can click on Create connector to complete the operation and return to the custom connector page. You should see the new connector in the list. Now it’s time to create our agent!

Use the MCP server with an agent

Now let’s go back to the Copilot Studio portal and create a new agent. For our scenario, we’re going to create a custom engine agent, which are agents which enables developers to customize all their building blocks and that can be published in multiple channels. At the time of writing this post, declarative agents (which are agents that are based on the Microsoft 365 Copilot stack and that can be used only through Copilot Chat) don’t support MCP yet.

Click on the Create button in the left panel and select New agent. By default, this will trigger the experience to create a new agent through an assisted wizard based on natural language. For our demo purposes, let’s just click on Skip to configure at the top right of the page.

Now let’s provide the following information:

Name: Vacation Tracker Agent
Description: An agent that can hel p employees to track their vacation requests
Instructions: You are an agent which is specialized in helping employees to track their vacation requests. You can help them to request new vacation days, check the status of their requests and get the number of vacation days left.

Now hit Create at the top right of the page and wait until the agent is created. Once the operation is completed, first we need to turn on Orchestration, which enables Copilot Studio to use Generative AI to automatically determine the best course of action based on the prompt of the user. This way, if we ask the agent a question about vacation requests, it will be able to determine automatically that it needs to call the MCP server. However, now we need to actually empower the agent to use the MCP server. We do this by going to the Actions section and clicking on the Add action button.

To easily find the MCP server from the list, click on the three dots and choose Custom connector. This way, the gallery will be filtered to display only the custom connectors. You should see the MCP Vacation Requests connector we have previously created.

Click on it. As the first step, you will be asked to setup a connection. You won’t have to do anything, just wait until a green check shows up next to the MCP Vacation Requests connector and click Next.

The next step is to setup the action, by providing a description, inputs and outputs. Wait a minute, isn’t the goal of MCP to move this type of configuration from the client to the server? You’re exactly right! We have already described the purpose of our tools and the input parameters when we implemented the MCP server, so we won’t have to do anything here. We can leave the default descriptions, even if at a glance they don’t make much sense since they’re very generic.

Just click on Add action and wait for the operation to complete. That’s it! All the plumbing between our agent and the features offered by our vacation tracking service will be managed by the MCP protocol. However, before testing the agent, we need to make a change in the MCP server configuration to support a Copilot Studio requirement.

Customizing the MCP server for Copilot Studio

As we have seen when we launched the MCP server, by default the main endpoint returns the following message:

1
2


event: endpoint
data: message?sessionId=-JFYOh9dO5GvqMxhhTXfMg

However, Copilot Studio requires that the data property doesn’t contain just the relative path to the endpoint, but also the full URL. As such, we need to customize the MCP server implementation we built to accommodate this requirement. Unfortunately, the MCP SDK for C# doesn’t provide a way to do this out of the box, so we need to manually implement a custom MCP endpoint. Copy and paste the following code in the Program.cs file, after app.Run():

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65


static void MapAbsoluteEndpointUriMcp(IEndpointRouteBuilder endpoints)
{
 var loggerFactory = endpoints.ServiceProvider.GetRequiredService<ILoggerFactory>();
 var options = endpoints.ServiceProvider.GetRequiredService<IOptions<McpServerOptions>>().Value;
 var routeGroup = endpoints.MapGroup("");
 SseResponseStreamTransport? session = null;

 routeGroup.MapGet("/sse", async context =>
 {
 context.Response.Headers.ContentType = "text/event-stream";

 var host = $"https://qfpn28w9-5248.euw.devtunnels.ms";
 var transport = new SseResponseStreamTransport(context.Response.Body, $"{host}/message");
 session = transport;
 try
 {
 await using (transport)
 {
 var transportTask = transport.RunAsync(context.RequestAborted);
 await using var server = McpServerFactory.Create(transport, options, loggerFactory, endpoints.ServiceProvider);

 try
 {
 await server.RunAsync(context.RequestAborted);
 }
 catch (OperationCanceledException) when (context.RequestAborted.IsCancellationRequested)
 {
 // Normal SSE disconnect.
 }
 catch (Exception ex)
 {
 // Handle other exceptions as needed.
 Log.Error(ex, "Error in SSE transport: {Message}", ex.Message);
 }

 await transportTask;
 }
 }
 catch (Exception ex)
 {

 }
 });

 routeGroup.MapPost("/message", async context =>
 {
 if (session is null)
 {
 await Results.BadRequest("Session not started.").ExecuteAsync(context);
 return;
 }

 var message = await context.Request.ReadFromJsonAsync<IJsonRpcMessage>(
 McpJsonUtilities.DefaultOptions, context.RequestAborted);
 if (message is null)
 {
 await Results.BadRequest("No message in request body.").ExecuteAsync(context);
 return;
 }

 await session.OnMessageReceivedAsync(message, context.RequestAborted);
 context.Response.StatusCode = StatusCodes.Status202Accepted;
 await context.Response.WriteAsync("Accepted");
 });
}

This code is a custom implementation of the MCP server that uses the SSE transport protocol. The key part is the host variable, which contains the public URL of the server. You must replace the URL in the sample with the dev tunnel that you have previously created in Visual Studio Code.

The final step is to remove the app.MapMcp() line we added before and replace it with the following one:

1

MapAbsoluteEndpointUriMcp(app);

This is required because we’re providing our own implementation of the MCP server, so if we would keep both we would get an error because we are trying to register two different implementations of the same endpoints.

If you now launch the server again and open the URL http://localhost:5248/sse, you should see a message like the following one:

1
2


event: endpoint
data: https://qfpn28w9-5248.euw.devtunnels.ms/message

This means that the server is now exposing the full URL of the endpoint, which is what Copilot Studio requires.

Now we can go back to the Copilot Studio portal and test our agent.

Testing the agent

Copilot Studio provides a built-in chat interface that you can use to test your agent. You will find it in a panel on the right side of the page. Type the following prompt:

1

Give me a list of all the employees and their vacation days left

The first time you execute a prompt like this, Copilot Studio will ask permissions to run this action, since it’s the first time we’re using the custom connector.

Click on Connect and you will be redirected to a page like this:

Click on Connect under the Status column and, in the pop-up that shows up, click on Submit. Wait until the operation is completed and the status becomes Connected with a green check. Now you can go back to the chat interface and click Retry.

If everything goes well, you should see the Activity map showing the agent calling the custom connector. In the test agent panel, instead, you should see a similar response to the one that in the previous post we have observed in GitHub Copilot: a list of employees and their vacation days left.

Well done!

Wrapping up

In this post, we have learned how to implement an MCP server using the SSE transport protocol and how to use it with Copilot Studio. The feature is still in preview and we have seen that the experience is still not perfect:

We had to implement a custom MCP server to support Copilot Studio’s requirement for the full URL of endpoints, which typically isn’t required by other MCP clients.
We have to wrap the MCP server in a custom connector, which is a bit cumbersome and not very user friendly.

Despite that, it works really well and it’s really effective in connecting our custom engine agents built with Copilot Studio to tools exposed by the MCP server.

If you want to experiment with the sample, you can find the code in the SSE folder of the GitHub repository. Be aware that in the folder you will see other projects. The reason is that, to simplify the deployment of the solution, I’ve wrapped it in an Aspire project, which tremendously help with logging, tracing and management. We will cover more about the Aspire implementation in the next posts, however for the moment you can just ignore it and simply launch the MCP.SSE.Server project directly.

Happy coding!

Using Model Context Protocol in agents - Introduction

Wed, 23 Apr 2025 00:00:00 +0000

Note: The original version of this post covered only stdio and SSE as transport protocols from MCP. Enrico Sabbadin made me aware on LinkedIn that, in the latest MCP specs, SSE has been marked as deprecated and replaced with Streamable HTTP. However, since Streamable HTTP supports backward compatibility with SSE, the MCP SDK for C# offers both transport protocols with a single implementation. The post has been updated to reflect this change. Thanks Enrico!

When you’re infusing AI in your solutions and applications, one of the the most common requirements is to connect it with your data and services. Generative AI is great at creating new text, but it doesn’t have the capability to do a research on stock market fluctuations of the last 2 weeks; or to create a ticket in your organizational’s ticketing platform. This is why, shortly after the advent of LLMs, the industry has introduced concepts like Retrieval Augment Generation (to expand the knowledge of the LLM) and tools, which are functions that the LLM can call to interact with other services and platform, like calling an API or updating a record in a database. In this blog, we have seen for example how you can use an orchestrator like Semantic Kernel to augment your AI applications with plugins, which enables the LLM to run local code functions. Or, if you’re building agents with Copilot Studio, you can use Power Platform connectors

If you’re following the AI space, however, I’m pretty sure you’ve heard a lot of buzz around the Model Context Protocol (MCP, in short), introduced by Anthropic as a new way to enable agents and AI applications to interact with various platforms and services through tools. At this point, you might be wondering: “What’s new here? We already have ways to add tools and knowledge to an LLM, why should I care about MCP?”.

The key difference is that MCP is a server-side protocol, which means that the burden of implementing a plugin shifts from the developer of the AI application to the owner of the service. This is very similar to building a traditional application that needs to connect to other systems via an API: the developer of the application doesn’t need to worry about how the API is implemented, but just needs to know how to call it and what data it returns. This is exactly what MCP does for LLMs: it abstracts the complexity of calling tools, exposing them in a way which is simple to be used by the developer and optimized for the LLM consumption, unlike a traditional REST API, which is built for general purpose scenarios.

Another difference is that MCP is a standardized protocol, based on JSON-RPC 2.0. This means that, as a developer, you don’t really need to know how the tool is implemented, which are the input parameters, what’s the output you get. Once the MCP server is up & running, a client can automatically see and use all of the available tools.

Many companies are already starting to integrate MCP as a way to expose their services to LLM. For example, GitHub offers a MCP implementation which allows to feed an LLM with information about repositories, issues, pull requests, etc. Or the Azure team recently released an MCP implementation that you can use to integrate your LLM with capabilities like managing Azure Storage accounts, Azure Cosmos DB, etc.

Also on the client side, many companies are starting to integrate MCP support in their AI products. For example, Microsoft recently released in GA a new capability for GitHub Copilot in Visual Studio Code called Agent mode, which transforms Copilot from a passive code suggestion tool into an active, autonomous coding assistant. It can plan, execute, and iterate on complex development tasks with minimal user input. As part of this release, Microsoft has also introduced MCP support, enabling the agent in GitHub Copilot to integrate tools exposed via MCP. This empowers your agent, for example, to query information from your Azure subscription, or to analyze issues in your GitHub repositories.

In this series of posts, we’re going to explore how to use MCP to enrich our agents. We’ll build a simple MCP server in .NET that exposes a set of tools, and then we’ll see how to use it in a custom agent, both with a pro code (with Semantic Kernel) and a low code (with Copilot Studio) approach.

Creating our first MCP server - The main building blocks

Before creating our first server, we first need to understand a bit more how MCP works. The communication between the client and the server is handled by a transport layer, which can be implemented in multiple ways. In our series we’ll explore the two most common approaches:

Standard Input / Output (stdio), which means that MCP will use the standard input and output streams offered by the operating system to communicate. In this scenario, both the server and the client are running locally. This approach is used by many vendors, who publish their MCP server implementation as a package (for NuGet, Node or Python) or as a Docker image, which is then consumed by the client application. This is the implementation we’re going to take a look at in this post. When you use this approach, in fact, the server isn’t a real server that needs to be always up & running, but rather an executable that is run by the client application whenever it needs to call a tool. This is the approach we’ll explore in this post.
Streamable HTTP, which is based on HTTP and GET / POST requests to exchange messages between the client and the server. In this scenario, the server is running on a remote machine and the client is running locally. This approach was originally implemented using Server-Sent Events (SSE), which is now deprecated but it’s backward compatible with Streamable HTTP. This means that, from a tooling perspective, you can build MCP server that can support both HTTP Streaming and SSE. This is important because Copilot Studio uses SSE to consume the tools exposed by the MCP server. This approach follows a more traditional client-server model, where the server must always be running to listen for incoming client connections. This is the approach we’ll explore in the next posts.

Let’s start to create our server with the stdio approach. As mentioned, we don’t need a real server, an executable is just fine. For our own purposes, we’re going to create a simple .NET console application. In our scenario, we’re going to simulate a service that the organization can use to track leave requests from employees. This service will enable employees to:

Find out how many leave days they have left
Request a leave day
Get the list of all the employees and their leave days left

The service implementation

Let’s start to go over the main service of our project. We won’t spend too much time in this section, since it’s mainly boilerplate code needed to handle vacation day requests. For our sample, I’m going to use Azure Table (part of Azure Storage) to store the leave requests. The main reason of this choice is that Visual Studio Code can be easily integrated with Azurite, which is a local Azure Storage emulator. This way, you’ll be able to run the code locally without needing any Azure subscription, and with a much more minimal setup compared to what you would need to use a proper database like SQL Server.

The starting point is to install the Azurite emulator in Visual Studio Code. You can do this by searching for Azurite in the extensions marketplace or by opening the following page in the browser and clicking on Install. After the extension is installed, you should see the following options in the lower command bar of Visual Studio Code:

By clicking on one of these options, you’re going to start the local emulator for the various components of Azure Storage: Table, Queue and Blob. For the moment, we can move forward, since we won’t need the emulator right away. Now we can start writing our code, by creating a new console application based on .NET 9.0. You can do this in Visual Studio, Visual Studio Code or by running the following command in a terminal:

1

dotnet new console -n MCP.Stdio.Server

Now we need some code to store the information about employees and their leave requests. First, we need to add the Azure.Data.Tables NuGet package to our project. If you’re using Visual Studio, you can right click on the project in the Solution Explorer and select Manage NuGet Packages. In the Browse tab, search for Azure.Data.Tables and install it. If you’re using Visual Studio Code, you can run the following command in a terminal:

1

dotnet add package Azure.Data.Tables

Now let’s add a folder to our project by right clicking on it and choosing New folder. Let’s call it Entities. Then, inside this new folder, add a new class called EmployeeVacationEntity.cs. This class will be used to store the information about the leave requests. Copy and paste the following code in the file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


using Azure;
using Azure.Data.Tables;

namespace MCP.Stdio.Server.Entities
{
 public class EmployeeVacationEntity : ITableEntity
 {
 public string PartitionKey { get; set; } = "Employee";
 public string RowKey { get; set; } = string.Empty; // Employee name
 public int VacationDaysLeft { get; set; }
 public DateTimeOffset? Timestamp { get; set; }
 public ETag ETag { get; set; }
 }
}

We only need to store two information: the name of the employee (we’re going to reuse the RowKey property for this) and the number of vacation days left. All the other properties you see (like PartitionKey and RowKey) are required by the Azure Table SDK, and they are used to identify the entity in the table.

For the purpose of our MCP implementation, however, we need also a simpler entity to track employees and their vacation days. When we need to return the data to the caller, in fact, we don’t need to return things like PartitionKey or ETag, which are part of the internal Azure Table SDK implementation and might just confuse the LLM. As such, let’s create a new class in the new folder called Employee.cs and copy and paste the following code in the file:

1
2
3
4
5


public class Employee
{
 public string EmployeeName { get; set; } = string.Empty; // Employee name
 public int VacationDaysLeft { get; set; }
}

Now we can start to implement the actual service which will provide the methods to interact with the Azure Table service and perform the operations we need for our scenario. Create a new folder called Services and add a new class called EmployeeVacationService.cs. Here is the full code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104


using Azure;
using Azure.Data.Tables;
using MCP.Stdio.Server.Entities;
using Microsoft.Extensions.Logging;

namespace MCP.Stdio.Server.Services
{

 public class EmployeeVacationService : IEmployeeVacationService
 {
 private readonly TableClient _tableClient;

 private readonly ILogger<EmployeeVacationService> _logger;

 public EmployeeVacationService(string storageConnectionString, string tableName, ILogger<EmployeeVacationService> logger)
 {
 _tableClient = new TableClient(storageConnectionString, tableName);
 _tableClient.CreateIfNotExists();
 _logger = logger;
 }

 public async Task<int?> GetVacationDaysLeftAsync(string employeeName)
 {
 try
 {
 var entity = await _tableClient.GetEntityAsync<EmployeeVacationEntity>("Employee", employeeName);
 _logger.LogInformation($"Fetched vacation days left for {employeeName}: {entity.Value.VacationDaysLeft}");
 return entity.Value.VacationDaysLeft;
 }
 catch (RequestFailedException ex) when (ex.Status == 404)
 {
 return null;
 }
 }

 public async Task<bool> ChargeVacationDaysAsync(string employeeName, int daysToCharge)
 {
 try
 {
 var response = await _tableClient.GetEntityAsync<EmployeeVacationEntity>("Employee", employeeName);
 var entity = response.Value;
 if (entity.VacationDaysLeft < daysToCharge)
 return false;
 entity.VacationDaysLeft -= daysToCharge;
 await _tableClient.UpdateEntityAsync(entity, entity.ETag);
 _logger.LogInformation($"Charged {daysToCharge} vacation days to {employeeName}. Remaining days: {entity.VacationDaysLeft}");
 return true;
 }
 catch (RequestFailedException ex) when (ex.Status == 404)
 {
 return false;
 }
 }

 public async Task<bool> IsTableEmptyAsync()
 {
 await foreach (var _ in _tableClient.QueryAsync<EmployeeVacationEntity>(maxPerPage: 1))
 {
 _logger.LogInformation("Table is not empty.");
 return false;
 }
 _logger.LogInformation("Table is empty.");
 return true;
 }

 public async Task SeedFakeEmployeesAsync()
 {
 var random = new Random();
 var names = new List<string>
 {
 "Alice Johnson", "Bob Smith", "Charlie Lee", "Diana Evans", "Ethan Brown",
 "Fiona Clark", "George Miller", "Hannah Davis", "Ian Wilson", "Julia Adams"
 };
 var tasks = new List<Task>();
 foreach (var name in names)
 {
 var employee = new EmployeeVacationEntity
 {
 RowKey = name,
 VacationDaysLeft = random.Next(5, 31) // Random vacation days between 5 and 30
 };
 tasks.Add(_tableClient.UpsertEntityAsync(employee));
 }
 await Task.WhenAll(tasks);
 }

 public async Task<List<Employee>> GetAllEmployeesAsync()
 {
 _logger.LogInformation("Fetching all employees from the table storage.");
 var result = new List<Employee>();
 await foreach (var entity in _tableClient.QueryAsync<EmployeeVacationEntity>())
 {
 result.Add(new Employee
 {
 EmployeeName = entity.RowKey,
 VacationDaysLeft = entity.VacationDaysLeft
 });
 }

 _logger.LogInformation($"Fetched {result.Count} employees from the table storage.");
 return result;
 }
 }
}

As mentioned, I won’t cover in details everything that this class does, since it’s mostly boilerplate code to read and write data from the Azure Table service. The class uses the Azure Table SDK to perform the following operations against the database:

GetVacationDaysLeftAsync(): retrieves the number of vacation days left for a specific employee.
ChargeVacationDaysAsync(): charges a number of vacation days to a specific employee.
IsTableEmptyAsync(): checks if the table is empty.
SeedFakeEmployeesAsync(): creates a set of fake employees with random vacation days left. This method is used at the project startup to populate the table with some data that we can use for testing.
GetAllEmployeesAsync(): retrieves all the employees in the table.

Now we need to initialize the console application, by setting up the dependency injection system and by registering the service we have just created in the Program.cs file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


var builder = Host.CreateEmptyApplicationBuilder(settings: null);

builder.Configuration.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true);

builder.Services
 .AddSingleton<IEmployeeVacationService>(sp =>
 {
 var connectionString = builder.Configuration.GetValue<string>("ConnectionStrings:tableServiceConnectionName");
 var tableName = "VacationsTable";
 return new EmployeeVacationService(connectionString, tableName);
 });


var app = builder.Build();

using (var scope = app.Services.CreateScope())
{
 var vacationService = scope.ServiceProvider.GetRequiredService<IEmployeeVacationService>() as EmployeeVacationService;
 if (vacationService != null && await vacationService.IsTableEmptyAsync())
 {
 await vacationService.SeedFakeEmployeesAsync();
 }
}

await app.RunAsync();

By default, unlike an ASP.NET app, a console app doesn’t have any initialization code to manage dependencies, logging, configuration, etc. As such, we must manually create a builder using the Host class, which is the main entry point for a .NET application, by calling Host.CreateEmptyApplicationBuilder(). Once we have a builder, we must load the configuration from the appsettings.json file, which is where we will store the connection string to the Azure Table service. Then we can register the EmployeeVacationService in the dependency injection container as a singleton, since we want to use a shared instance across the application. We won’t let the dependency injection container create a new instance of the service automatically, but we’re going to supply a specific one created by ourselves, so that we can pass the two parameters required by the constructor: the connection string (loaded from the configuration file) and the table name (which is hardcoded in this case, it’s called VacationsTable).

Now that we’ve finished configuring the application, we can create it by calling builder.Build(). Before executing it, however, we need to seed some test data we could use to test our application. We do this with the following block of code:

1
2
3
4
5
6
7
8


using (var scope = app.Services.CreateScope())
{
 var vacationService = scope.ServiceProvider.GetRequiredService<IEmployeeVacationService>() as EmployeeVacationService;
 if (vacationService != null && await vacationService.IsTableEmptyAsync())
 {
 await vacationService.SeedFakeEmployeesAsync();
 }
}

This code creates a new scope, which is a way to create a new instance of the service with its own lifetime. This is useful when we want to create a new instance of the service that is not shared with the rest of the application. Inside the scope, we retrieve the instance of the EmployeeVacationService we have previously registered and we use the IsTableEmptyAsync() method to check if the table is empty. This way, we avoid populating the table with test data every time the application starts, but only the first time we execute it. If that’s the case, we call another method, SeedFakeEmployeesAsync(), to populate the table with some fake data.

Finally, we can run the application by calling await app.RunAsync(). This will start the console application and wait for incoming requests.

The last step is to add the connection string for Azure Table to the appsettings.json file. Create a new file in the root of the project called appsettings.json and copy and paste the following content in it:

1
2
3
4
5


{
 "ConnectionStrings": {
 "tableServiceConnectionName": "UseDevelopmentStorage=true;"
 }
}

The connection string is set to UseDevelopmentStorage=true;, which means that the application will use the Azurite emulator running locally. This is the default connection string for Azurite, so you don’t need to change anything. If you want to publish this project in the cloud and use a real Azure Storage account, you can replace this value with the proper connection string.

So far, all the code we have seen is only about setting up the proper infrastructure we need to manager our “Leave requests” scenario. Now it’s time to start adding the specific MCP code to expose the service we have just created.

Setting up the MCP server

To set up the MCP server, first we need to add the MCP SDK for C# to our project. The package is published as pre-release, which means that:

If you’re using Visual Studio, you need to check the Include pre-release checkbox in the NuGet package manager and then search for the package called ModelContextProtocol.
If you’re using Visual Studio Code, you can run the following command in a terminal:
1

dotnet add package ModelContextProtocol --prerelease

Now we need to create a class to expose all the methods of our EmployeeVacationService as MCP tools. Create a new class in the project, call it EmployeeVacationTool.cs and copy and paste the following code in it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


using System.ComponentModel;
using MCP.Stdio.Server.Entities;
using MCP.Stdio.Server.Services;
using ModelContextProtocol.Server;

[McpServerToolType]
public class EmployeeVacationTool
{
 private readonly IEmployeeVacationService _employeeVacationService;
 public EmployeeVacationTool(IEmployeeVacationService employeeVacationService)
 {
 _employeeVacationService = employeeVacationService;
 }

 [McpServerTool, Description("Get the vacation days left for a given employee.")]
 public async Task<int?> GetVacationDaysLeftAsync([Description("The name of the employee")] string employeeName)
 {
 return await _employeeVacationService.GetVacationDaysLeftAsync(employeeName);
 }

 [McpServerTool, Description("Charge vacation days for a given employee.")]
 public async Task<bool> ChargeVacationDaysAsync([Description("The name of the employee")] string employeeName, [Description("The number of days to charge")] int daysToCharge)
 {
 return await _employeeVacationService.ChargeVacationDaysAsync(employeeName, daysToCharge);
 }

 [McpServerTool, Description("Get the list of employees with their number of vacation days left")]
 public async Task<List<Employee>> GetAllEmployeesAsync()
 {
 return await _employeeVacationService.GetAllEmployeesAsync();
 }
}

In the constructor, we’re injecting the EmployeeVacationService instance we created before, so that the .NET dependency injection system will take care of creating the instance for us. The first important step is to decorate the class with the [McpServerToolType] attribute. This tells the MCP SDK that this class contains tools that we want to expose to the client. MCP supports other type of attributes: for example, a MCP server could expose prompts (using the [McpServerPromptType] attribute).

Then, we just need to write a bunch of methods that are actually wrappers around the ones we have already created in the EmployeeVacationService class. They will get, as input, the same parameters we have in the original methods, and they will return the same type of data. The only difference is that we need to add the [McpServerTool] attribute to each method we want to expose. This tells the SDK that this method is a tool that can be called by the client. Additionally, very importantly, for each method and parameter we must add a description using the [Description] attribute. This is critical and what makes MCP really tailored for LLM usage: thanks to these descriptions, we can help the LLM to understand what the tool does, how to use it and if it’s the right choice for the task at hand. Without using MCP, the responsibility of understanding how to use a tool is on the developer. For example, if you’re using Semantic Kernel as orchestrator, you need to provide a description of the function you’re calling and the parameters as part of the plugin code.

Now that we have defined the tools, we need to initialize the MCP server. With the help of C# SDK, the task is really easy. We just need to add a few initialization lines in the Program.cs file. We must add the following block before the app is built:

1
2
3
4
5
6


builder.Services
 .AddMcpServer()
 .WithStdioServerTransport()
 .WithToolsFromAssembly();

var app = builder.Build();

We’re calling, in order:

AddMcpServer(): this method adds the MCP server to the dependency injection container.
WithStdioServerTransport(): this method configures the server to use the standard input/output transport. This is the transport we’ll use in this post.
WithToolsFromAssembly(): this method scans the assembly for any classes decorated with the [McpServerToolType] attribute and registers them as tools in the MCP server. This is how the server knows which tools to expose to the client.

If we want to be more specific (for example, you have added some testing tools that you don’t want to expose), we can replace WithToolsFromAssembly() with WithTools<T>(), where T is the type of the class that contains the tools we want to expose. In our case, we can use WithTools<EmployeeVacationTool>() to expose only the EmployeeVacationTool class:

1
2
3
4
5
6


builder.Services
 .AddMcpServer()
 .WithStdioServerTransport()
 .WithTools<EmployeeVacationTool>();

var app = builder.Build();

That’s it. Now the MCP server is ready to be used. As we mentioned in the beginning of the post, the server is not a real server that needs to be always up & running, but rather an executable that is run by the client application whenever it needs to call a tool. This means that we don’t need to run the server in a separate process or listen for incoming requests. We just need to call it from the client.

Testing the MCP server

Now we’re ready to test the MCP server we’ve build and, guess what, we don’t really need to write a client application! At the beginning of the post, in fact, I mentioned many applications have been updated to add MCP support, including GitHub Copilot and Visual Studio Code. As such, all you need is to open Visual Studio Code and make sure that Agent mode is enabled for GitHub Copilot:

You can click the following link to automatically open Visual Studio Code on the right setting and turn it on.

Before setting up the MCP server, however, we need to start the Azurite Table Service emulator, otherwise our MCP server will fail to start because it won’t be able to connect to the Azure Table service. To do this, you can:

Click on the [Azurite Table Service] option in the application bar of Visual Studio Code.
Press Ctrl+Shift+P to open the command palette and look for the command Azurite: Start Table Service.

Now that everything is up and running, the easiest option to configure it is to add a new folder to your project called .vscode and, inside it, create a new file called mcp.json. Inside it, copy and paste the following content:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


{
 "servers": {
 "MyFirstMCP": {
 "type": "stdio",
 "command": "dotnet",
 "args": [
 "run",
 "--project",
 "${workspaceFolder}\\Stdio\\MCP.Stdio\\MCP.Stdio.Server.csproj"
 ]
 }
 }
}

Make sure to replace the last parameter of the args array with the path of your project. This file tells to GitHub Copilot where to find the MCP server and how to run it. As you can see, we don’t even have to build the tool and point the configuration to the executable, but we can just run the project directly using the dotnet CLI. Behind the scenes, Visual Studio Code is going to run the project with the command:

1

dotnet run --project ${workspaceFolder}\\Stdio\\MCP.Stdio\\MCP.Stdio.Server.csproj

The key parameter that makes this possible is type, which we set to stdio to use standard input/output as transport. Once you’ve added this file, Visual Studio Code should recognize it as the configuration for an MCP server and should give you the option to start it. If everything goes well, you should be able to see the number of tools exposed by the server:

Now open GitHub Copilot, by clicking on its icon at the top, near the search bar. In the textarea where you can write the prompt, click on the dropdown menu which is set to Ask. You should see an option called Agent. Make sure to select it.

When you’re in agent mode, you should see in the toolbar a toolbox icon, with the number of tools that are exposed by the MCP server. If you click on it, you should see that GitHub Copilot has properly recognized the three tools we have exposed in the MCP server:

Now let’s write a prompt that will require the usage of a tool to be solved. For example, we can ask:

1

Give me the list of employees and their vacation days left.

GitHub Copilot will ask the permission to call one of the available tools, in this case the one called GetAllEmployees. If you expand the panel, you can see also the input parameters that will be submitted to the tool (in this case, it will be empty since it doesn’t require any input).

If you hit Continue, GitHub Copilot will run the tool, get the results back and process it to give you a response:

If you want to test the ability of the LLM to extract input parameters from the prompt, you can try to use a prompt like this:

1

Please charge 2 vacation days to Bob Smith

GitHub Copilot will ask your permission again to call the MCP server, in this case the tool called ChargeVacationDays. This time, if you expand the panel, you should see that the LLM has extracted the input parameters from the prompt and it’s passing them to the tool:

Wrapping up

In this post, we have seen how to create a simple MCP server that exposes a set of tools. Thanks to our MCP server, we can enhance the LLM to interact with a fictitious platform to manage vacation days request. In this blog post, we have tested the MCP server with GitHub Copilot, which was an interesting experiment, but not really practical. In the next posts, we’ll see how we can use the same tools in different custom agents, built with pro code and low code approaches.

You can find the final code of the project in the GitHub repository. Specifically, the implementation based on standard input/output we have built in this post is in the Stdio folder. The code is basically the same we have built in this post, you will notice a few differences since in the final version I’ve also added a bit of logging.

Happy coding!

Start a rap battle between agents with AutoGen - Using AutoGen in a declarative agent for Microsoft 365 Copilot

Thu, 06 Mar 2025 00:00:00 +0000

This is a guest post, written by Bharath Bath and Anubhav Ranjan, who work in my team at Microsoft and they are specialized in pushing the boundaries of Microsoft 365 copilot agents to support customer scenarios.

In the previous posts, we built a multi-agent solution/workflow using AutoGen where we had a MC agent kicking off a rap battle by providing a topic to 2 rapper agents that created lyrics in the styles as per their Persona. We also had a Judge agent who rated whose lyrics was better. This scenario helped understand how multiple agents can be configured to run independently on a particular task. This can be very well mapped to any real-world scenarios where we may need multiple agents to work in a collaborative fashion and help arrive at a solution.

Introducing Copilot extensibility

Let’s pause here for a moment and talk about Microsoft 365 Copilot. As you may already be aware, it is possible to extend Copilot to connect it to 3rd party systems and extract information or perform actions on behalf of the user. Wouldn’t it be great if we could get Copilot to also connect with these multi-agent frameworks to get our job done?
Jumping back into the rap battle scenario, let’s assume we want to initiate the battle via Microsoft 365 Copilot and get AutoGen agents to work in the background and get us the best judged lyrics. To achieve this, we will need to do the following:

In AutoGen
1. Download the Rap Battle workflow
2. Use AutoGen Studio to serve the workflow, so it is available to be called as an API.
Create a Declarative Agent using Teams Toolkit (TTK) via Visual Studio Code that can invoke this call.
1. Configure the declarative agent action and the request/response message formats expected by Autogen
2. Provision the Declarative agent using TTK, so it’s available with Teams on the right pane inside Copilot, ready to be triggered.
3. Forward the port used by AutoGen studio and make it publicly available via Teams toolkit.

Let’s dive into the details of each of these steps.

The AutoGen workflow

Let’s first run the AutoGen Studio and go to the AutoGen Studio browser tab and download the Rap Battle workflow by clicking on the download icon. This is the same step that we did in the code post to execute our team using code. A file named team-config.json will be downloaded. Let’s tove it to the folder from where we ran the AutoGen Studio. We will need this file later to leverage one of the AutoGen Studio features that we haven’t covered yet: the ability to expose a team through a REST API.

Configuring & provisioning the Declarative Agent

To perform the following steps, you will need Visual Studio Code with the Teams Toolkit installed and access to an environment with a Microsoft 365 Copilot license.

When you open the Teams Toolkit within Visual Studio Code, ensure that you have logged in to the environment with Copilot access as highlighted below. I have also highlighted 2 other options Create New App that will help create a new Declarative Agent and Provision that will help provision and test the Declarative agent within Copilot. So, let’s start with creating the Declarative Agent.

Clicking on Create New App will show the various options available. We will select the Declarative Agent option.

As we will be making a call to the AutoGen workflow, we will select Add an Action:

Next, we will select Start with an OpenAPI Description Document. For this exercise, we have already created a OpenAPI message specification document that we will use, that you can download from here.

Now, let’s browse and select the OpenAPI message specification file:

Once the file is parsed, we will be presented with the API endpoints that are described in the file. Select the GET call and press Ok.

Then, select the location for the project’s root folder:

Give the application a name of your choice and enter to confirm.

A window with the Declarative Agent project loaded should appear. I have highlighted the purpose of the 4 main files in the image below for reference:

declarativeAgent.json contains the main definition of the agent. It points to the instruction.txt file for instructions and ai-plugin.json for details on the actions that the Declarative Agent can perform.
ai-plugin.json contains details on the functions that the agent can trigger. In our case, we have just 1 function called getRapBattleResult. If you look at line 20, in the pic below, you will see that it refers to the openapi.yaml file that contains the actual OpenAPi specification of the API calls, which we previously imported during the wizard. The run_for_functions section at the bottom of the file indicates the list of functions out of all the functions available in the OpenAPI specification file that the declarative agent will have access to. We have only provided getRapBattleResult as that is the only call we want to make.
openapi.yaml contains the full OpenAPI specification of the API we have imported. There are three key points to highlight in the file:
- For every API endpoint, we have the full structure of the request and the response.
- The operationId parameter, which contains the same function name getRapBattleResult as we used it in the api-plugin.json file.
- The url parameter, which is the base endpoint of the server that the declarative agent will use to make the API calls. We’re going to set it later, once we have created a Dev Tunnel that we can use to expose our local server to the internet, so that Copilot can reach it.

Setting up port forwarding

Now that we have configured the declarative agent we need to provision it so it can be available within Copilot. But before we do that we have one last task to complete. The AutoGen Studio server will run locally and point to port 8084. We will need to make this port publicly available, since Copilot can’t reach our local server. Let’s go to Ports view in the Panel region of Visual Studio Code and select Forward a Port. If you haven’t logged in with GitHub before, you’ll be prompted to sign in. Enter 8084 as the port we want to be forwarded.

Visual Studio Code will generate a public URL for you, as you can see from the following image:

Now we have a forwarded address but the visibility of this address is still private. Lets change it to public, by right clicking on it and choose Port Visibility -> Public.

We will now need to copy the forwarded address URL (the one in the 2nd column) in the openapi.yaml file, as this will be the base url to make the API call:

Testing our scenario

Now, we are done with all changes and its time to test the solution. So, we need to first to start the AutoGen Studio server. Let’s open the Anaconda PowerShell Prompt and use conda to activate the virtual environment we have created in the first post. Then, we can execute AutoGen Studio, this time with a different command however since we don’t want to launch the UI, but host the team as an API:

1

autogenstudio serve --team path/to/team-config.json --port 8084

Make sure to provide the correct path to the team-config.json file that we downloaded earlier.

Now we can go back to Visual Studio code to provision the declarative agent. Select the Teams Toolkit icon from the left pane and select Provision under the LIFECYCLE section.

If the operation completes successfully, you should see message at the bottom right saying 5/5 actions in provision stage executed successfully.

Now that we have AutoGen Studio running in server mode with the port forwarded and publicly available and the Declarative agent provisioned into Copilot in Teams, its time to test it! Open Microsoft 365 Copilot in one of the many ways you can access it (the official website, the Windows app or inside Teams), and you should see the Declarative Agent available in the right pane. Select it and provide the prompt to start the rap battle, for example:

1

Start a rap battle on marshmallows

As this is the first time we are running the agent, once we prompt it to run the Rap Battle, it will ask for our consent. Let’s select Always allow.

And you should start seeing the results of the Rap Battle showing up in the Copilot response 🙂

If you want to see what’s going on behind the scenes, you can:

Move to the terminal window. You will see that the agent as called the API endpoint exposed by AutoGen Studio:
You can turn on developer mode in Copilot by typing the following prompt:
1

-developer on
Submit again a new prompt to start another rap battle, this time you will the logs of API calls right after the response

Wrapping up

In this post, we have seen how we can invoke a multi-agent architecture built with AutoGen from a Declarative Agent within Copilot. The rap battle was just an example to showcase the approach we can follow, which can be further explored with more relevant enterprise scenarios in which want to handoff tasks to a multi-agent team to be handled and executed separately.

You can find the declarative agent that we have built in this post on GitHub.

Happy agenting! 🤖🎤

Adding a Custom Engine Agent to Microsoft 365 Copilot Chat

Tue, 04 Mar 2025 00:00:00 +0000

If you’re following the Microsoft 365 and Copilot ecosystem, you have probably heard that we have agents now. 😊 Joking aside, we distinguish between two types of agents in the Microsoft 365 ecosystem:

Declarative agents, which are agents that are built on top of the Microsoft 365 Copilot platform and that users can leverage from the Copilot Chat experience. Makers and developers can customize a declarative agent by providing a custom set of instructions, define one or more knowledge sources (like a website or a SharePoint site) and add a set of actions, like sending a mail or retrieving data from a 3rd party data source. You can learn more here. Being built on top of Microsoft 365 Copilot, declarative agents use the same LLMs, orchestration engine and UI.
Custom Engine Agents, which are agents that can offer a completely customized experience. You can use your own LLMs and orchestration engine, and you can publish them across multiple channels: Microsoft 365, Teams, a custom web application, a mobile application, etc.

In the Microsoft ecosystem, both type of agents can be built with two approaches:

Makers can use Copilot Studio, a low-code tool that allows them to create agents using a simplified experience, based on a visual designer and the wide range of Power Platform connectors.
Pro developers can use the Teams Toolkit, an extension for Visual Studio and Visual Studio Code that accelerate the development of Teams and Copilot apps based on .NET, TypeScript or Python.

Until not long ago, declarative agents and custom engine agents were two disconnected experiences. Declarative agents shows up in the right sidebar of the Copilot Chat experience: users can chat with them by clicking on their name (to use the immersive experience) or they can mention them in the standard Microsoft 365 Copilot chat to leverage the in-context experience.

Custom engine agents, instead, could be integrated into the Microsoft 365 ecosystem only as Teams applications: users can see them as personal apps (so they show up in the left sidebar of Teams) or in the chat history.

Many customers, however, expressed the interest of using the Copilot Chat as a single entry point for all the agentic experiences in the organization. The distinction between a declarative agent and a custom engine agent is a technical implementation detail that should not be visible to the end user. For this reason, Microsoft is rolling out the capability to use Custom Engine Agents in Microsoft 365 Copilot Chat. In this blog post, we’re going to explore how you can do that both with Copilot Studio and the Teams Toolkit.

The timeline for the rollout is the following one: Public Preview: From early February 2025 and expect to complete in mid-March 2025. General Availability (Worldwide): From late March 2025 and expect to complete by early April 2025. As such, if you still don’t see it available, don’t worry. It’s a gradual rollout, so it might just not be available in your tenant yet.

Adding a custom engine agent to Copilot Chat using Copilot Studio

When you create or edit a custom engine agent in Copilot Studio, one of the most important tabs is the one called Channels, which you can use to make the agent available across multiple platforms: Teams, Slack, Facebook, a custom website, etc. If your tenant already received the new capabilities, you will see that the Teams channels has been rebranded as Teams + Microsoft 365:

When you click on it, you will have the usual option to add it as a channel, but you will see also a new checkbox labeled Make agent available in Microsoft 365 Copilot.

As the note in the panel says, if you have an existing custom engine agent you have created in the past and which is already available in Teams, you will need to republish it after you turn on the option to make it available in Microsoft 365 Copilot. You can do it by clicking on the Publish button in the main page of your agent:

To make sure that the new version with the Microsoft 365 Copilot integration shows up in your organizational catalog, I strongly suggest you to make any change in one of the agent’s properties, like the description or developer’s name. This way, you’ll increase the version number, making sure that the new version will be submitted for approval. You can do this by clicking on Edit details in the Agent preview section of the Teams + Microsoft 365 channel.

Now you can go back to the Teams + Microsoft 365 channel and, under Availability options, click on Show to everyone in my org to start the process of submitting the agent to the application catalog of your organization:

From this option, you can click on Submit for admin approval to start the submission process. Once completed, it’s now the turn of the Microsoft 365 tenant administrator to approve the agent from the Integrated Apps section of the admin portal. You will see the agent under the Requested apps section, and you can notice immediately that there’s an important difference compared to the custom engine agents you might have previously created and published:

The one at the top (Feedback Processing Agent) is a custom engine agent that was created before the new capability was rolled out. As you can see, the only supported host is Teams. The one at the bottom (Travel Insight Agent) was created, instead, enabling the new option to make the agent available in Copilot. As you can see, in fact, other than Teams, the agent is also available in Copilot as a host.

Once the agent has been approved by the tenant administrator, it will show up in the Microsoft 365 and Teams catalog, under the Built for your org section:

You can notice here as well that there’s a way to immediately recognize agents that can be integrated into Copilot Chat: agents like Travel Insights Agent and List creator agent support the new option, so there’s a little sparks icon after their name. Feedback Processing Agent, instead, hasn’t been updated to support the new capability, so the icon is missing.

Once you click on Add to install the agent for your user, you will see that it will show up in the Copilot Chat experience, alongside any other declarative agent you might have:

Adding a custom engine agent to Copilot Chat using the Teams Toolkit

When it comes to agents created with a pro-code approach, the integration between the agent and Microsoft Teams as a channel is managed by the Azure Bot Service. The Azure Bot Service is a service that allows you to create, manage and publish agents that, similarly to Copilot Studio, can be published to multiple channels, like Teams, Slack, Facebook, etc.

However, since we aren’t anymore in a low code scenario, the Azure Bot Service doesn’t provide any feature to build your agent, but it acts only as a middle man between your agent and the channels. You’ll need to provide, in fact, the URL of the messages endpoint of the agent you have built using the Bot Framework:

For this reason, the Azure Bot Service doesn’t require any new feature to support publishing a custom engine agent in Copilot Chat. It’s enough to keep using the Teams channel.

The change must be done at the Teams Toolkit level, specifically inside the manifest of a Teams application. The starting point is an agent created with one of the available pro-code options: it could be the Teams Toolkit for Visual Studio Code, if you want to use TypeScript or Python; it could be the Teams Toolkit for Visual Studio, if you want to use .NET; it could be the new Microsoft 365 Agents SDK, which is a new iteration of the Bot Framework SDK.

Regardless of the technology you choose, your agent will always be encapsuled into a Teams application, which has a manifest file that describes its capabilities.

Let’s take the Teams Toolkit for Visual Studio Code as an example, which we can use to create a custom engine agent powered by the Teams AI library, which facilitates integrating Generative AI capabilities into your agent. In the Teams Toolkit tab (the Teams logo in the Visual Studio Code sidebar), click on Create a New App and choose Custom Engine Agent.

For the sake of this post, let’s go with the simplest scenario, which is Basic AI Chatbot, then pick up the language you prefer between JavaScript, TypeScript and Python. The next step is to choose with AI provider you want to use:

In case you choose OpenAI, you will need to provide the API key and the name of the model you want to use.
In case you choose Azure OpenAI, you will need to provide the API key, the URL endpoint and the deployment name.

At the end of the process, the Teams Toolkit will create a base project with everything you need to support the basic experience, which is an agent powered by a Large Language Model. What’s interesting for us, for the purpose of this blog post, is the appPackage folder, which contains all the files that compose a Teams application: the manifest and two icons. Let’s focus on the manifest.json file. At the time of writing this post, the manifest will use the latest version of the Teams schema, which is 1.19. You wil find this information in the manifestVersion property of the JSON:

1
2
3
4


"$schema": "https://developer.microsoft.com/en-us/json-schemas/teams/v1.19/MicrosoftTeams.schema.json",
"manifestVersion": "1.19",
"version": "1.0.0",
"id": "${{TEAMS_APP_ID}}",

In order to use the new Copilot Chat integration capability, we must switch to the developer preview of Teams apps, by doing two changes:

Change the $schema property to https://developer.microsoft.com/en-us/json-schemas/teams/vDevPreview/MicrosoftTeams.schema.json
Change the manifestVersion to devPreview.

This is how the first lines of the manifest should look like:

1
2
3
4


"$schema": "https://developer.microsoft.com/en-us/json-schemas/teams/vDevPreview/MicrosoftTeams.schema.json",
"manifestVersion": "devPreview",
"version": "1.0.0",
"id": "${{TEAMS_APP_ID}}",

The next step is to add a new section called copilotAgents at the root level of the manifest. If you have ever built a declarative agent with the Teams Toolkit, this section will be familiar to you, since it’s where you can define that your application includes a declarative agent. With the devPreview version of the manifest, we can add custom engine agents as well, as in the following example:

1
2
3
4
5
6
7
8


"copilotAgents": {
 "customEngineAgents": [
 {
 "type": "bot",
 "id": "${{BOT_ID}}"
 }
 ]
 },

The customEngineAgents property is a collection, since your application can host more than one custom engine agent. For each of them, you must specify two properties:

The type property, which can be only bot as of today.
The id property, which is the ID of the agent you have created in the Azure Bot Service.

If you use the ${{BOT_ID}} placeholder, the Teams Toolkit will take care of replacing this value with the actual bot id that has been registered on the Azure Bot Service. One of the most useful features of the Teams Toolkit, in fact, is the capability of automatically provisioning all the resources that are needed to run your agent:

It’s going to deploy the Azure Bot Service on your Azure subscription.
It’s going to register an application in Microsoft Entra to manage the authentication.
It’s going to create a Dev Tunnel so that you can test your agent locally.

That’s it. Now, if you want, you can test your agent in your tenant. First, you must make sure that, in the Teams Toolkit panel, you are signed in both with your Azure subscription and your Microsoft 365 account in the Accounts section. Then, under the Debug and Run section in Visual Studio Code, just click on Debug in Teams (Edge). The Teams Toolkit will start to deploy all the required resources and, at the end of the process, a new instance of Edge will open up directly on Teams, with the pop-up to install the agent, followed by the option to open it right into Copilot:

You can notice how, like we have seen with Copilot Studio, the agent will have the Copilot logo near its name. And if you decide to publish it in the organizational catalog, also in this case you will see the Host application isn’t just teams, but Copilot as well:

And now, exactly like we have seen with Copilot Studio, you can install the agent in your tenant and start to use it right in the Copilot Chat experience:

Voilà!

Wrapping up

In this post, we have explored a new exciting capability that is being rolled out to the Microsoft 365 Copilot ecosystem: the ability to use Custom Engine Agents in Microsoft 365 Copilot Chat. This feature greatly improves the user experience for employees in your organization, since now they can use agents from a single entry point, regardless of the technology and the approach used to build them. This is an important milestone towards the Microsoft vision of Copilot as the “UI for AI”.

Happy coding!

Start a rap battle between agents with AutoGen - Using code

Mon, 03 Mar 2025 00:00:00 +0000

Let’s continue our journey through our rap battle between agents! In the first post, we have introduced AutoGen Studio, a low code tool that we can use to build multi-agent scenarios using a visual interface and JSON; in the second post, we have enhanced the rap battle by adding the ability to use tools and to support a more complex team orchestration; in this post, we’re going to switch from AutoGen Studio to AutoGen code and see how we can run our rap battle from an application.

Let’s start!

Setting up the environment

To run the code I’m going to use a Jupyter notebook, which is a convenient way to run Python code interactively without having to write a full application. Jupyter notebooks can run pretty much everywhere, even in the browser. In my case, I’m going to use the built-in Jupyter notebook support in Visual Studio Code. Open Visual Studio Code and choose File -> New file. Visual Studio Code will ask you for the type of file you want to create, one of the options will be Jupyter Notebook. Choose it.

A Jupyter notebook is made up by different blocks, which can contain code or Markdown. When a block contains code, you will see on the left a Run button, which you can use to run the code in the block.

The first step is to select the Python kernel we want to use to run our code. You can choose it by clicking on Select Kernel at the top right corner of the notebook:

You will see a series of options, choose Python environments. You should be able to see not just the default Python installation you might have in your system, but also the virtual environment we created in the first post with Conda. Choose this one, since it already contains all the dependencies we need to support AutoGen, because we have already installed AutoGen Studio in the same environment.

Now we’re ready to run some code! It shouldn’t be necessary, but just to be safe and ensure we have everything we need, let’s run the following code in the first block to install the AutoGen dependencies using the Python package manager:

1

%pip install -U "autogen-agentchat" "autogen-ext[openai,azure]"

Now we’re ready to actually use AutoGen!

Running the rap battle reusing the work we did in AutoGen Studio

As the first step, create a new code block by clicking on the + Code button at the top of the notebook. We’re going to use a new block, since we don’t want to repeat the installation of the dependencies every time we run the notebook. The first approach to running the rap battle in code is to reuse the work we did in AutoGen Studio. In fact, the JSON that defines a team configuration can be directly loaded in code.

First, we need to go back to AutoGen Studio and export the team configuration we created in the first post (Make sure to read that post if you haven’t gone through the AutoGen Studio configuration).

From the Start menu or from the Windows Terminal, launch the Anaconda PowerShell Prompt.
Activate the environment we created in the first post by running the following command:
1

conda activate autogen
Execute the following command to launch AutoGen Studio:
1

autogenstudio ui

Once the server has started, open the browser on the default AutoGen Studio URL, which is http://127.0.0.1:8081 and look for the team we created in the Team Builder section.

In the top right of the team, you will see a download button. Click on it, AutoGen Studio will start the download of the JSON file with the team configuration.

Make sure to copy the JSON file you’ve downloaded in the same folder where you saved the Jupyter notebook. Now go back to Visual Studio Code and let’s add the following code in the second code block of our notebook:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


from autogenstudio.teammanager import TeamManager

# Initialize the TeamManager
manager = TeamManager()

# Run a task with a specific team configuration
result = await manager.run(
task="Start a rap battle on the importance of eating fruit",
team_config="team-config.json"
)
print(result)

The code is really simple. We import a class called TeamManager from AutoGen and then we create a new instance of it. The class offers a method called run(), which requires two parameters:

task: the prompt we want to send to the team. In this case, we’re providing the topic for the rap battle.
team_config: the path to the JSON file we downloaded from AutoGen Studio.

Now click on the Run button near the code block. The rap battle will start and, in the end, you will get an output similar to the one you got in AutoGen Studio, made by 4 messages:

The Rap MC introduces the topic, calls the Python function to get the list of words and shares them with the rappers.
The two rappers, Eminem and Drake, generate lyrics based on the topic and including the words.
The judge determines the winner and terminates the conversation.

However, in this case, since we don’t have a UI, the output won’t be as pretty as in AutoGen Studio. If you want to make it a little bit better, you can replace the code with the following one:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


from autogenstudio.teammanager import TeamManager

# Initialize the TeamManager
manager = TeamManager()

# Run a task with a specific team configuration
result = await manager.run(
task="Start a rap battle on the importance of eating fruit",
team_config="team-config.json"
)

# Extract the response output of every agent
agent_responses = {}

for message in result.task_result.messages:
 if message.source not in agent_responses:
 agent_responses[message.source] = []
 agent_responses[message.source].append(message.content)

# Print the responses of each agent
for agent, responses in agent_responses.items():
 print(f"Responses from {agent}:")
 for response in responses:
 print(response)
 print("\n---\n")

This code will extract each message from the result.task_result property and print the responses of each agent separately, making them easier to read.

But what if you don’t want to use AutoGen Studio and start directly in code? Let’s explore how we can use the raw AutoGen library.

Running the rap battle using the AutoGen library

Let’s create a new code block, so that we can run the two versions of the rap battle separately. In this case, we’re going to use the AutoGen library directly, without using AutoGen Studio.

First, let’s import the classes we need:

1
2
3
4
5


from autogen_agentchat.agents import AssistantAgent
from autogen_agentchat.teams import SelectorGroupChat
from autogen_agentchat.ui import Console
from autogen_ext.models.openai import OpenAIChatCompletionClient
from autogen_agentchat.conditions import ExternalTermination, TextMentionTermination

Then, we need to define the LLM we want to use. In my case, I’m going to use the OpenAI APIs, so I create an OpenAIChatCompletionClient instance:

1
2
3
4


model_client = OpenAIChatCompletionClient(
 model="gpt-4o-mini",
 api_key="<your-api-key>",
)

If you prefer to use Azure OpenAI, instead, you must use the AzureOpenAIChatCompletionClient class, which requires a couple of extra parameters:

1
2
3
4
5
6


az_model_client = AzureOpenAIChatCompletionClient(
 azure_deployment="{your-azure-deployment}",
 model="{model-name, such as gpt-4o}",
 api_version="2024-06-01",
 azure_endpoint="https://{your-custom-endpoint}.openai.azure.com/",
 api_key="sk-...", # For key-based authentication.

If you want to use other LLMs or SLMs, the official documentation covers the usage of other providers like Gemini, Ollama or Azure AI Foundry.

Now let’s define the function we need to get the list of words, which is the exact same one we saw in the previous post:

1
2
3


def get_list_of_words() -> str:
 words = "apple, banana, cherry, date, elderberry, Christmas"
 return words

Now we can start to define the agents, which are represented by the AssistantAgent class, which is the equivalent of the AssistantAgent object we used in AutoGen Studio.

Let’s start by the Rap MC:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


mc_agent = AssistantAgent(
 name="mc_agent",
 description="An agent that acts as an MC in a rap battle",
 system_message="""You are a rap MC and your role is run a rap battle with two contestants.
 The user will provide you the topic for the battle.
 Your main task is to be the master of ceremony and introduce the rap battle.
 You must share some words of encouragement and describe the topic for the battle that was given by the user.
 YOU MUST USE THE TOPIC GIVEN BY THE USER, you can't make up one on your own.
 You have access to a skill that gives you a list of words for rap lyrics.
 YOU MUST USE this skill to get a list of words. First, you're going to introduce the rap battle and then YOU MUST SHARE the list of words with the contestants.
 The contestants will need to create lyrics for the given topic and they must use all the words in the list you have shared.
 You must be very enthusiastic and use a lot of exclamation marks. You must also use a lot of slang words and rap terms.
 You must also use emojis.
 """,
 model_client=model_client,
 tools=[get_list_of_words]
)

The definition of the agent is pretty easy to understand after we have worked with AutoGen Studio, since we’re setting the same properties, we’re just doing it in code: name, description and system_message, which are the instructions that the agent will follow. Then we set the model_client property with the model definition we have previously created. Finally, for the mc_agent, we also set the tools property with the collection of functions that the agent can use. In this case, it’s only one function: get_list_of_words().

Now we can move on to the other two rappers:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


eminem_agent = AssistantAgent(
 name="eminem_agent",
 description="An agent that can rap using the style of Eminem",
 system_message="""You are a rapper, and you rap in the style of Eminem.
 You are participating in a rap battle.
 You will be given a topic, and you will need to create the lyrics and rap about it.
 Be creative but remember to stay on topic.
 Do your best, since you're going to compete against another rapper.""",
 model_client=model_client
)

drake_agent = AssistantAgent(
 name="drake_agent",
 description="An agent that can rap using the style of Drake",
 system_message="""You are a rapper, and you rap in the style of Drake.
 You are participating in a rap battle.
 You will be given a topic, and you will need to create the lyrics and rap about it.
 Be creative but remember to stay on topic.
 Do your best, since you're going to compete against another rapper.""",
 model_client=model_client
)

The two agents are very similar, only the system_message property is slightly different. We use the same model_client and, in this case, we don’t define any tools, since the rappers don’t need to use any function.

Finally, we need to define the judge:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


 judge_agent = AssistantAgent(
 name="judge_agent",
 description="An agent that acts as a judge in a rap battle",
 system_message="""You are a judge in a rap battle.
 You will be given the lyrics of the rap battle, and you must give a score to each contestant.
 You must give a score between 1 and 10 to each contestant.
 You must ensure that the final score is given after all the other contestants have shared their lyrics.
 In addition to the score, you also need to provide an explanation of why you assigned that score.
 If the final score is a tie or the score difference between the two rappers is minimum (for example, Eminem scores 8 while Drake scores 9), YOU MUST ASK to run a new round, and the two rappers must generate new lyrics.
 YOU MUST NOT declare a winner in this case.
 However, you can't run more than 3 rounds.
 If, instead, the score difference is 2 or more (for example. Eminem scores 7 and Drake scores 9), you must declare a winner.
 When you have a winner and the plan is complete, you can declare the winner and then YOU MUST RESPOND with TERMINATE.""",
 model_client=model_client
 )

Again, nothing special, only the system_message property is different since it contains all the instructions that the judge must follow, which are quite long and detailed due to the more complex task it needs to fulfill.

The next step is to define the condition termination:

1

text_termination = TextMentionTermination("TERMINATE")

We’re using the TextMentionTermination class, which requires the string we want to use as a signal that the conversation is completed. In our case, it’s the word TERMINATE, which we have included in the instructions.

Finally, we can put everything together in a team:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


agent_team = SelectorGroupChat(
 participants=[mc_agent, eminem_agent, drake_agent, judge_agent],
 max_turns=15,
 termination_condition=text_termination,
 model_client=model_client,
 allow_repeated_speaker=True,
 selector_prompt="""
 You are managing a dynamic, interactive rap battle with multiple AI agents. You must select an agent to perform the task:

 {roles}

 The workflow is the following one:

 - The rapmc_agent begins the battle by introducing the event, setting the topic, and providing a list of words that the rappers must use.
 - Once the introduction is complete, eminem_agent and drake_agent will take turns to generate lyrics based on the provided topic.
 - After eminem_agent and drake_agent have performed, the judge_agent will evaluate the lyrics and give a score to each rapper. If the battle is a tie or the score difference between the two rappers is only 1 point (for example, eminem_agent scores 8 while drake_agent scores 9), the judge must request another round, until a maximum of 3.
 The new round will start directly with the two rappers, without an introduction from the Rap MC agent.

 The battle continues until the Judge Agent explicitly states "TERMINATE" in their response.

 Current conversation context:
 {history}

 Read the above conversation, then select an agent from {participants} to perform the next task.
 Make sure the planner agent has assigned tasks before other agents start working.

 Only select one agent.
 """,
)

This is the exact same configuration we did in Agent Studio, just using code:

participants is a collection of all the agents we have created so far.
max_turns is the maximum number of turns we want to allow in the conversation.
termination_condition is the condition we have created before to use to terminate the conversation.
model_client is the model we want to use.
allow_repeated_speaker is a boolean value that indicates if we want to allow the same agent to speak multiple times in a row. In this case, we set it to True, since we want the rap battle to continue in case of a tie or a close call.
selector_prompt is the set of instructions that explains, in natural language, the flow of the conversation, so that the SelectorGroupChat can orchestrate the conversation and pick the best agent for the task.

Now we have everything we need to run the rap battle! We just need to run it and add some code to display the messages generated by the agent:

1
2
3
4


user_input = "Start a rap battle on the importance of education."
# Run the team and stream messages to the console.
stream = agent_team.run_stream(task=user_input)
await Console(stream)

The user_input variable must be set with the prompt we want to send to kick off the team chat, in our case the topic of the rap battle. Then, we pass it as parameter to the run_stream() method of the agent_team object we have previously created. We are using the message streaming feature, so that we can see the messages as they are generated, instead of waiting until the whole conversation is completed.

We just need a final step. Since the whole process involves using some asynchronous operations, we need to wrap all the code we have written so far inside a method that we can call using the await keyword. Look at the following example:

1
2
3
4


async def main() -> None:
 # Code to initialize agents, teams, etc.

await main()

You can see the whole code in the GitHub repository.

Now you can just hit the Run button near the code block to start the rap battle! The output will be very similar to what we have observed in AutoGen Studio or in the previous test by loading the AutoGen Studio JSON workflow file.

Being regular Python code, you can use it in any scenario that works best for you, other than testing it in a Jupyter notebook. For example, in the GitHub repository you will find an API implemented with FastAPI, which you can call with any HTTP client to run the rap battle and get back the conversation in JSON. In the same repository, you will find the instructions to run the API locally.

Wrapping up

In this blog post, we have learned how we can turn the multi-agent scenario that we have built with AutoGen Studio into code that we can integrate into a web application, call from an API, etc. If you want to test the integration with other apps, however, there’s a simple way to do that without writing code: by using AutoGen Studio again! In fact, we can use it not just to test the team workflow, but also to expose it as an API. In the next post, I will be honored to greet two members of my team for a guest post, who will share their experience in adding multi-agent scenarios to a declarative agent for Microsoft 365 Copilot thanks to AutoGen Studio!

Remember to check all the files we saw in this post in the GitHub repository.

Happy coding!

Start a rap battle between agents with AutoGen - Adding tools

Wed, 26 Feb 2025 00:00:00 +0000

In the previous post, we started to explore the capabilities of AutoGen, a framework by Microsoft to create multi-agent solutions. In the post, we started approaching it with AutoGen Studio, a “low code” web application that you can use as a playground to validate your scenario before actually writing code. As a scenario, we used a fun one to demonstrate the potential of AutoGen: a rap battle between two agents, with an MC agent introducing the battle and a judge agent deciding who has created the best rap lyrics.

In this post, we’re going to enhance our rap battle, by using one of the capabilities of AutoGen that we haven’t used so far but that it’s very powerful: tools! Tools allow you to add extra capabilities to an agent, and they are represented as Python functions. Being code, we can do anything with them, from calling an API to querying a database or reading the content of a file. This means we can expand an agent’s skills beyond what it can do using only the LLM that powers it.

Let’s start!

Adding our first tool

For our demonstration, we’re going to tweak our rap battle scenario a bit. The Rap MC, instead of just introducing the topic, will now share a list of words to the two rappers, that they have to use in their rap lyrics. However, this list of words won’t be hardcoded in the instructions or randomly generated by the LLM, but it will be returned by a Python function that we’re going to create. For the sake of this post, the Python function will be incredibly simple:

1
2
3
4


def get_list_of_words() -> str:
 print("Get_list_of_words invoked")
 words = "apple, banana, cherry, date, elderberry, Christmas"
 return words

The function simply returns a static list of words, nothing more, nothing less. We could have retrieved the list from an API; or read it from a text file. The result would have been the same.

Let’s see how we can add this function to our project. In AutoGen Studio, open the Team Builder section and select the team you have created in the previous post. In the Component Library, under Tools, you fill find the skill that we have deleted in the original post, called calculator. Drag and drop it under the Tools section of the Rap MC agent:

Click on the pencil icon near the agent’s name and scroll down at the end of the Configuration section. First, make sure to turn on the option Reflect on Tool Use. Then, click on the FunctionTool to open the editor. We’ll need to customize a few things since, by default, the tool will contain a Python function to perform basic math operations. You can leave the standard description as it is (which is only for human reading), so let’s focus on the Configuration section:

Name: call the function get_list_of_words

Description: this is the description of the function that will be used by the LLM to understand what it does. Use the following one:

1

Get a list of words that can be used to generate the lyrics for a rap song. Participants to the rap battle must use all the words returned by this function.

Source code: this is the actual code of the function. Copy and paste the Python function I shared before:

1
2
3
4


def get_list_of_words() -> str:
 print("Get_list_of_words invoked")
 words = "apple, banana, cherry, date, elderberry, Christmas"
 return words

Then click Save Changes. The next step is to tweak the instructions of the Rap MC agent. We need to make it aware that now it must use this function before starting the rap battle, so that it can share the list of words with the rappers. Click again on the pencil icon near the agent’s name and scroll down System Message section. Replace the instructions with the following ones:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


You are a rap MC and your role is run a rap battle with two contestants. The user will provide you the topic for the battle.
Your main task is to be the master of ceremony and introduce the rap battle.
You must share some words of encouragement and describe the topic for the battle that was given by the user.
YOU MUST USE THE TOPIC GIVEN BY THE USER, you can't make up one on your own.
You have access to a skill that gives you a list of words for rap lyrics.
YOU MUST USE this skill to get a list of words.
First, you're going to introduce the rap battle and then YOU MUST SHARE the list of words with the contestants.
The contestants will need to create lyrics for the given topic and they must use all the words in the list you have shared.
You must be very enthusiastic and use a lot of exclamation marks.
You must also use a lot of slang words and rap terms.
You must also use emojis.

As you can see, we have instructed the agent that now it has access to a skill that gives it a list of words for rap lyrics and that it must use it to get the list of words before starting the rap battle. Click on Save Changes.

Testing our changes

We don’t need to make further changes for the moment. This tool will be needed only by the Rap MC agent, so we don’t need to add it to the other agents. Let’s test our changes. Click on the Playground button and create a new session, by choosing the team you have just updated. Then, start a new rap battle by providing a topic, for example:

1

Start a rap battle about the importance of eating fruit

If everything goes as planned, you will see that, this time, the Rap MC agent will call the function get_list_of_words() before starting the rap battle:

And if you read the message generated by the Rap MC agent, you will notice that, this time, other than introducing the topic, it also shares the list of words with the rappers:

Finally, if you read the lyrics generated by the two rapper agents, you will see that they have used all the words in the list.

Spicing up the competition

Let’s spice up a bit the competition. Let’s change our scenario so that the judge agent, after having scored the lyrics, will have the opportunity to run a second round in case the two scores are too close. You will soon understand why we’re doing this 😊

Let’s go back to the Team Builder section of AutoGen Studio and let’s edit the Judge agent, by clicking on the pencil icon near its name. Update the System Instructions with the following ones:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


You are a judge in a rap battle. You will be given the lyrics of the rap battle, and you must give a score to each contestant.
You must give a score between 1 and 10 to each contestant.

You must ensure that the final score is given after all the other contestants have shared their lyrics.

In addition to the score, you also need to provide an explanation of why you assigned that score.

If the final score is a tie or the score difference between the two rappers is minimum (for example, Eminem scores 8 while Drake scores 9), YOU MUST ASK to run a new round, and the two rappers must generate new lyrics.
YOU MUST NOT declare a winner in this case. However, you can't run more than 3 rounds.

If, instead, the score difference is 2 or more (for example. Eminem scores 7 and Drake scores 9), you must declare a winner.

When you have a winner and the plan is complete, you can declare the winner and then YOU MUST RESPOND with TERMINATE.

This is the part that we have changed, compared to the previous instructions:

1
2
3
4
5


If the final score is a tie or the score difference between the two rappers is minimum (for example, Eminem scores 8 while Drake scores 9), YOU MUST ASK to run a new round, and the two rappers must generate new lyrics.
YOU MUST NOT declare a winner in this case.
However, you can't run more than 3 rounds.

If, instead, the score difference is 2 or more (for example. Eminem scores 7 and Drake scores 9), you must declare a winner.

This way, the judge agent will be able to ask for a new round in case the two scores are too close. Click on Save Changes.

Now go back to the Playground section and start a new session. Start a new chat experience by sharing the topic you want to use, like:

1

Start a rap battle about the importance of eating vegetables

You will see that things will go almost as planned, but not exactly quite right. The judge agent will score the two rappers and, in case of a tie or a very close score, it will ask to run a new round. However, instead of giving the stage directly to the two rappers to generate a new round of lyrics, you will see the workflow starting again from scratch:

The Rap MC agent will call the function get_list_of_words() to get the list of words
The Rap MC agent will introduce the topic and share the list of words with the two rappers
The two rappers will generate a new round of lyrics
The judge agent will score the two rappers

This is not what we want. We want the judge agent to give the stage directly to the two rappers, without going through the Rap MC agent again. Why is this happening?

The answer lies in the type of team we are using, which is RoundRobinGroupChat. As the name says, this type of team is capable only to run a round-robin workflow, where each agent has the opportunity to speak in turn. This means that, when the judge agent asks for a new round, the workflow starts again from scratch, with the Rap MC agent speaking first.

An easy way to observe this behavior is to change the order in which we have added the agents to the team. If we add them in an order different than the current one (Rap MC, Rappers, Judge), our scenario will be broken, because the order doesn’t match the workflow we have envisioned.

The following image shows what happens when you scramble the order of the agents in the team (first judge, then one rapper, then Rap MC and then another rapper). As you can notice, in this case the conversation ended not because the battle is over, but because we have reached the maximum number of turns (10) since the judge agent wasn’t able to properly evaluate and conclude the battle.

Is there a way to solve this problem? The answer is yes!

Introducing the SelectorGroupChat

The SelectorGroupChat is another type of team supported by AutoGen, which is more powerful. Unlike the RoundRobinGroupChat, the SelectorGroupChat allows you to create a workflow where each agent can speak when it is needed, without following a strict order. This is possible thanks to three extra features:

The selector prompt: using natural language, we can provide a prompt that explains which is the expected workflow of the conversation.
LLM integration: the SelectorGroupChat is powered by a LLM on its own, which enables AutoGen to reason about the conversation and the context and decide which agent should speak next.
The ability to support calling an agent multiple times.

To use a SelectorGrupChat, we must switch to JSON mode in our Team Builder, since it isn’t supported yet by the UI. The first step is to change the provider property of the main object. When you switch to JSON mode, the first line will be:

1
2
3
4


{
 "provider": "autogen_agentchat.teams.RoundRobinGroupChat",
 ...
}

You must change it to:

1
2
3
4


{
 "provider": "autogen_agentchat.teams.SelectorGroupChat",
 ...
}

This is enough since, once you do that, the UI will be updated to support the extra properties which are required by the SelectorGroupChat. Go back to visual mode and click on the Edit icon. You will see that we have some new properties we can customize:

Let’s start with the** Selector Prompt**, which is the set of instructions that describes the workflow we want to implement. Copy and paste the following text:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


You are managing a dynamic, interactive rap battle with multiple AI agents.
You must select an agent to perform the task:

{roles}

The workflow is the following one:

- The rapmc_agent begins the battle by introducing th event, setting the topic, and providing a list of words that the rappers must use.
- Once the introduction is complete, eminem_agent and drake_agent will take turns to generate lyrics based on the provided topic.
- After eminem_agent and drake_agent have performed, the judge_agent will evaluate the lyrics and give a score to each rapper.
If the battle is a tie or the score difference between the two rappers is only 1 point (for example, eminem_agent scores 8 while drage_agent scores 9), the judge must request another round, until a maximum of 3.
The new round will start directly with the two rappers, without an introduction from the Rap MC agent.

The battle continues until the Judge Agent explicitly states "TERMINATE" in their response.

Current conversation context:
{history}

Read the above conversation, then select an agent from {participants} to perform the next task.
Make sure the planner agent has assigned tasks before other agents start working.

Only select one agent.

This prompt is quite detailed and explain the various possibilities that can happen during the rap battle. We are also explicitly calling the opportunity that an agent can be called multiple times. You can see that we have also a series of placeholder, like {roles} or {history}. These are required by AutoGen, so that the team can retain the entire context to take the best possible decision.

The next step is to turn on the option Allow repeated speaker, which enables the team to call multiple times the same agent if needed. Finally, we need to set up the LLM which will be used by the SelectorGroupChat to reason about the conversation. The process is exactly the same we did for agents in the previous post:

If you want to use OpenAI, you can just provide your API key in the API Key field.
If you want to use Azure OpenAI or another LLM, you will need to switch to JSON mode and add the missing properties.

Another option you may want to adjust is Max Turns in the team configuration.. It’s currently set to 10 but, since we have tuned the instructions so that the Judge agent can run up to 3 rounds, we may want to increase this value to 15 or 20 to make sure the battle doesn’t terminate earlier.

Now we’re ready to test our changes!

Running our improved rap battle

Go back to the Playground section and start a new session, like we did so far by sharing the topic you want to use, like:

1

Start a rap battle about the importance of eating vegetables

You should see that the battle is now much more dynamic and, in case of a tie or a very close score, the Judge agent will ask for a new round without going through the Rap MC agent again, like in the following screenshot:

The following workflow created by AutoGen Studio shows the entire conversation, which clearly shows how we aren’t using anymore a round robin approach but, right after the Judge declares that the battle is not over yet, the two rappers are called again to generate new lyrics:

Please note that, when you use the SelectorGroupChat, it becomes more important than ever that you provide a proper description of the agent in the Description property that you can find in the Configuration section of the agent. This is because the SelectorGroupChat will use this description to understand what the agent is capable of doing and, if the description is not clear, it may select the wrong agent to perform a task.

Wrapping up

In this post, we have explored two new features offered by AutoGen:

How we can empower agents with tools, which expand the capabilities of an agent beyond what it can do with the LLM alone and enabling scenarios like calling an API, querying a database or reading the content of a file.
How we can use the SelectorGroupChat to create a more dynamic workflow, where agents can be called multiple times and the order in which they speak is not fixed.

In the next post, we’ll move from playground to production and we’ll see how we can use AutoGen in our code, both by exporting the workflow we have created with AutoGen Studio and by writing code from scratch using the AutoGen APIs.

Happy coding!

Start a rap battle between agents with AutoGen - AutoGen Studio

Mon, 24 Feb 2025 00:00:00 +0000

Last year, we dived into the world of multi-agents scenarios and we learned how this approach can help to solve complex problems, by splitting them into smaller tasks that can be solved by different agents. At that time, we chose Semantic Kernel as a framework to build our agents, continuing on the explorations we did on how Semantic Kernel can help to infuse AI into your applications.

However, Semantic Kernel isn’t the only framework created by Microsoft to support AI workloads. A while ago, a Microsoft Research team started to work on AutoGen which, unlike Semantic Kernel (which is a general puprose AI orchestration framework that offers agents support as well), it’s a framework specifically designed for building multi-agent systems. AutoGen recently went through a major refactoring, which led to the stable 0.4 release.

In this series of posts, we’ll explore AutoGen and its capabilities with a scenario that isn’t very enterprise-friendly, but it shows the potentials of a multi-agent approach in an effective way. We’re going to run… drum roll… a rap battle between agents! 🎤

Our solution will consist of four agents:

An MC agent, that will introduce the rap battle.
Two rapper agents, who will generate lyrics for the rap battle starting from a given topic.
A judge agent, who will evaluate the lyrics and declare a winner.

In this post, we’ll start to experiment with this scenario using AutoGen Studio, a web application that you can use as a playground to test your agent configuration before actually coding it.

Let’s start!

Setting up AutoGen Studio

AutoGen Studio is a web application based on Python, so the easiest way to install it is through PyPi, the Python package manager. However, in my experience, I had a few challenges when I tried to do that using my default Python installation, due to some conflicts with existing dependencies. As such, the best way to avoid them is to create a new Python virtual environment, which is a self-contained directory that contains a Python installation for a particular version of Python, plus several additional packages.

The easiest way to manage virtual environments is to leverage Conda, a package system and environment manager for Python. To install it, you have two options:

If you want to install the most comprehensive version, you can get Anaconda from the official website. However, this version comes with a lot of packages and features that you might not need.
If you prefer to install a more lightweight version, you can get Miniconda from the official website. This version is a minimal installer for conda and only includes conda, Python, the packages they depend on, and a small number of other useful packages.

Regardless of your choice, at the end of the installation, you will find a couple of new Anaconda-specific terminals in the Start menu and Windows Terminal.

Run the Anaconda PowerShell Prompt and type the following command:

1
2


conda create -n autogen python=3.10
conda activate autogen

Now you have a dedicated Python environment you can use to isolate all the dependencies required by AutoGen Studio. You can now proceed to the actual installation, by typing the following command:

1

pip install autogenstudio

Once the installation is complete, you can start AutoGen Studio by typing the following command:

1

autogenstudio ui

You will see, at the end of the initialization process, a message with the URL to open, which is http://127.0.0.1:8081. Open it inside your favorite browser to finally get access to AutoGen Studio:

Let’s explore the various building blocks!

Getting started with AutoGen Studio

AutoGen Studio is made by four sections:

Team Builder is where you’re going to spend most of time, since it provides the components to create a multi-agent scenario.
Playground is where you’re actually going to test your multi-agent scenario, by starting a chat with your agents.
Gallery is where you can find a few pre-built scenarios that you can use as a starting point. What’s important for us is that we have a Default Component Gallery, which contains all the main building blocks we’ll need to use to build our scenario.
Deploy is where you will find instruction on how you can turn the scenario you build into a real application, by using the AutoGen APIs or Docker.

Let’s take a deeper look at the Default Component Gallery, since it contains the main building blocks that AutoGen offers for building multi-agent scenarios, whether you build them with AutoGen Studio or in code.".

There are five main building blocks:

Agents are the core component of a multi-agent scenario. They are the ones that will actually do the work, by executing tasks and following the set of instructions we have provided. Typically, in a multi-agent architecture, each agent has different skills and capabilities and they are specialized in one task. For example, in our rap battle, every agent will play a different role: one will introduce the battle, two will generate lyrics following a specific rap style, one will judge the lyrics and declare a winner.
Models are the AI engines used by agents to perform their tasks. We can use OpenAI, Azure OpenAI or open-source models like llama or DeepSeek R1. Typically, models are associated with an agent, but we’ll see how they can be used to empower a team in some specific scenarios.
Tools are custom Python functions that an agent can use to perform a task. They are equivalent to Semantic Kernel plugins and, being built with Python code, you can use them to extend the capabilities of an LLM and interact with APIs, databases, read files, etc.
Terminations are the conditions that you can use to define when the multi-agent conversation is completed. The most common type of terminations are based on the number of messages (e.g. terminate the conversation after 10 messages) or a specific keyword mentioned in the conversation (e.g. terminate the conversation when the word “terminate” is mentioned).
Teams are the glue that puts everything together. A team orchestrates the work of agents and it’s composed by the agents themselves, optionally a model and the termination conditions.

Now that we have a better understanding of the building blocks we’re going to use, let’s start to build our rap battle!

Creating the multi-agent scenario

The starting point in AutoGen Studio to create our scenario is the Team Builder section. Click on New Team. This will create a new default configuration with:

A team object, which is the main orchestrator of the scenario.
One default agent, called weather_agent, with a default model (gpt-4o-mini).

AutoGen Studio offers a visual drag & drop interface. For example, if we want to add another agent to our team, we can pick one of the available agents in the Component Library and drag it inside the team.

Behind the scenes, the configuration will be stored in JSON format, which we can see by turning off the Visual builder mode option:

The JSON shows that the team is the main object of the scenario. Inside it, we have a config property which has a sub collection called participants, which stores the configuration of all the agents that participates to the scenario. From the provider property, we can also understand a bit more the approach that this team is going to use. By default, in fact, the provider property will be set to autogen_agentchat.teams.RoundRobinGroupChat, which refers to a round-robin approach, where each agent will take turns to respond to the conversation. We will explore more the JSON as we’re going to build our rap battle.

For the moment, let’s keep the team, but let’s delete the default weather_agent, by clicking on the trash can icon. As you can see from the Component Library on the left, we have different agents available. Let’s pick the AssistantAgent one, which represents a default agent powered by the OpenAI Chat Completion APIs, and drag it innside the Agents section of the team. This will be our MC agent, which is going to introduce the rap battle. Click on the pencil icon to start editing it using the visual interface:

Let’s start to customize the properties in the Component Details section:

Label: this is the “verbose” name of the agent. Let’s call it Rap MC Agent.
Description: this is the description of the agent. It doesn’t need to follow a specific format, but it should be a short sentence that describes the role of the agent. Let’s set it to An agent that acts as the master of ceremonies in a rap battle.

Now let’s move to the Component details one:

Name: this property is more important, because it’s the actual name of the agent. It can be used also within a conversation or in the instructions to refer to it. Let’s set it to rapmc_agent.
Description: this is, again, a description of the agent, which is used by the LLM to understand the role of the agent in the conversation. We can safely reuse the one we have used in the Component Details section, which is An agent that acts as a master of ceremony in a rap battle.
System Message: this is a key property, since it’s the set of instructions that our agent will follow. Here we must describe, in a detailed way and in natural language, how the agent must behave. Let’s set the one for our MC agent:

1
2
3
4
5
6
7
8


You are a rap MC and your role is to run a rap battle with two contestants.
The user will provide you with the topic for the battle.
Your main task is to be the master of ceremony and introduce the rap battle.
You must share some words of encouragement and describe the topic for the battle that was given by the user.
YOU MUST USE THE TOPIC GIVEN BY THE USER, you can't make up one on your own.
You must be very enthusiastic and use a lot of exclamation marks.
You must also use a lot of slang words and rap terms.
You must also use emojis.

Let’s take a look now at the final configuration section:

For the moment, let’s keep the option Reflect on tool use off, since our agent won’t actually use any tool, just instructions. However, we must set up the model in order for our agent to work. The agent already comes with a default configuration, called Model Client. Click on the edit icon near it to get access to the configuration to the OpenAIChatCompletionClient object, which wraps the Chat Completion APIs provided by OpenAI. By default, the configuration is set to use the gpt-4o-mini model and the OpenAI API. If you’re comfortable with this setup, the only field you need to modify is the API Key. Paste the API Key provided by OpenAI, which you can obtain from https://platform.openai.com/settings/organization/api-keys after you have logged in with your OpenAI account.

If you want to use Azure OpenAI or an entirely different model, you will need to directly manipulate the JSON file that describes the team, since the UI doesn’t support this scenario as of today. Let’s take a look at how we can switch to Azure OpenAI, for example. After you have saved your changes, switch to JSON mode and, inside the config -> participants property, you will find the configuration of the rapmc_agent.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


{
 "provider": "autogen_agentchat.agents.AssistantAgent",
 "component_type": "agent",
 "version": 1,
 "component_version": 1,
 "description": "An agent that acts as a master of ceremony in a rap battle",
 "label": "Rap MC Agent",
 "config": {
 "name": "assistant_agent",
 "model_client": {
 "provider": "autogen_ext.models.openai.OpenAIChatCompletionClient",
 "component_type": "model",
 "version": 1,
 "component_version": 1,
 "description": "Chat completion client for OpenAI hosted models.",
 "label": "OpenAIChatCompletionClient",
 "config": {
 "model": "gpt-4o-mini",
 "api_key": "<your-openai-api-key>"
 }
 },
 "tools": [
 {
 "provider": "autogen_core.tools.FunctionTool",
 "component_type": "tool",
 "version": 1,
 "component_version": 1,
 "description": "Create custom tools by wrapping standard Python functions.",
 "label": "FunctionTool",
 "config": {
 "source_code": "def calculator(a: float, b: float, operator: str) -> str:\n try:\n if operator == '+':\n return str(a + b)\n elif operator == '-':\n return str(a - b)\n elif operator == '*':\n return str(a * b)\n elif operator == '/':\n if b == 0:\n return 'Error: Division by zero'\n return str(a / b)\n else:\n return 'Error: Invalid operator. Please use +, -, *, or /'\n except Exception as e:\n return f'Error: {str(e)}'\n",
 "name": "calculator",
 "description": "A simple calculator that performs basic arithmetic operations",
 "global_imports": [],
 "has_cancellation_support": false
 }
 }],
 "handoffs": [],
 "model_context": {
 "provider": "autogen_core.model_context.UnboundedChatCompletionContext",
 "component_type": "chat_completion_context",
 "version": 1,
 "component_version": 1,
 "description": "An unbounded chat completion context that keeps a view of the all the messages.",
 "label": "UnboundedChatCompletionContext",
 "config": {}
 },
 "description": "An agent that acts as a master of ceremony in a rap battle",
 "system_message": "You are a rap MC and your role is run a rap battle with two contestants. The user will provide you the topic for the battle. Your main task is to be the master of ceremony and introduce the rap battle. You must share some words of encouragement and describe the topic for the battle that was given by the user. YOU MUST USE THE TOPIC GIVEN BY THE USER, you can't make up one on your own. \nYou must be very enthusiastic and use a lot of exclamation marks. You must also use a lot of slang words and rap terms. You must also use emojis.",
 "model_client_stream": false,
 "reflect_on_tool_use": false,
 "tool_call_summary_format": "{result}",
 "label": "Rap MC Agent"
 }
}

Let’s focus on the model_client property:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


"model_client": {
 "provider": "autogen_ext.models.openai.OpenAIChatCompletionClient",
 "component_type": "model",
 "version": 1,
 "component_version": 1,
 "description": "Chat completion client for OpenAI hosted models.",
 "label": "OpenAIChatCompletionClient",
 "config": {
 "model": "gpt-4o-mini",
 "api_key": "<your-openai-api-key>"
 }
}

This JSON representation reflects the configuration set in the UI: it specifies the gpt-4o-mini model and includes our API key. If we want to use Azure OpenAI, we must switch to another provider, which is autogen_ext.models.azure.AzureOpenAIChatCompletionClient. The configuration looks a bit different since you need some extra information to connect to Azure OpenAI:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


{
 "provider": "autogen_ext.models.openai.AzureOpenAIChatCompletionClient",
 "component_type": "model",
 "version": 1,
 "component_version": 1,
 "description": "Chat completion client for Azure OpenAI hosted models.",
 "label": "AzureOpenAIChatCompletionClient",
 "config": {
 "model": "gpt-4o-mini",
 "api_key": "sk-...",
 "azure_endpoint": "https://{your-custom-endpoint}.openai.azure.com/",
 "azure_deployment": "{your-azure-deployment}",
 "api_version": "2024-06-01"
 }
}

Compared to OpenAI, we need to add a few extra properties in the config section:

azure_endpoint: the endpoint of your Azure OpenAI resource.
azure_deployment: the deployment name of your Azure OpenAI resource.
api_version: the API version to use.

Since we’re already examining the agent’s JSON configuration, let’s make another change not currently supported by the UI. If you take a look again at the JSON, you will find a property called tools, which stores all the tools that we want the agent to be able to use to perform their tasks:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


"tools": [
 {
 "provider": "autogen_core.tools.FunctionTool",
 "component_type": "tool",
 "version": 1,
 "component_version": 1,
 "description": "Create custom tools by wrapping standard Python functions.",
 "label": "FunctionTool",
 "config": {
 "source_code": "def calculator(a: float, b: float, operator: str) -> str:\n try:\n if operator == '+':\n return str(a + b)\n elif operator == '-':\n return str(a - b)\n elif operator == '*':\n return str(a * b)\n elif operator == '/':\n if b == 0:\n return 'Error: Division by zero'\n return str(a / b)\n else:\n return 'Error: Invalid operator. Please use +, -, *, or /'\n except Exception as e:\n return f'Error: {str(e)}'\n",
 "name": "calculator",
 "description": "A simple calculator that performs basic arithmetic operations",
 "global_imports": [],
 "has_cancellation_support": false
 }
 }
]

By default, AutoGen Studio adds a FunctionTool to our agent called calculator, which is a custom Python function to do simple math operations. Our RAP MC agent won’t need it, so we can safely remove it from the JSON. Pay attention that we can’t remove the entire tools property, otherwise the JSON won’t be valid, but we just need to turn it into an empty array:

1

"tools": []

Creating the other agents

Now that we have seen how to create the first agent, we just need to replicate the same steps, but for the other agents.

Let’s start with the first rapper agent, which will be called Eminem Agent, since we’re going to configure it in a way that it will generate lyrics following the style of Eminem. Drag and drop a new AssistantAgent inside the team and configure it as follows:

Label: Eminem Agent
Description: An agent that generates rap lyrics in the style of Eminem
Name: eminem_agent
Description: An agent that generates rap lyrics in the style of Eminem
System Message:

1
2
3
4
5


You are a rapper, and you rap in the style of Eminem.
You are participating in a rap battle.
You will be given a topic, and you will need to create the lyrics and rap about it.
Be creative, but remember to stay on topic.
Do your best, since you're going to compete against another rapper.

Now let’s move to the second agent, which will be called Drake Agent since, in this case, it’s going to generate lyrics following the style of Drake. Drag and drop another AssistantAgent inside the team and configure it as follows:

Label: Drake Agent
Description: An agent that generates rap lyrics in the style of Drake
Name: drake_agent
Description: An agent that generates rap lyrics in the style of Drake
System Message:

1
2
3
4
5


You are a rapper, and you rap in the style of Drake.
You are participating in a rap battle.
You will be given a topic, and you will need to create the lyrics and rap about it.
Be creative, but remember to stay on topic.
Do your best, since you're going to compete against another rapper.

Finally, let’s create the judge agent, which will evaluate the lyrics and declare a winner. Drag and drop for the last time an AssistantAgent inside the team and configure it as follows:

Label: Judge Agent
Description: An agent that evaluates the rap lyrics and declares a winner
Name: judge_agent
Description: An agent that evaluates the rap lyrics and declares a winner
System Message:

1
2
3
4
5
6
7


You are a judge in a rap battle.
You will be given the lyrics of the rap battle and you must give a score to each contestant.
You must give a score between 1 and 10 to each contestant. The rapper who gets the highest score wins.
You must ensure that the final score is given after all the other contestants have shared their lyrics.
In addition to the score, you also need to provide an explanation of why you assigned that score.
YOUR FINAL RESPONSE MUST BE THE SCORE AND THE WINNER DECLARATION.
When the plan is complete and the winner has been declared, you can respond with TERMINATE.

As you can see, for the last agent the instructions are a bit more detailed, since we need to manage the signal that the rap battle is over, so the multi-agent conversation must be wrapped up. In this case, we’re asking to the Judge Agent to respond with the word TERMINATE when the rap battle is over and the winner has been declared.

Don’t forget, for every agent you have just created, to make the same changes we applied to the MC agent:

Provide the API key for using the OpenAI model (or switch to the JSON editor to change the configuration to use Azure OpenAI).
Remove the calculator tool from the configuration.

Now we’re ready to put everything together and set up the team.

Setting up the team

Since we dragged and dropped the agents inside the team in the previous section, most of the configuration is already completed. We just need to set up the termination condition, which is the last step before we can start the rap battle. Since we configured the Judge Agent to respond with the word TERMINATE when the rap battle is over, we can use this keyword as termination condition. Drag the TextMentionTermination condition from the Component Library and drop it inside the team, under the Terminations section. Now click on the pencil icon to edit the team. Since we’re here, let’s give to the team also a more meaningful name, by changing the label property for example to Rap Battle. Then go on and perform the two following actions:

Change the Max Turn property to 10 otherwise, after the first agent response, the conversation will terminate.
Click on the edit button near Termination Condition in the Configuration section. You should see that the termination condition is already set in the proper way for our scenario: the Text property is set to TERMINATE, which is the keyword we have configured for the Judge Agent to signal that the rap battle is over.

Now, we’re ready to start the rap battle! Before doing that, however, as a reference let me share the complete JSON configuration of the team after we have completed all the changes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198


{
 "provider": "autogen_agentchat.teams.RoundRobinGroupChat",
 "component_type": "team",
 "version": 1,
 "component_version": 1,
 "description": "A team of agents that chat with users in a round-robin fashion.",
 "label": "Rap Battle v2",
 "config": {
 "participants": [
 {
 "provider": "autogen_agentchat.agents.AssistantAgent",
 "component_type": "agent",
 "version": 1,
 "component_version": 1,
 "description": "An agent that acts as a master of ceremony in a rap battle",
 "label": "Rap MC Agent",
 "config": {
 "name": "rapmc_agent",
 "model_client": {
 "provider": "autogen_ext.models.openai.OpenAIChatCompletionClient",
 "component_type": "model",
 "version": 1,
 "component_version": 1,
 "description": "Chat completion client for OpenAI hosted models.",
 "label": "OpenAIChatCompletionClient",
 "config": {
 "model": "gpt-4o-mini",
 "api_key": "sk-..."
 }
 },
 "tools": [],
 "handoffs": [],
 "model_context": {
 "provider": "autogen_core.model_context.UnboundedChatCompletionContext",
 "component_type": "chat_completion_context",
 "version": 1,
 "component_version": 1,
 "description": "An unbounded chat completion context that keeps a view of the all the messages.",
 "label": "UnboundedChatCompletionContext",
 "config": {}
 },
 "description": "An agent that acts as a master of ceremony in a rap battle",
 "system_message": "You are a rap MC and your role is to run a rap battle with two contestants. The user will provide you with the topic for the battle. Your main task is to be the master of ceremony and introduce the rap battle. You must share some words of encouragement and describe the topic for the battle that was given by the user. YOU MUST USE THE TOPIC GIVEN BY THE USER, you can't make up one on your own. \nYou must be very enthusiastic and use a lot of exclamation marks. You must also use a lot of slang words and rap terms. You must also use emojis.",
 "model_client_stream": false,
 "reflect_on_tool_use": false,
 "tool_call_summary_format": "{result}",
 "label": "Rap MC Agent",
 "config": {
 "name": "assistant_agent",
 "description": "An agent that acts as a master of ceremony in a rap battle",
 "system_message": "You are a helpful assistant. Solve tasks carefully. When done, say TERMINATE.",
 "reflect_on_tool_use": false,
 "tool_call_summary_format": "{result}"
 }
 }
 },
 {
 "provider": "autogen_agentchat.agents.AssistantAgent",
 "component_type": "agent",
 "version": 1,
 "component_version": 1,
 "description": "An agent that generates rap lyrics in the style of Eminem",
 "label": "Eminem Agent",
 "config": {
 "name": "eminem_agent",
 "model_client": {
 "provider": "autogen_ext.models.openai.OpenAIChatCompletionClient",
 "component_type": "model",
 "version": 1,
 "component_version": 1,
 "description": "Chat completion client for OpenAI hosted models.",
 "label": "OpenAIChatCompletionClient",
 "config": {
 "model": "gpt-4o-mini",
 "api_key": "sk-..."
 }
 },
 "tools": [],
 "handoffs": [],
 "model_context": {
 "provider": "autogen_core.model_context.UnboundedChatCompletionContext",
 "component_type": "chat_completion_context",
 "version": 1,
 "component_version": 1,
 "description": "An unbounded chat completion context that keeps a view of the all the messages.",
 "label": "UnboundedChatCompletionContext",
 "config": {}
 },
 "description": "An agent that generates rap lyrics in the style of Eminem",
 "system_message": "You are a rapper, and you rap in the style of Eminem. You are participating in a rap battle. You will be given a topic, and you will need to create the lyrics and rap about it. Be creative, but remember to stay on topic. Do your best, since you're going to compete against another rapper.",
 "model_client_stream": false,
 "reflect_on_tool_use": false,
 "tool_call_summary_format": "{result}"
 }
 },
 {
 "provider": "autogen_agentchat.agents.AssistantAgent",
 "component_type": "agent",
 "version": 1,
 "component_version": 1,
 "description": "An agent that generates rap lyrics in the style of Drake",
 "label": "Drake Agent",
 "config": {
 "name": "drake_agent",
 "model_client": {
 "provider": "autogen_ext.models.openai.OpenAIChatCompletionClient",
 "component_type": "model",
 "version": 1,
 "component_version": 1,
 "description": "Chat completion client for OpenAI hosted models.",
 "label": "OpenAIChatCompletionClient",
 "config": {
 "model": "gpt-4o-mini",
 "api_key": "sk-..."
 }
 },
 "tools": [],
 "handoffs": [],
 "model_context": {
 "provider": "autogen_core.model_context.UnboundedChatCompletionContext",
 "component_type": "chat_completion_context",
 "version": 1,
 "component_version": 1,
 "description": "An unbounded chat completion context that keeps a view of the all the messages.",
 "label": "UnboundedChatCompletionContext",
 "config": {}
 },
 "description": "An agent that generates rap lyrics in the style of Drake",
 "system_message": "You are a rapper, and you rap in the style of Drake. You are participating in a rap battle. You will be given a topic, and you will need to create the lyrics and rap about it. Be creative, but remember to stay on topic. Do your best, since you're going to compete against another rapper.",
 "model_client_stream": false,
 "reflect_on_tool_use": false,
 "tool_call_summary_format": "{result}"
 }
 },
 {
 "provider": "autogen_agentchat.agents.AssistantAgent",
 "component_type": "agent",
 "version": 1,
 "component_version": 1,
 "description": "An agent that evaluates the rap lyrics and declares a winner",
 "label": "Judge Agent",
 "config": {
 "name": "judge_agent",
 "model_client": {
 "provider": "autogen_ext.models.openai.OpenAIChatCompletionClient",
 "component_type": "model",
 "version": 1,
 "component_version": 1,
 "description": "Chat completion client for OpenAI hosted models.",
 "label": "OpenAIChatCompletionClient",
 "config": {
 "model": "gpt-4o-mini",
 "api_key": "sk-..."
 }
 },
 "tools": [],
 "handoffs": [],
 "model_context": {
 "provider": "autogen_core.model_context.UnboundedChatCompletionContext",
 "component_type": "chat_completion_context",
 "version": 1,
 "component_version": 1,
 "description": "An unbounded chat completion context that keeps a view of the all the messages.",
 "label": "UnboundedChatCompletionContext",
 "config": {}
 },
 "description": "An agent that evaluates the rap lyrics and declares a winner",
 "system_message": "You are a judge in a rap battle. You will be given the lyrics of the rap battle and you must give a score to each contestant. You must give a score between 1 and 10 to each contestant. The rapper who gets the highest score wins. \\\\n You must ensure that the final score is given after all the other contestants have shared their lyrics. On top of the score, you need also to provide an explanation on why you assinged that score. YOUR FINAL RESPONSE MUST BE THE SCORE AND THE WINNER DECLARATION. When the plan is complete and the winner has been declared, you can respond with TERMINATE.",
 "model_client_stream": false,
 "reflect_on_tool_use": false,
 "tool_call_summary_format": "{result}"
 }
 }
 ],
 "termination_condition": {
 "provider": "autogen_agentchat.conditions.TextMentionTermination",
 "component_type": "termination",
 "version": 1,
 "component_version": 1,
 "description": "Terminate the conversation if a specific text is mentioned.",
 "label": "TextMentionTermination",
 "config": {
 "text": "TERMINATE",
 "label": "TextMentionTermination",
 "description": "Terminate the conversation if a specific text is mentioned.",
 "config": {
 "text": "TERMINATE"
 }
 }
 },
 "max_turns": 1,
 "label": "Rap Battle v2",
 "description": "A team of agents that chat with users in a round-robin fashion.",
 "config": {
 "max_turns": 1
 }
 }
}

Test the multi-agent scenario

To test our rap battle, we must head to the Playground section of AutoGen Studio and click on New session. Give it a name you like and, under Team, pick the team we have just created. Click on Create to create the new session.

You will start a chat session, which will give you the chance to send a message to the team and see how it responds. For our rap battle, let’s just provide the topic, by typing for example the following message:

1

The topic of the rap battle is "marshmallow"

After sending the message, you should see the following happening:

The Rap MC agent will share an introduction to the rap battle, by describing the topic. You will notice how the agent will follow the style we included in the instructions, like adding emojis and using a rap slang. In my case, for example, the agent created the following introduction:

1
2
3
4
5
6
7
8
9


🎤🎶 Yo! Yo! Yo! What’s good, rap fam?! Welcome to the hottest rap battle of the century! It’s your boy, the MC with the flyest flow in the game, and I’m here to serve up an epic showdown for y’all! 💥🔥

Tonight, we're diving into a topic that’s sweeter than candy and fluffier than clouds in the sky— "MARSHMALLOW!" That’s right, folks! We’re talkin’ about that gooey goodness that melts in your mouth and sticks in your heart! 🍬❤️

So, whether you’re vibin’ with s’mores around the campfire or just chillin’ with a hot cocoa, these two contestants are ready to spit some fire and mix it up with those marshmallow bars! 🚀🍫

I wanna hear your loudest cheers, your hype, and your energy for our dope contenders! Let’s lift them up and show ‘em some love! 💚💛 You see, rap battles are all about creativity and passion, so let's get ready to witness some epic flows, clever wordplay, and marshmallow magic! 🎤🔥

Remember, fam, it’s all in good fun— no beef, just beats! Now, without further ado, let’s bring these lyrical legends to the stage!! Who’s ready to get this marshmallow madness poppin’?! Let’s go!!! 🎉🎤✌️

After the MC agent, the Eminem agent will respond with its lyrics, followed by the Drake agent, which which will generate its own version of the lyrics.
Finally, the Judge agent will respond with the score and the winner. In my case, for example, it generated the following response:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


Scores:

Eminem: 9/10
Drake: 8/10
Explanation:

Eminem delivered a powerful performance with clever wordplay and an engaging flow that seamlessly adapted the marshmallow theme into a vivid and memorable rap experience. His verses incorporated a narrative about the essence of marshmallows while cleverly intertwining personal and nostalgic elements, making it relatable and fresh. The repetition of the chorus further emphasized his theme, and the rhythm maintained the audience's excitement. All of this culminated in a strong performance worthy of a high score.

Drake brought a lighter, melodic vibe that was enjoyable but felt slightly less impactful compared to Eminem's approach. His lyrics were catchy and showcased his signature style, yet some of the lines lacked the same punch and depth that made Eminem's verses stand out. While the composition was solid and fit well with the theme, it didn't quite reach the same level of innovation.

In conclusion, Eminem's overall execution and creativity in writing earned him the victory in this rap battle!

Winner: Eminem
Final Score: Eminem 9, Drake 8

TERMINATE

As you can see, the response ends with the TERMINATE word, which will lead to the termination of the conversation.

Once the rap battle is ended, at the top will see the last message posted in the conversation and the reason why the conversation is ended.

Additionally, you can see two features that make AutoGen Studio a very interesting tool to test these multi-agent scenarios ahead of their implementation in code:

We can track the number of messages and tokens used to estimate the conversation’s potential cost.
We can see a visual representation of the workflow, including the number of tokens used in each step

From the visual workflow, we can see that in our case our conversation was quite linear: every agent responded one after the other. This is because the type of team we used is the RoundRobinGroupChat one, which does exactly that: invoking one agent after the other, according to the order we have added them into the team. However, AutoGen Studio also supports more complex scenarios where agents can respond in a non-linear fashion. We’ll see more in the next blog post, in which we’ll add an extra feature to our rap battle: the ability to use tools.

Conclusion

In this post, we have seen how to create a multi-agent scenario using AutoGen Studio. We have created a rap battle scenario, in which we had an MC agent, two rapper agents and a judge agent. We have also seen how to configure the team and the termination condition, so that the conversation would end when the judge agent declared the winner. However, this was just the first post of a series. In the next one, we’ll add a new feature to our rap battle: a tool to generate a list of words, that the Rap MC agent will use to encourage the two rappers to use them in their lyrics.

Then, we will see also how to turn the playground test we did into a real application, by leveraging AutoGen as an application framework.

Happy coding!

Getting started with declarative copilots in Microsoft 365

Thu, 22 Aug 2024 00:00:00 +0000

Please note! Declarative copilots and API plugins are currently in private preview. As such, if you want to put in practice what you’re going to read in this post, you need to be part of the private preview program (which isn’t open for submissions at the moment) or wait until the features will be publicly released. However, all the information you’re going to find in this post are public. Even if you can’t actually build them, declarative copilots and API plugins have been unveiled at Build and they are publicly documented. I will make sure to update the post once the feature will go live.

If you have ever worked with Copilot for Microsoft 365, I’m pretty sure at some point someone from your company came up with the following requirements: “Can I restrict the scope of Copilot for Microsoft 365 so that it can give me answers starting only from a specific data source, like a SharePoint site?”. Copilot for Microsoft 365, in fact, is very powerful but, for some scenarios, it might be too powerful. For example, let’s say that you want your employees to easily get answers about the IT policies of the company. In this scenario, you’ll probably want Copilot to ground its responses only from the official internal SharePoint site for the IT department, instead of using every document or information that might be on SharePoint, OneDrive or in your emails. Up until today, the only answer to this scenario was to build a custom copilot, either with Copilot Studio or with a pro-code solution like the the Teams AI library. It’s a good solution, but it introduces a level of complexity: being a custom solution, you need to take care of setting up the LLM, the deployment, the hosting, finding the right channel to use, etc.

In this blog post, we’ll build a declarative copilot using the pro code approach with the Teams Toolkit.

A little bit of context

Declarative copilots are one of the elements of the Copilot for Microsoft 365 extensibility story. Up until Build, the two main components of the extensibility story were plugins and connectors. Their goal was to enable customers to integrate 3rd party platforms into the Copilot ecosystem:

With Graph Connectors, you can expand the Copilot knowledge, by importing external data into the Microsoft 365 tenant. This way, Copilot can generate answers not only from your Microsoft 365 data (documents on SharePoint, meetings, mails, etc.), but also from data coming from another platform like SAP, Jira Cloud, Salesforce or your own LOB application.
With plugins, you can expand the Copilot capabilities, by enabling it to get data and perform actions related to a 3rd party platform. For example, by building a plugin for your support ticket platform, you enable Copilot to generate answers starting from your tickets, or to change the status of a ticket with a prompt.

In addition to these two options, now you can add declarative copilots into the picture, which serve a slightly different purpose, but they are still heavily connected with the whole extensibility ecosystem. As we mentioned in the beginning of the post, in fact, a Declarative Copilot is a way to build a customized Copilot, but still based on the Copilot for Microsoft 365 foundations: the AI models, the orchestration engine, the safety features, etc. When you create a declarative copilot, you can customize the following options:

You can provide your own set of instructions, which will act as a system prompt. This way, you can instruct Copilot to behave the way which is more compliant for your scenario. Let’s take the IT scenario we introduced before: you could provide a system prompt that instructs the LLM to answer only to questions related to IT topics and to ignore every other question.
You can provide a list of conversation starters, which are suggested prompts that are displayed when you use the declarative copilot.
You can provide a list of SharePoint / OneDrive files and folders. The declarative copilot will use only them to ground its responses.
You can turn the web plugin on, enabling the declarative copilot to use results from the web to ground its responses.

Among these features, declarative copilots participate into the extensibility ecosystem since they can use plugins and connectors. This means that:

You can provide a list of plugins that the Copilot will automatically use.
You can provide a list of Graph Connectors that the Copilot will be able to use to ground responses on external data.

You can clearly see that declarative copilots are based on the Copilot for Microsoft 365 architecture because the starting point to use them is, indeed, the Copilot for Microsoft 365 application. There are two ways to use a declarative copilot:

From the right panel in Copilot. At the top, before the chat history, you will find a list of all the declarative Copilots that are available to you. When you click on the new chat icon near a declarative copilot, you will start a focused session with it.
From the main Copilot chat, you can use the @ symbol to reference your declarative copilot and “forward” the prompt to it:

Now that we have understood what’s the purpose of a declarative copilot and how to use it, let’s create one!

Create our first declarative copilot

An important concept to share before we start is that a declarative copilot doesn’t include any code. Since you’re using the Copilot for Microsoft 365 engine behind the scenes, you don’t have a way to customize it or change it. All the configurations happen through a JSON file, which you’re going to use to setup all the scenarios we’ve highlighted before, like providing your own set of instructions or restricting the knowledge source to a specific SharePoint site.

You’re going to see this soon. The easiest way to create a declarative copilot is by using the Teams Toolkit, so make sure you have installed it in Visual Studio Code. You’ll need the pre-release version, the standard version doesn’t support the new extensibility options yet. Then, open a new instance of Visual Studio Code, click on the Teams Toolkit icon in the sidebar and choose Create a New App. The first option you’ll see is Copilot Extensions:

Pick up Declarative Copilot then, for the moment, choose No plugin. Select a folder in which you want to create the project and then give it a name. The Teams Toolkit will create the basic structure for your project:

Once your project has been created, you will immediately realize what I just shared: the project doesn’t contain any JavaScript / TypeScript file or C# class, but just a bunch of JSON files and images.

The first file you’ll find is the manifest.json one, which shouldn’t come as a surprise if you have experience with developing extensions for Copilot for Microsoft 365. Declarative copilots are wrapped as Teams application, so this is the standard manifest which describes the application and it includes information like the package id, the name, the descriptions, etc.

The only special component is the copilotExtensions section, which has been recently introduced to support all the Copilot extensibility scenarios. It’s a new section and, in fact, you can notice at the top of the file that the manifest is using a preview version of the schema:

1

"$schema": "https://developer.microsoft.com/json-schemas/teams/vDevPreview/MicrosoftTeams.schema.json"

Inside the copilotExtensions sections, we can see the declaration that this Teams application includes a declarative copilot:

1
2
3
4
5
6
7
8


"copilotExtensions": {
 "declarativeCopilots": [
 {
 "id": "declarativeCopilot",
 "file": "declarativeCopilot.json"
 }
 ]
}

As you can see, the definition is extremely simple: just an identifier (which is a fixed value, declarativeCopilot) and a reference to the JSON file which defines the configuration of the declarative copilot.

This is where things are more interesting. Let’s take a look at the declarativeCopilot.json file!

The declarative Copilot configuration

By default, the declarativeCopilot.json file includes a bunch of properties: name, description and instructions. name and description are quite easy to understand and it’s important to customize them, because they will be displayed to the user when he chooses to use your declarative copilot.

The most important one is instructions, which is the prompt that will act as a system prompt. It should describe with as many details as possible the role of your declarative copilot, what it can do and what it can’t do.

For instance, let’s say that we’re building the IT Copilot for our company that we have taken as example in the beginning of the post. This is how your JSON file might look like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


{
 "$schema": "https://aka.ms/json-schemas/copilot-extensions/v1.0/declarative-copilot.schema.json",
 "name": "Contoso IT Copilot",
 "description": "Your starting point for all your IT questions!",
 "instructions": "You are a Copilot for the Contoso company who is specialized in supporting employees with their HR questions.
 You should start every response and answer to the user with \"Thanks for using the Contoso IT Copilot!\\n\" and then answer the questions and help the user.
 However, you can help the user only if he asks a question which is related to IT, like company policies, benefits, career development.
 If the user asks a question which isn't about one of these topics, say 'I don't know'.
 Never try to answer it anyway."
}

We have crafted a set of instructions with two key features:

We have defined an introductory phrase that the Copilot must use whenever it starts a new interaction. This is a good way for us, as developers, to make sure that the declarative copilot is indeed following our instructions.
We have stated what it can do and what it can’t do. It’s an IT Copilot, so it should answer only questions related to this topic, otherwise it should say that it doesn’t know.

This is good enough to start testing your declarative copilot! These properties, in fact, are the minimum ones required to deploy it. Again, thanks to the Teams Toolkit, it’s a really easy operation. First, click on the Teams Toolkit icon in the Visual Studio Code sidebar. Then, make sure that at the top, in the Accounts section, you are logged in with your Microsoft 365 account and that the check Custom App Upload Enabled is green:

Now click on Provision, under the Lifecycle section. The Teams Toolkit will create a ZIP file with the manifest, the configuration file, the assets and it will sideload it on your Microsoft 365 tenant.

Please ignore the Deploy and Publish options in the Teams Toolkit. Unlike with other types of Teams applications (like custom copilots or message extensions), there is no infrastructure to deploy in this case, since there is no code. All the required files will bundled inside the ZIP file that gets uploaded to the tenant.

Once the operation is completed, go to the Microsoft 365 portal, login with your Microsoft 365 account and click on the Copilot icon in the left sidebar. You should see your declarative copilot showing up in the right pane, as you’ve seen in one of the previous screenshots.

Click on the new chat icon near it and try to ask some questions. You should observe that, if you ask a question which is related to a IT topic, your copilot will reply. If, instead, it’s totally out of topic, you should see the message I don’t know.

For example, the image below shows what happens when I ask What’s the weather in Milan in December?.

You can see that Copilot followed both our instructions:

It started the response with the phrase “Thanks for using the Contoso IT Copilot!”.
It replied that it doesn’t know the answer.

This is a good starting point, but the experience is a bit limited. Providing a set of instructions helped to narrow down the scope of our Copilot, but it misses access to the proper data and tools to respond correctly. Let’s see how we can add new capabilities!

Adding conversation starters

One thing you might have noticed when you have tried your declarative copilot is that, despite you are using a more scoped down version, Copilot is still suggesting generic Microsoft 365-related prompts, like Summarize the meeting X or Catch up on your tasks with Y. This can be misleading, because some of these prompts might not even work, given the new set of instructions we have provided. Luckily, the configuration file easily supports replacing them with your own, thanks to a section in the JSON file called conversation_starters:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


"conversation_starters": [
 {
 "title": "Enrolling your mobile device",
 "text": "How can I use company resources on my mobile device?"
 },
 {
 "title": "Getting Help",
 "text": "Which are the steps to follow if my PC doesn't work?"
 }
]

It’s simply a collection of key / value pairs, where title is the bold text shown at the top of the card, while text is the text shown in the middle. The following image will help you to understand how they are used:

Adding a SharePoint or OneDrive source

Now that we have customized the look and feel of our declarative copilot, let’s focus on the functionality. The most common requirement you get about when you decide to build a declarative copilot is scoping the data source. In our case, we want our IT Copilot to answer only using, as knowledge, the content that is published on the internal IT SharePoint site, instead of using every content available in the Microsoft 365 tenant. You can specify one or more data source using the OneDriveAndSharePoint capability:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


"capabilities": [
 {
 "name": "OneDriveAndSharePoint",
 "items_by_url": [
 {
 "url": "https://contoso.sharepoint.com/sites/IT"
 }
 ]
 }
]

Inside this capability you can use the items_by_url property to specify the URL of one or more SharePoint or OneDrive locations. In this case, we have specified only one URL, which is the URL of the IT department SharePoint site. This means that our declarative copilot will use only the content that is published on this site to ground its responses.

You can test it by using again the Teams Toolkit to provision your declarative copilot and by asking questions which are covered by the content you have published on your SharePoint site. You’ll notice that, among the references listed at the bottom of the response, Copilot won’t list anymore mails, Teams messages or content from other sites other than the one you have specified in the configuration file.

Adding a Graph Connector

On this blog we have seen different posts covering what Graph Connectors are and how you can create custom ones. They are, essentially, treated like every other data source in Microsoft 365 (Exchange, Teams, SharePoint, etc.), except that the data has been imported from an external data source, like an internal wiki or a knowledge base. By default, Copilot can use this data source to ground its responses. With a declarative copilot, we can restrict it to use only one or more of the Graph Connectors that are available in the tenant. We can do this thanks to the GraphConnectors capability, as in the following sample:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


{
 "capabilities": [
 {
 "name": "GraphConnectors",
 "connections": [
 {
 "connection_id": "devcantina"
 }
 ]
 }
 ]
}

Inside the connections property, you can provide one or more connection_id properties, with the id of the connector you want to enable. If you have built the connector yourself, you should know the ID, since it’s the one you have provided when you have defined the connection’s configuration. In case you’re using one of the pre-built connectors, instead, you can find the ID by going to the Microsoft 365 Admin portal, heading to the Settings -> Search & Intelligence -> Data sources section and clicking on the connector you are interested to. A panel with all the details of the connector will open up, in the image below you can see highlighted in red the id:

Adding a plugin

The last capability we’re going to see is the ability to embed plugins which, as we mentioned in the beginning, are a way to integrate 3rd party platforms into the Copilot ecosystem. The two key differences compared to a Graph Connector are:

Plugins are mapped around API calls, so they retrieve data in real time from the 3rd party platform. There is no data ingestion into the tenant, but there’s a real time interaction with an external platform.
Plugins are used not just to expand knowledge, but also to perform actions.

Plugins, like Graph Connectors, aren’t tied to declarative copilots, they can be used also with the standard Copilot for Microsoft 365 experience. In this scenario, however, you must manually open the plugin flyout and turn on the one (or the ones) you want to use and then explicitly refer to them:

However, when you bundle them together with a declarative copilot, they are automatically turned on. You don’t need to call them out explicitly but, if Copilot understands that a plugin can help to ground a response, it will use it.

The Teams Toolkit supports the option to create a declarative copilot with a plugin, using another one of the new extensibility features announced back at Build: API plugins. Up until today, the only way to create a plugin using a pro-code approach was with Message Extensions plugins, which are very flexible but they require you to write code to receive the query search from Copilot, to call the 3rd party platform to get the results and then to generate an Adaptive Card to return the information.

The idea behind API plugins, instead, is that most of the time, when you want to integrate a 3rd party platform, you just need to call the APIs that it offers. As such, API plugins don’t include any code, but they are wrapped around an OpenAPI definition. OpenAPI is a standard way to describe a REST APIs, using JSON or YAML. It simply describes all the available endpoints, the inputs and outputs, the authentication methods, etc. It’s agnostic, so it doesn’t contain any specific information about the implementation. Behind the scenes, you could have a .NET API hosted on Azure or a Node.js API hosted on AWS, it doesn’t matter.

For the rest of the post, I’m going to assume that you already have an API with an OpenAPI definition that you can leverage. Otherwise, you can easily build a test one with many technologies, like .NET Web APIs or Azure Functions. Both technologies offer built-in OpenAPI support, so you can generate an OpenAPI definition out of your ASP.NET Web API or Azure Function simply by decorating them with special attributes.

You can also use my sample Ticket API which I’ve published on GitHub.

The Teams Toolkit support generating the required files to integrate an API plugin for you. Open a new Visual Studio Code instance, click again on the Teams Toolkit icon, choose Create a new app and pick again Copilot Extension, then Declarative Copilot. This time, however, choose Add plugin in the next step, followed by Start with an OpenAPI Description Document. Then click on the first option, the one with the cloud icon called Enter OpenAPI Description Document location and provide the URL of your OpenAPI definition. If you have created your API with .NET Web APIs or Azure Functions and you’re using the Swagger integration, the URL will be something like https://yourapi.com/api/swagger.json (or https://yourapi.com/api/swagger.yaml, both formats are supported).

If the OpenAPI definition is properly formatted, the Teams Toolkit will identify the available operations and it will ask you one which ones you want to import. The following image shows what I get when I use my test API which simulates a ticket management platform. As you can see, it properly recognized the three supported operations:

GET, to get the list of tickets
POST, to create a new ticket
DELETE, to delete a ticket

Once you have selected the operations you want to import, the Teams Toolkit will create two files for you inside the appPackage folder:

One called openapi.json (or openapi.yaml, depending on the source format you have used), which is basically a local copy of the OpenAPI definition API (eventually trimmed down, in case you have chosen to import only a subset of all the available operations). For example, this is how the file looks for my scoped down API (I’ve imported only the GET operation):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69


{
 "openapi": "3.0.1",
 "info": {
 "title": "Ticket APIs",
 "version": "1.0"
 },
 "servers": [
 {
 "url": "https://<my-api.com>/api"
 }
 ],
 "paths": {
 "/tickets": {
 "get": {
 "description": "Get the tickets with a given keyword in the title",
 "operationId": "GetTickets",
 "parameters": [
 {
 "name": "search",
 "in": "query",
 "description": "The search keyword",
 "schema": {
 "type": "string"
 }
 }
 ],
 "responses": {
 "200": {
 "description": "OK",
 "content": {
 "application/json": {
 "schema": {
 "type": "array",
 "items": {
 "$ref": "#/components/schemas/ticket"
 }
 }
 }
 }
 }
 }
 }
 }
 },
 "components": {
 "schemas": {
 "ticket": {
 "type": "object",
 "properties": {
 "id": {
 "type": "string"
 },
 "title": {
 "type": "string"
 },
 "description": {
 "type": "string"
 },
 "assignedTo": {
 "type": "string"
 },
 "severity": {
 "type": "string"
 }
 }
 }
 }
 }
}

As you can see, it’s just a description of the API: in this case, the /tickets endpoint can be called with a GET operation, which optionally supports a query string parameter called search. The response is a JSON array of objects, each one representing a ticket. The ticket object has a few properties, like id, title, description, etc.

One called ai-plugin.json, which is a sort of enhanced version of the OpenAPI definition specialized for Copilot. It takes care of adding Copilot specific configurations, like name and descriptions (which are used by the model to determine if the plugin should be called), adaptive card to be used as part of the response, etc. This is an example file generated out of my Ticket API:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82


 {
 "$schema": "https://aka.ms/json-schemas/copilot-extensions/v2.1/plugin.schema.json",
 "schema_version": "v2.1",
 "name_for_human": "Ticket Plugin",
 "description_for_human": "A plugin to manage tickets",
 "namespace": "ticketplugin",
 "functions": [
 {
 "name": "GetTickets",
 "description": "Get the tickets with a given keyword in the title",
 "capabilities": {
 "response_semantics": {
 "data_path": "$",
 "properties": {
 "title": "$.title",
 "subtitle": "$.id"
 },
 "static_template": {
 "type": "AdaptiveCard",
 "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
 "version": "1.5",
 "body": [
 {
 "type": "Container",
 "$data": "${$root}",
 "items": [
 {
 "type": "TextBlock",
 "text": "id: ${if(id, id, 'N/A')}",
 "wrap": true
 },
 {
 "type": "TextBlock",
 "text": "title: ${if(title, title, 'N/A')}",
 "wrap": true
 },
 {
 "type": "TextBlock",
 "text": "description: ${if(description, description, 'N/A')}",
 "wrap": true
 },
 {
 "type": "TextBlock",
 "text": "assignedTo: ${if(assignedTo, assignedTo, 'N/A')}",
 "wrap": true
 },
 {
 "type": "TextBlock",
 "text": "severity: ${if(severity, severity, 'N/A')}",
 "wrap": true
 }
 ]
 }
 ]
 }
 }
 }
 }
 ],
 "runtimes": [
 {
 "type": "OpenApi",
 "auth": {
 "type": "None"
 },
 "spec": {
 "url": "apiSpecificationFile/openapi.json"
 },
 "run_for_functions": [
 "GetTickets"
 ]
 }
 ],
 "capabilities": {
 "localization": {},
 "conversation_starters": [
 {
 "text": "Get the tickets with a given keyword in the title"
 }
 ]
 }
 }

As you can see, the file lists the same operations that are available in the OpenAPI definition. The whole definition of the operation is demanded to the OpenAPI specification (you can see that, in the runtimes section, we have a reference to the openapi.json file). However, on top of the definition, we are adding a few more properties which are Copilot specific, like:

A name and a description
A response_semantics section, which includes the properties section (which specifies which are the key properties of our data that Copilot should ues when it references it) and the static_template one (which is an Adaptive Card that Copilot will use to render the data). In this case, we have a simple Adaptive Card with a few TextBlocks, one for each property of the ticket.

Notice, at the end of the file, how also this file contains the conversation_starters section. This is because also plugins can have their own set of default prompts, which are displayed when a plugin is turned on.

The glue between a declarative copilot and the plugins it can use happens, as you can expect, in the declarativeCopilot.json file, with a new section called actions:

1
2
3
4
5
6


"actions": [
 {
 "id": "ticketplugin",
 "file": "ai-plugin.json"
 }
]

Each action has an id and a file property, which contains the path to the ai-plugin.json file.

If you provision and test the declarative copilot, you won’t actually notice any different. As I mentioned, plugins are automatically turned on, you don’t have to manually activate them (in fact, the plugin flyout icon is disabled when you use a declarative copilot). However, if you write a prompt that requires using the plugin to perform the task, you will see your plugin being called. The first time you use it, you will be asked if you want to allow the plugin to access the data it needs. This is because, when Copilot invokes the plugin, is effectively sending data outside your tenant, since the search query is passed to an external API. You can see also see a preview of the data that will be shared with the API.

In the image below, you can see how Copilot has translated the prompt Give me the list of tickets about CSS into an API call passing, as value for the search parameter, the keyword CSS:

If you proceed, Copilot will get the data from the API and use it to generate a response. If you have provided an Adaptive Card for the selected operation, you will see it rendered when you hover one of the references (or directly in the UI, in case the response includes only a single result).

As I mentioned, unlike Graph Connectors, plugins can be used also to perform actions. The image below shows what happens after I have configured my plugin to include not just the GET operation, but also the POST one, which is used to create new tickets:

In this case, since the plugin is actually going to write data on the 3rd party platform, Copilot will always ask you to confirm the operation. You don’t have a way, unlike for reading operations, to always allow them.

Using a plugin outside a declarative copilot

If you want to use a plugin outside the scope of a declarative copilot and make it available as a general Copilot for Microsoft 365 plugin, the Teams Toolkit gives you the option to create a standalone one by choosing Create a New App -> Copilot Extension -> API plugin. The flow is exactly the same: you will be asked to provide the OpenAPI definition of the API and the Teams Toolkit will generate the same files: a local copy of the OpenAPI definition and an ai-plugin.json file. In fact, you can take an existing plugin you have imported in a declarative copilot and make it standalone, if you want.

The only difference is that, in case of a standalone plugin, instead of being declared as action in the declarativeCopilot.json file, it’s declared in the manifest.json file of the application, in the copilotExtensions section, like this:

1
2
3
4
5
6
7
8


"copilotExtensions": {
 "plugins": [
 {
 "file": "ai-plugin.json",
 "id": "ticketPlugin"
 }
 ]
}

No one prevents you to have the same plugin declared both in the declarativeCopilot.json file and in the manifest.json one. It will be used by the declarative copilot, but it will be also available to be used by the standard Copilot for Microsoft 365 experience through the plugin flyout. To make it simpler to use it, Copilot now supports a new syntax based on the concept of mention. You can reference a plugin by mentioning it with the @ symbol. As you can see in the image below, the context switches to the plugin and it will be used to ground the response. In the past, instead, to make the plugin selection more effective, you needed to explicitly call it (something like Use Ticket Plugin to give me the list of tickets about CSS).

Debugging a plugin

What if something goes wrong and Copilot doesn’t return the response you’re expecting? Debugging an API plugin can be challenging because, as you have seen, unlike for Message Extension plugins, there is no code. You can’t attach a debugger and see what happens behind the scene (unless you have the chance to do that on the API side).

Luckily, you can get some help by Copilot itself, thanks to the developer mode, which has been greatly enhanced to show more details about the API calls performed by Copilot. To turn it on, just write the following prompt in the chat panel:

1

-developer on

Now write a prompt that is going to trigger the plugin. Below the response, you will find many useful information. First, we can see the whole reasoning logic of Copilot. The image below shows that, when I use the prompt Give me the list of available tickets about CSS, Copilot has determined that:

The enabled plugin is Ticket Plugin.
The plugin exposes three functions: DeleteTicket, GetTickets and CreateTicket.
The function that should be called is the GetTickets one.

If we take a look at the execution details, we can see the full API call that Copilot performed:

Wrapping up

In this blog post we have seen how we can create a declarative copilot using the Teams Toolkit and how we can customize it to fit our needs. We have seen all the available options we have:

Providing a custom set of instructions
Adding conversation starters
Scoping the data source to a specific SharePoint site or OneDrive folder
Enabling the use of Graph Connectors
Embedding plugins

Thanks to declarative copilots, we can leverage the full power of the Copilot for Microsoft 36t engine, but customizing it in a way that can better fit more specialized scenarios. In this blog post, we have seen an example with an IT Copilot, but the same approach can be used for many other scenarios, like HR, finance, legal, or simply a specific project.

You can find the sample I built for this post on GitHub.

Happy coding!

Using Dev Proxy to troubleshoot issues with Graph Connectors

Tue, 13 Aug 2024 00:00:00 +0000

In a previous post we have learned how you can use the create-graph-connector project to speed up the development of a custom Graph Connectors. In that post, we have also briefly covered Dev Proxy, an open-source tool from Microsoft that you can use to better test your APIs. Thanks to Dev Proxy, you can intercept API calls, simulate API errors or throttling, mock responses so that you can test your code without needing the real APIs, and much more. In this post, we’ll focus on a specific feature of Dev Proxy that saved my life a few times while I was developing custom Graph Connectors: the ability to intercept and inspect requests and responses.

Configuring DevProxy

Dev Proxy is configured through a JSON file called devproxyrc.json. This file is used to define the rules that Dev Proxy will use to intercept the requests, the plugins you want to turn on, if you want to enable the logic to test failures or throttling, etc. Since you might want to use Dev Proxy with different projects, each of them with its own configuration, the tool supports two ways to define it:

A global devproxyrc.json file, which is stored in the %LOCALAPPDATA%\Programs\Dev Proxy folder.
A local devproxyrc.json file, which you can store in any folder of your project (e.g., the root folder of your project).

When you open the terminal and you type devproxy, the tool will automatically pick the “closest” one: if you are in a folder that contains a devproxyrc.json file, it will use that one; otherwise, it will use the global one.

The project that is created with the create-graph-connector tool is a great starting point, because it already includes a devproxyrc.json file configured to intercept requests against the Microsoft Graph. If you take a look, in fact, you will find the following urlsToWatch section:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


 "urlsToWatch": [
 "https://graph.microsoft.com/v1.0/*",
 "https://graph.microsoft.com/beta/*",
 "https://graph.microsoft.us/v1.0/*",
 "https://graph.microsoft.us/beta/*",
 "https://dod-graph.microsoft.us/v1.0/*",
 "https://dod-graph.microsoft.us/beta/*",
 "https://microsoftgraph.chinacloudapi.cn/v1.0/*",
 "https://microsoftgraph.chinacloudapi.cn/beta/*"
 ]

This list cover all the possible endpoints of the Microsoft Graph, including the beta, US Government and the China ones.

As such, the easiest way to start using Dev Proxy to test your custom Graph Connector is either to copy this section into your configuration file or to grab a copy the entire devproxyrc.json from a Graph Connector sample project (like this one) and copy it into your own. In the second scenario, however, there’s a slight change we must make. If you take a look at the list of enabled plugins, you will find the following entry:

1
2
3
4
5
6


{
 "name": "GraphMockResponsePlugin",
 "enabled": true,
 "pluginPath": "~appFolder/plugins/dev-proxy-plugins.dll",
 "configSection": "graphConnectorMocks"
}

And, a bit below, the following configuration section:

1
2
3


"graphConnectorMocks": {
 "mocksFile": ".devproxy/graph-connector-mocks.json"
}

Dev Proxy supports the concept of plugins: instead of making the tool too complex, by supporting all the possible scenarios, Dev Proxy itself is able only to intercept API requests. Any other feature is delegated to a plugin. In this case, the default Graph Connector project is configured to use the GraphMockResponsePlugin, which is a plugin that allows you to mock the responses of the Microsoft Graph. The configuration section graphConnectorMocks is used to define the path to the file that contains the mocks. In the default configuration, this file is stored in the .devproxy folder of the project.

However, for our scenario, we don’t need to mock the responses: we want to interact with the real Microsoft Graph APIs, but we want to intercept the requests and responses to understand what’s going on. As such, we can remove the GraphMockResponsePlugin from the list of enabled plugins or you can just set the enabled property to false.

Adding the DevTools plugin

NOTE: As of the time of writing (13th August 2024), the DevTools plugin included in the latest Dev Proxy stable version (0.19.1) contains a bug that will likely prevent it from working properly on Windows. If you need to use this feature right now, you will need to use as minimum Dev Proxy 0.20.0, which is available in the Beta channel or on GitHub. Once 0.20.0 will move to the stable channel, you won’t need to use this workaround anymore.

When I started to work on my Graph Connector + .NET Aspire project (a sample micro-services architecture to create and deploy a custom Graph Connector), I started to hit immediately an issue. If you remember what we have learned in the previous posts, creating a custom connector involves three operations with the Microsoft Graph:

Create a connection
Define the schema
Upload the external items

In my case, I was already stuck with the first operation, due to an error I never encountered before. The request was failing because the body of the request (which means the configuration of the connection) was malformed. Since I was using the Microsoft Graph .NET SDK (which wraps for me all the HTTP requests), I wasn’t really sure what was going on, since the exception details weren’t provided many insights.

Please welcome the DevTools plugin, which enables you to combine Dev Proxy with the DevTools included in all the Chromium-based browsers, like Edge or Chrome. Thanks to this plugin, you can inspect all the requests and responses captured by DevProxy directly in the browser, by using the Network tool. Let’s see how to set this up.

As first step, you must enable the plugin, by adding the following entry in your configuration file, inside the plugins section:

1
2
3
4
5
6


{
 "name": "DevToolsPlugin",
 "enabled": true,
 "pluginPath": "~appFolder/plugins/dev-proxy-plugins.dll",
 "configSection": "devTools"
}

Second, you must add a new configuration section, called devTools, where you can define which browser you want to use to inspect the requests:

1
2
3


"devTools": {
 "preferredBrowser": "Edge"
}

The supported options are:

Edge (if you’re using the stable version of Edge)
EdgeDev (if you’re using the Dev channel of Edge)
EdgeBeta (if you’re using the Beta channel of Edge)
Chrome

That’s it!

Using Dev Proxy with the DevTools plugin

Now you just need to open a terminal and type devproxy. If you have made the configuration changes in a specific file in your project instead of using the global configuration file, remember to start it from the same folder where your custom devproxyrc.json file is stored. When you launch it for the first time, you might notice the following error:

This error happens when an instance of the browser is already running, so you will need to close it before Dev Proxy can launch the DevTools. This is required because, in some operating systems and scenarios, opening up a new instance of the browser when there’s already one running will lead to the DevTools not starting up properly. This error is very common, especially if you’re using Edge on Windows. The browser engine, in fact, is used to power up many experiences, so it’s very likely that you will have multiple instances already running (you can see how, in the image, the list of process IDs is quite long).

If you want to kill all the instances easily, on Windows you can just open a PowerShell terminal and run the following command:

1

Get-Process -Name "msedge" | Stop-Process -Force

If you’re on MacOS, instead, you can use the killall command:

1

killall -9 msedge

If you’re using Chrome, just replace msedge with chrome.

Once you have terminated all the instances, you can run devproxy again. This time, you should see your favorite browser starting up with the DevTools already opened. Now click on the Network tab and start to interact with your custom Graph Connector. For example, if you’re using a project created with create-graph-connector, you can try to create a connection by using the following command:

1

npm run start:createConnection:proxy

You will see all the requests and responses captured by Dev Proxy, and you can inspect them to understand what’s going on:

In the Payload section, you can see the exact JSON that has been sent to the API. In the Response section, instead, you can see the response, which includes detailed information about any error that might have happened. In my case, for example, thanks to this tool, I was able to find out that the ID I was using for the connector wasn’t valid, and that’s why the request was failing as malformed.

Wrapping up

In this post, we have seen how you can use DevProxy to intercept and inspect the requests and responses of your custom Graph Connector. Thanks to the DevTools plugin, you can actually see the network request details, giving you the ability to see exactly the JSON that is sent to the Microsoft Graph and the response that is returned back. This is super useful when you’re developing a custom Graph Connector, especially if you aren’t calling the APIs manually but you’re using a wrapper like the Microsoft Graph SDK for .NET or TypeScript, which tends to hide the inner communication with the Microsoft Graph.

If you want to test what we have learned in this post and you don’t have a custom Graph Connector, you can reuse the following sample I’ve published on GitHub, which ingests into the tenant the posts published on this blog.

Happy coding!!

Using Semantic Kernel to create multi-agent scenarios

Thu, 08 Aug 2024 00:00:00 +0000

If you have followed my posts about Semantic Kernel and you are familiar with the AI terminology, you should have realized by now that basically every sample we built so far could be identified as a single-agent scenario. In the AI ecosystem, in fact, we call agents intelligent actors that are able to perform a task, given a set of instructions and, optionally, tools that they can use to perform it. Semantic Kernel makes really easy to build such an agent: thanks to features like plugins, prompt functions and function calling, we can easily let an LLM to figure out the best plan to perform a task and return the result to us.

What if, however, the task is too complex to be performed by a single agent? What if, in order to full fill the task, you need the LLM to interpret different roles to achieve it? Think, for example, to software development. When you build a project, you need a PM that fleshes out the specifications; you need an architect that designs the architecture; you need a developer that writes the code; you need an IT expert that deploys the needed infrastructure. Let’s say now we want to use an LLM to implement this scenario. Since LLMs have been trained with a huge amount of knowledge, technically you could try to write a prompt that instructs the LLM to play all these different roles and come up with a solution. Practically, however, this would be a nightmare: the prompt would be too complex, and the LLM would struggle to understand it.

This is why, in the AI ecosystem, many frameworks are introducing the concept of multi-agents. Instead of trying to accomplish a task with a single agent, you can split the task into multiple sub-tasks and assign each sub-task to a different agent. Each agent will then perform its task and communicate with the other agents to achieve the final goal. This way, the complexity of the task is split into smaller, more manageable tasks, and the agents can focus on their specific role. Additionally, each agent can be more carefully specialized, which simplifies the orchestration for the LLM. For example, if you have a plugin to generate code, you’ll assign it only to the agent who plays the role of a developer.

In this post, we’ll take a look at how to create multi-agent scenarios in Semantic Kernel. This feature is still experimental, and requires the usage of a preview NuGet package, but it works! We’re going to use to create a travel agency scenario: we want to visit a city and different agents will help us to plan the trip. Let’s start!

Setting up the project

Setting up the project is no different than what we did in the past. For this sample, we’re going to create a standard .NET Console application and we’re going to install the following NuGet packages:

Microsoft.SemanticKernel
Microsoft.SemanticKernel.Agents.Core. This one is in marked as prerelease so, if you’re installing the package using the NuGet Package Manager, you need to check the “Include prerelease” checkbox.

Then, you’ll need to initialize the kernel using the OpenAI or Azure OpenAI services. In my case, I’m going to use the Chat Completion APIs from Azure OpenAI:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


var configuration = new ConfigurationBuilder()
 .AddUserSecrets<Program>()
 .Build();

string apiKey = configuration["AzureOpenAI:ApiKey"];
string deploymentName = configuration["AzureOpenAI:DeploymentName"];
string endpoint = configuration["AzureOpenAI:Endpoint"];

var kernel = Kernel.CreateBuilder()
 .AddAzureOpenAIChatCompletion(deploymentName, endpoint, apiKey)
 .Build();

The Azure OpenAI configuration, in my case, is stored in a user secret file.

The last step is to suppress all the warnings that the compiler will raise because the Microsoft.SemanticKernel.Agents.Core package is still in preview. The easiest way to do that is to open the project file and add the following line:

1

 <NoWarn>$(NoWarn);CS8618,IDE0009,CA1051,CA1050,CA1707,CA1054,CA2007,VSTHRD111,CS1591,RCS1110,RCS1243,CA5394,SKEXP0001,SKEXP0010,SKEXP0020,SKEXP0040,SKEXP0050,SKEXP0060,SKEXP0070,SKEXP0101</NoWarn>

Now that the project is set up, let’s create the agents!

Creating the agents

Let’s explain in a more detailed way the scenario that we’re going to build. We’re going to envision a travel agency, which has the following employees:

A travel expert, who has experience in finding the best hotels, restaurants, and attractions in a city.
A flight expert, who has experience in finding the best travel options via plane.
A travel manager, who reviews the entire plan and makes sure that everything is in order.

Let’s start by defining these three roles. You’ll realize that the most complex part of defining an agent isn’t really writing the code, but rather writing a good prompt that instructs the LLM to play the role. Let’s start with the travel expert:

The travel expert

Let’s define the instructions for the travel expert:

1
2
3
4
5
6
7
8
9


const string travelAgentName = "TravelAgent";
string travelAgentInstructions = """
 You are a travel agent and you help users who wants to make a trip to visit a city.
 The goal is to create a plan to visit a city based on the user preferences and budget.
 You don't have expertise on travel plans, so you can only suggest hotels, restaurants and places to see. You can't suggest traveling options like flights or trains.
 You're laser focused on the goal at hand.
 Once you have generated a plan, don't ask the user for feedback or further suggestions. Stick with it.
 Don't waste time with chit chat. Don't say goodbye and don't wish the user a good trip.
 """;

The instructions are pretty detailed and they explain to the LLM what the travel expert can do and what it can’t do. The travel expert can suggest hotels, restaurants and places to see, but it can’t suggest traveling options. It’s also laser focused on the goal and it doesn’t waste time with chit chat. Providing these details is important, because LLMs can be very chatty and they can easily go off topic. With these instructions, we make sure that the travel agent will stick on suggesting hotels, restaurants and places to see, leaving the travel organization to another expert. Notice that we have also defined the name of the agent. This is important because we’ll use this name to refer to the agent in the orchestration phase.

Now we can use the Semantic Kernel APIs included Microsoft.SemanticKernel.Agents.Chat namespace to create the agent:

1
2
3
4
5
6


ChatCompletionAgent travelAgent = new ChatCompletionAgent
{
 Name = travelAgentName,
 Instructions = travelAgentInstructions,
 Kernel = kernel
};

We use the ChatCompletionAgent class and we supply the name of the agent, the instructions we have defined and a reference to the kernel.

Now let’s move to the flight expert.

The flight expert

The code is the same we have seen before, we just change the name of the agent and the set of instructions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


const string flightExpertName = "FlightExpert";
string flightExpertInstructions = """
 Your are an expert in flight travel and you are specialized in organizing flight trips by identifying the best flight options for your clients.
 Your goal is to create a flight plan to reach a city based on the user preferences and budget.
 You don't have experience on any other travel options, so you can only suggest flight options.
 You're laser focused on the goal at hand.
 You can provide plans only about flights. Do not include plans around lodging, meals or sightseeing.
 Once you have generated a flight plan, don't ask the user for feedback or further suggestions. Stick with it.
 Don't waste time with chit chat.
 Don't say goodbye and don't wish the user a good trip.
 """

Similarly to what we did with the travel expert, we create a detailed set of instructions, so that the agent will stick in creating flight plans and it won’t try to give suggestions about other related topics, like hotels or restaurants, or other travel options, like trains.

Then we create a new agent out of the instructions:

1
2
3
4
5
6


ChatCompletionAgent flightAgent = new ChatCompletionAgent
{
 Name = flightExpertName,
 Instructions = flightExpertInstructions,
 Kernel = kernel
};

Let’s move to the final agent of our agency: the travel manager.

The travel manager

The travel manager is the agent that will validate the work of the other agents. It will receive the plans from the travel expert and the flight expert and it will review them to make sure that everything is in order. Let’s define the instructions:

1
2
3
4
5
6
7
8


string travelNamangerName = "TravelManager";
string travelManagerInstructions = """
 You are a travel manager and your goal is to validate a given trip plan.
 You must make sure that the plan includes all the necessary details: transportation, lodging, meals and sightseeing.
 If one of these details is missing, the plan is not good.
 If the plan is good, recap the entire plan into a Markdown table and say "the plan is approved".
 If not, write a paragraph to explain why it's not good and then provide an improved plan.
 """;

The travel manager can lead the conversation to two different paths: if the plan is good, it’s going to recap it with a table and it’s going to say “the plan is approved”. As you’re going to see in a bit, we’re going to use this sentence as a signal that the conversation is completed. If the plan can be improved, instead, it’s going to create a new one.

Let’s create the agent, as usual:

1
2
3
4
5
6


 ChatCompletionAgent travelManager = new ChatCompletionAgent
 {
 Name = travelManagerName,
 Instructions = travelManagerInstructions,
 Kernel = kernel
 };

Now we can start to enter into the orchestration phase.

Defining the terminate logic

When you build a multi-agent scenario, you need to define a termination logic. This logic is used to determine when the conversation is completed and the chat between the agents must end. Semantic Kernel provides a few classes to support different scenarios. One of the most powerful, which we’re going to use today, it’s the KernelFunctionTerminationStrategy, which enables to define the logic with a prompt, wrapped by the KernelFunction class.

Let’s see how to define the termination logic for our scenario:

1
2
3
4
5
6
7
8
9


KernelFunction terminateFunction = KernelFunctionFactory.CreateFromPrompt(
 $$$"""
 Determine if the travel plan has been approved. If so, respond with a single word: yes.

 History:

 {{$history}}
 """
 );

We use the KernelFactionFactory.CreateFromPrompt() method to create a new KernelFunction from a prompt, which describes our logic. Remember that, when we have set up the instructions for the travel manager, we said that it must use the sentence “the plan is approved” when the plan is good enough? As such, we use the prompt to determine if the plan has been approved and, in this case, to respond with the word “yes”. We also supply a parameter called history, which is going to incorporate all the chat history so that the agent has the full context.

Defining the selection logic

The next step is to define the selection logic, which means how the orchestration is going to pick up who is the next agent in the conversation. Semantic Kernel includes a few classes as well to manage this scenario. The simplest one is called SequentialSelectionStrategy, which is simply going to execute the agents one after the other. In our case, however, we need a more complex strategy, since agents might need to go back and forth to complete the task. As such, also in this case we’re going to use a strategy based on a prompt, by leveraging the KernelFunctionSelectionStrategy class. Let’s take a look at our prompt:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


 KernelFunction selectionFunction = KernelFunctionFactory.CreateFromPrompt(
 $$$"""
 Your job is to determine which participant takes the next turn in a conversation according to the action of the most recent participant.
 State only the name of the participant to take the next turn.

 Choose only from these participants:
 - {{{travelManagerName}}}
 - {{{travelAgentName}}}
 - {{{flightExpertName}}}

 Always follow these steps when selecting the next participant:
 1) After user input, it is {{{travelAgentName}}}'s turn.
 2) After {{{travelAgentName}}} replies, it's {{{flightExpertName}}}'s turn.
 3) After {{{flightExpertName}}} replies, it's {{{travelManagerName}}}'s turn to review and approve the plan.
 4) If the plan is approved, the conversation ends.
 5) If the plan isn't approved, it's {{{travelAgent}}}'s turn again.

 History:
 {{$history}}
 """
 );

The prompt is pretty detailed and it explains to the orchestration how to pick up the next agent in the conversation. As you can see, this logic is a bit more complex thant the termination one. To avoid as much as possible hallucinations, we provide a set of very strict information, leveraging the name of the agents that we have previously defined and incorporating them using the {{{ }}} syntax.

First, we list who are the available participants. Then we define the following order of interaction:

After the user has provided information about the trip, it’s the travel agent’s turn to create a trip plan.
Due to the instructions we have provided, the trip plan will miss the travel options. As such, we’re going to pass the conversation to the flight expert, who will generate a flight plan.
Then it’s travel manager’s turn, who is going to review the plan and approve it if it’s good enough. If the plan isn’t good, the conversation will go back to the travel agent, who will generate a new plan.

Putting all together

Now we have everything we need to kick off the conversation:

Our agents
A termination strategy
A selection strategy

A conversation in Semantic Kernel is represented by the AgentGroupChat class. Let’s see how to initialize it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


AgentGroupChat chat = new(travelManager, travelAgent, flightAgent)
{
 ExecutionSettings = new()
 {
 TerminationStrategy = new KernelFunctionTerminationStrategy(terminateFunction, kernel)
 {
 Agents = [travelManager],
 ResultParser = (result) => result.GetValue<string>()?.Contains("yes", StringComparison.OrdinalIgnoreCase) ?? false,
 HistoryVariableName = "history",
 MaximumIterations = 10
 },
 SelectionStrategy = new KernelFunctionSelectionStrategy(selectionFunction, kernel)
 {
 AgentsVariableName = "agents",
 HistoryVariableName = "history"
 }
 }
};

First, in the constructor, we pass the list of agents that are going to participate in the conversation. Then we set up the execution settings, where we define the termination strategy and the selection strategy. As we said, we’re going to use the KernelFunctionTerminationStrategy and the KernelFunctionSelectionStrategy classes. However, we need provide more than the prompt that we have created to describe the strategy:

For the termination strategy, we need to provide:

In the constructor, the KernelFunction object with the termination prompt we have previously created and a reference to the kernel.
The agents that are in charge of the strategy. In this case, it’s the travel manager, since it’s the one that can approve the plan, leading to the conclusion of the conversation.
The signal that the conversation is completed. In this case, we’re going to check if the result contains the word yes (remember that the termination prompt instructs the LLM to say “yes” if the plan has been approved).
The name of the variable which holds the history of the conversation, which in our case is history.
The maximum number of iterations. This is a safety measure to avoid infinite loops, in case the termination strategy isn’t met before.

For the selection strategy, instead, we must provide:

Also in this case, the KernelFunction object with the prompt that represents our selection strategy that we have previously created and a reference to the kernel.
The name of the variables which contains the list of agents, which is agents.
The name of the variable which contains the history of the conversation, which is history.

Kicking off the conversation

Now that we have everything in place, we can start the conversation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


string prompt = "I live in Como, Italy and I would like to visit Paris. I'm on a budget, I want to travel by plane and I would like to stay for maximum 3 days. Please craft a trip plan for me";

chat.AddChatMessage(new ChatMessageContent(AuthorRole.User, prompt));
await foreach (var content in chat.InvokeAsync())
{
 Console.WriteLine();
 Console.WriteLine($"# {content.Role} - {content.AuthorName ?? "*"}: '{content.Content}'");
 Console.WriteLine();
}

Console.WriteLine($"# IS COMPLETE: {chat.IsComplete}");

First, we define the prompt that we want to submit as a user. In this case, we state that we live in Italy and we want to visit Paris. We also provide some information about the budget, the travel preferences and the duration of the trip. Then we wrap the prompt into a ChatMessageContent object and we add it to the conversation using the AddChatMessage() method offered by the AgentGroupChat class. The results are returned by calling the InvokeAsync() method, which returns an asynchronous enumerator, so we can use a foreach loop prefixed by the await keyword to iterate over them. The user prompt will kick off the conversation according to our selection strategy and so, each time we call the InvokeAsync() method, we’ll kick off the next agent in the conversation. For each generated message, we print on the console:

The role of the user that generated the message (the user, the assistant, etc.)
The name of the agent
The content of the message

Once the conversation is completed because the termination strategy has been met, the InvokeAsync() method won’t return any more result and the IsComplete property of the AgentGroupChat object will be set to true.

Now you can run the code! The output will be quite long, so let me split into multiple sections so that you can see more easily the different parts of the conversation. The first agent to intervene is the travel agent, which generates a 3-day plan that includes the hotels, restaurants and attractions in Paris.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62



# assistant - TravelAgent: '### Day 1:

**Morning:**
- **Arrival in Paris**
- **Check-in at Hotel des Nations St-Germain**
 - Budget-friendly hotel located in the Latin Quarter

**Afternoon:**
- **Lunch at L'As du Fallafel**
 - Affordable and delicious falafel in the Marais neighborhood
- **Visit Notre-Dame Cathedral**
 - Iconic Gothic cathedral, free to enter

**Evening:**
- **Walk along the Seine River**
- **Dinner at Bouillon Chartier**
 - Traditional French cuisine at budget prices

### Day 2:

**Morning:**
- **Breakfast at Breizh Café**
 - Affordable and highly-rated crêperie
- **Visit the Louvre Museum**
 - Free admission on the first Saturday of each month after 6 PM, otherwise consider getting a Paris Museum Pass if you plan on visiting multiple museums

**Afternoon:**
- **Lunch at Café de l'Industrie**
 - Economical French brasserie
- **Explore Montmartre**
 - Visit Sacré-Cour Basilica (free entry)

**Evening:**
- **Dinner at Le Relais de l'Entrecôte**
 - Budget-friendly set menu specializing in steak-frites
- **Stroll in the Tuileries Garden**

### Day 3:

**Morning:**
- **Breakfast at Le Pain Quotidien**
 - Budget-friendly breakfast spot
- **Visit Musée d'Orsay**
 - Affordable entry, free on the first Sunday of each month

**Afternoon:**
- **Lunch at Le Petit Cler**
 - Affordable bistro near the Eiffel Tower
- **Visit the Eiffel Tower**
 - Consider going up only to the second floor for a cheaper ticket

**Evening:**
- **Dinner at Les Philosophes**
 - Reasonable rates for classic Parisian cuisine
- **Walk through Champs-Élysées and see the Arc de Triomphe** (optional, free to walk around)

**Additional Tips:**
- **Public Transportation:** Buy a 3-day Paris Visite travel pass for unlimited rides on public transport.
- **Picnics:** Consider having picnics with items from local bakeries and grocery stores to save money on meals.

Enjoy Paris!'

Then, the next actor in the conversation is the flight expert, who generates a flight plan to reach Paris:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


# assistant - FlightExpert: '### Flight Plan from Como, Italy to Paris, France

#### Outbound Flight

**Date:** October 20, 2023
- **Departure:** Milan Malpensa Airport (MXP), 10:00 AM
- **Arrival:** Paris Charles de Gaulle Airport (CDG), 11:30 AM
- **Airline:** EasyJet
- **Flight Number:** U2 1234
- **Duration:** 1 hour 30 minutes
- **Price:** ?50 (including taxes and fees)

#### Return Flight

**Date:** October 23, 2023
- **Departure:** Paris Charles de Gaulle Airport (CDG), 8:00 PM
- **Arrival:** Milan Malpensa Airport (MXP), 9:30 PM
- **Airline:** EasyJet
- **Flight Number:** U2 5678
- **Duration:** 1 hour 30 minutes
- **Price:** ?50 (including taxes and fees)

#### Total Cost
- Total Flight Cost: ?100

**Note:** To get to Milan Malpensa Airport from Como, consider taking a shuttle bus or train to keep within budget. Budget airlines like EasyJet typically offer the most economical fares but check-in luggage may incur additional fees - consider traveling with hand luggage only to minimize costs.

Enjoy your trip to Paris!'

Finally, the travel manager will review the plan, provide a recap with a table and approve it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90


# assistant - TravelManager: 'Your trip plan to Paris from Como by plane, with a stay of 3 days, includes all necessary details: transportation, lodging, meals, and sightseeing.

### Day 1:

**Morning:**
- **Transportation:**
 - Depart from Milan Malpensa Airport (MXP) to Paris Charles de Gaulle Airport (CDG) via EasyJet
 - Estimated total travel time: ~2 hours
- **Lodging:** Check-in at the **Hotel des Nations St-Germain** (Budget-friendly hotel)

**Afternoon:**
- **Meals:**
 - Lunch at **L'As du Fallafel** (Affordable and delicious)
- **Sightseeing:**
 - **Visit Notre-Dame Cathedral**

**Evening:**
- **Meals:**
 - Dinner at **Bouillon Chartier** (Traditional and budget-friendly)
- **Sightseeing:**
 - **Walk along the Seine River**

### Day 2:

**Morning:**
- **Meals:**
 - Breakfast at **Breizh Café** (Affordable crêperie)
- **Sightseeing:**
 - **Visit the Louvre Museum**

**Afternoon:**
- **Meals:**
 - Lunch at **Café de l'Industrie**
- **Sightseeing:**
 - **Explore Montmartre** (Visit Sacré-Cour Basilica)

**Evening:**
- **Meals:**
 - Dinner at **Le Relais de l'Entrecôte** (Budget-friendly steak-frites)
- **Sightseeing:**
 - **Stroll in the Tuileries Garden**

### Day 3:

**Morning:**
- **Meals:**
 - Breakfast at **Le Pain Quotidien**
- **Sightseeing:**
 - **Visit Musée d'Orsay**

**Afternoon:**
- **Meals:**
 - Lunch at **Le Petit Cler**
- **Sightseeing:**
 - **Visit the Eiffel Tower** (Consider second floor for a cheaper ticket)

**Evening:**
- **Meals:**
 - Dinner at **Les Philosophes** (Economical classic Parisian cuisine)
- **Sightseeing:**
 - **Walk through Champs-Élysées and see the Arc de Triomphe**

**Additional Tips:**
- **Public Transportation:** Buy a 3-day Paris Visite travel pass.
- **Picnics:** For budget-friendly meals.

## Plan Recap

| Day | Time | Activity |
|-----------|----------|---------------------------------------------------------|
| Day 1 | Morning | Transport from Milan to Paris via EasyJet flight |
| | | Check-in at Hotel des Nations St-Germain |
| | Afternoon| Lunch at L'As du Fallafel |
| | | Visit Notre-Dame Cathedral |
| | Evening | Dinner at Bouillon Chartier |
| | | Walk along the Seine River |
| Day 2 | Morning | Breakfast at Breizh Café |
| | | Visit Louvre Museum |
| | Afternoon| Lunch at Café de l'Industrie |
| | | Explore Montmartre and Sacré-Cour Basilica |
| | Evening | Dinner at Le Relais de l'Entrecôte |
| | | Stroll in the Tuileries Garden |
| Day 3 | Morning | Breakfast at Le Pain Quotidien |
| | | Visit Musée d'Orsay |
| | Afternoon| Lunch at Le Petit Cler |
| | | Visit the Eiffel Tower |
| | Evening | Dinner at Les Philosophes |
| | | Walk through Champs-Élysées and see Arc de Triomphe |

"The plan is approved." Enjoy your trip to Paris!'

And now we’re ready to fly to Paris :-)

A different conversation

If you’re going to run the code we’ve written multiple times, you’ll notice that the travel manager will almost always just approve the plan. Since the travel expert and the flight expert have worked to put together all the details, the plan is going to be good enough most of the time.

What if we want to test what happens in case, instead, the plan is not good? We can do that by changing our orchestration and removing the flight expert from the equation.

First, we must change the selection logic, since the flight expert won’t be involved in the conversation anymore:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


KernelFunction selectionFunction = KernelFunctionFactory.CreateFromPrompt(
 $$$"""
 Your job is to determine which participant takes the next turn in a conversation according to the action of the most recent participant.
 State only the name of the participant to take the next turn.

 Choose only from these participants:
 - {{{travelManagerName}}}
 - {{{travelAgentName}}}

 Always follow these four when selecting the next participant:
 1) After user input, it is {{{travelAgentName}}}'s turn.
 2) After {{{travelAgentName}}} replies, it's {{{travelManagerName}}}'s turn to review the plan.
 3) If the plan is approved, the conversation ends.
 4) If the plan isn't approved, it's {{{travelAgent}}}'s turn again.

 History:
 {{$history}}
 """
 );

Now we list only as participants the travel manager and the travel agent. Then, we tweak the steps to follow to decide who is the next participant in the conversation: after the travel agent has provided the plan, the travel manager will immediately review it.

Finally, when we initialize the AgentGroupChat object, we must keep in the collection only the two agents:

1
2
3
4


AgentGroupChat chat = new(travelManager, travelAgent)
{
 // the initialization code is unchanged
};

Now you can run the code, passing as input the same prompt we use to describe the city we want to visit and our budget and travel preferences. You should see the following conversation happening:

The travel agent will generate a plan that, given the instructions we provided, will miss the travel options.
The travel manager will report that the plan isn’t good enough because travel options are missing, so it will generate a new one which includes them.
The travel manager will finalize and approve it.

What’s next?

You can use this sample project as a starting point to experiment with other scenarios and add even more complexity. In the final sample published on GitHub, for example, you’ll find that I’ve added another agent that can suggest train options to reach the destination. To support this new agent, you’ll find a slightly tweaked selection logic:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


KernelFunction selectionFunction = KernelFunctionFactory.CreateFromPrompt(
 $$$"""
 Your job is to determine which participant takes the next turn in a conversation according to the action of the most recent participant.
 State only the name of the participant to take the next turn.

 Choose only from these participants:
 - {{{travelManagerName}}}
 - {{{travelAgentName}}}
 - {{{flightExpertName}}}
 - {{{trainExpertName}}}

 Always follow these four when selecting the next participant:
 1) After user input, it is {{{travelAgentName}}}'s turn.
 2) After {{{travelAgentName}}} replies, it's {{{flightExpertName}}}'s turn to generate a flight plan for the given trip.
 - If the user prefers to travel by train, it's {{{trainExpertName}}}'s turn.
 - If the user prefers to travel by flight, it's {{{flightExpertName}}}'s turn.

 3) Finally, it's {{{travelManagerName}}}'s turn to review and approve the plan.
 4) If the plan is approved, the conversation ends.
 5) If the plan isn't approved, it's {{{travelAgent}}}'s turn again.

 History:
 {{$history}}
 """
 );

As you can see, other than adding the train expert to the list of participants, we have also added a new step in the selection logic. After the travel agent has provided the plan, the selection logic will decide if the next participant is the flight expert or the train expert, based on the user preferences. This way, you can create a more complex scenario where the conversation can take different paths based on the user input. To test this, try to specify your preference to travel by train in the user prompt:

1
2
3
4
5


string prompt = """
I live in Como, Italy and I would like to visit Paris.
I'm on a budget, I want to travel by train and I would like to stay for maximum 3 days.
Please craft a trip plan for me
""";

If you run the conversation, you’ll notice that this time the flight expert will be skipped and the conversation will flow from the travel agent to the train expert to the travel manager for approval.

Wrapping up

In this post, we have tasted the potential of AI agents and multi-agents scenarios. It may look a bit like role-playing, but it opens up a new world of possibilities in the AI ecosystem. By splitting a complex task into multiple sub-tasks and assigning each sub-task to a different agent, you are much more likely to achieve the task you want to perform in a more efficient and reliable way. This is especially true when you’re working with LLMs, which can easily get lost in complex prompts. By using multi-agents scenarios, you can provide a more focused and specialized set of instructions to each agent, which will make the orchestration much easier and the results more reliable. Semantic Kernel greatly helps in building these scenarios, by providing all the infrastructure you need to create agents, orchestrate them, defining the termination and selection strategies, and much more. Additionally, we haven’t touched this scenario in this post, but don’t forget that all the Semantic Kernel features we have learned in the past posts are still available in multi-agent scenarios. This means that you can still use plugins, prompt functions, and function calling to create more complex and powerful agents.

You can find the sample I’ve created in this post on GitHub.

A smarter way to manage prompts with Semantic Kernel using Prompty

Tue, 23 Jul 2024 00:00:00 +0000

One of the needs that often comes up when you’re building an AI powered application is a smart way to manage a prompt library. Even if, due to the nature of LLMs, chat is the predominant interfaced offered by AI applications, there are many scenarios where AI stays behind the scenes and augments the capabilities of the application. Think, for example, to Copilot for Microsoft 365. Many experiences are chat based, but there are also many experiences which are triggered directly in the user interface. In Outlook, when you want to get a summary of a thread, you click on the Summary By Copilot banner at the top of the conversation; in Word, when you want to convert a content into a table, you select it, right click on it and you choose Copilot -> Rewrite as a table. In all these scenarios, the user doesn’t see “the magic behind” but, in the background, Copilot is just executing a prompt.

In all these scenarios, you can’t just rely on having a prompt hard coded in the application. You need a way to manage it, test it, easily change it without rebuilding the application. On this blog, we have learned a way to do that by using one of the features offered by Semantic Kernel: prompt functions. However, recently Microsoft has introduced a new solution to standardize prompts and its execution into a single asset that we can use to improve the management of our prompts in our applications. It’s called Prompty and it’s part of PromptFlow, a suite of development tools from Microsoft to streamline the end-to-end development of LLM based applications.

Specifically, Prompty relies on PromptFlow to give you the ability to test prompts without needing to write the code to execute it or to run the full application.

Let’s take a deeper look!

Start with Prompty

The easiest way to start with Prompty is by installing the dedicated Visual Studio Code extension. Once you have installed it, you will get the ability to right click on Explorer and choose New prompty. This will create a new file in the folder called basic.prompty, which is a YAML file that contains an example of a prompt.

As you can see, the file is more complex than the prompt functions we have seen in the past and it allows much more flexibility. Let’s take a look at the content of the file and, along the way, we’re going to customize it.

At the top of the file, you will see a section wrapped into three dashes (---). This is the configuration section which, however, will cover later, you’ll understand why shortly. Let’s start with what we find below, which is the actual prompt. You can immediately notice how you can express a full conversation using the Chat Completion API style:

We can define multiple actors in the conversation (system prompt, user prompt, etc.)
We can provide variables that, at runtime, will be replaced with values coming from the application.

This is how my section looks like for my prompt:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16



system:
You are an AI assistant who helps people find information. As the assistant, 
you answer questions briefly, succinctly, and in a personable manner using 
markdown and even add some personal flair with appropriate emojis.

# Customer
You are helping {{firstName}} to find answers to their questions.
Use their name to address them in your responses.

# Context
Use the following context to provide a more personalized response to {{firstName}}:
{{context}}

user:
{{question}}

First, we can notice how we are basically setting up a whole agent, since we are providing a system prompt (to instruct the LLM on how to respond) and a user prompt (which is the message we’ll get from the application).

Second, despite the prompt is generic (you are an AI assistant who helps people find information), we are using variables to make it more dynamic. We are instructing the LLM to always address the user with their name and we are providing some information the LLM can use to generate an answer (think of it like a super simplified RAG implementation).

All these three variables (name of the user, context and question) are supplied with a parameter, using the {{variable}} syntax. When we’re going to invoke this prompt from our application, we’ll have to provide the values for these variables.

The sample section

At the beginning of the post, we have said that one of the benefits of Prompty is that it allows to test prompts without needing to write the code to execute it or to run the full application. How can we do that? That’s where the sample section comes into play. In this section, which belongs to the Prompty configuration at the top, you can provide the values for the variables you have defined in the prompt and they will be used when we execute the prompt using the built-in tools provided by Visual Studio Code.

Let’s take a look at my sample section:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


sample:
 firstName: Matteo
 context: >
 Contoso Electronics is a leader in the aerospace industry, providing advanced electronic
 components for both commercial and military aircraft. We specialize in creating cutting edge systems that are both reliable and efficient. Our mission is to provide the highest
 quality aircraft components to our customers, while maintaining a commitment to safety
 and excellence. We are proud to have built a strong reputation in the aerospace industry
 and strive to continually improve our products and services. Our experienced team of
 engineers and technicians are dedicated to providing the best products and services to our
 customers. With our commitment to excellence, we are sure to remain a leader in the aerospace industry for years to come
 question: Which type of srvices does Contoso Electronics provide?

As you can see, I’m providing the values for the variables I have defined in the prompt. When I’m going to execute the prompt, these values will be used to generate the response. Again, it’s very simplistic, but it’s a very basic RAG implementation: we are asking a question (the type of services offered by a fictional company called Contoso Electronics) and, in the context, we are providing the information that the LLM can use to answer it.

Testing the prompt

Now that we have completed the creation of the prompty file, we can actually test it without needing to write any code. There are multiple ways to do that, let’s start from the easiest one, which is running it. Before doing that, however, we must setup the AI service we want to use to execute the prompt. To do that, you must go in the Settings of Visual Studio Code and search for prompty with the internal search engine. You’ll see a section like the following one:

Click on Edit in settings.json below Model configurations to open the settings file, in which you will find a section named prompty.modelConfigurations. Here, you can configure one or more AI services, which could be based on Azure OpenAI or OpenAI. For example, here below is the configuration to use gpt-4o from my Azure OpenAI instance:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


 "prompty.modelConfigurations": [
 {
 "name": "gpt4-o",
 "type": "azure_openai",
 "api_version": "2024-02-15-preview",
 "azure_endpoint": "<endpoint>",
 "azure_deployment": "gpt-4o",
 "api_key": "<api-key>"
 },
 {
 "name": "gpt-3.5-turbo",
 "type": "openai",
 "api_key": "<api-key>",
 "organization": "<org-id>",
 "base_url": "<base-url>"
 }
 ],

As you can see, you can specify more than one configuration. They will all show up in the Visual Studio Code status bar at the bottom:

If you click on it, you will be able to choose one of the services that you have previously added in the prompt.modelConfigurations section.

Once you have selected the model you want to use, make sure to focus Visual Studio Code on the prompty file, then click on the play button at the top right of the file to execute the prompt:

If you have set up the AI service in the right way, Visual Studio Code will run the prompt against the LLM and will fill the variables with the values we have set in the sample section. You will see the response in the output panel:

1
2
3


Hey Matteo! Contoso Electronics provides advanced electronic components for both commercial and military aircraft.
Our services include designing and manufacturing cutting-edge systems that are reliable and efficient.
We pride ourselves on our commitment to safety, excellence, and continual improvement in our products and services. ✈️🔧

If you prefer, in the panel list, you will find also a panel called Prompty Output (Verbose), which will show you the entire API communication with the AI service and the JSON request and response.

Pretty cool, right? We have been able to test the quality of our prompt without needing to write any code. And, if we want to make any change, we can just edit the prompty file and run it again. For example, let’s slightly change the system prompt:

1
2
3
4
5


system:
You are an AI assistant who helps people find information. As the assistant, 
you answer questions briefly, succinctly, and in a personable manner using 
markdown and even add some personal flair with appropriate emojis.
Respond using the JSON format, by including the original question and the response.

We have added an instruction to return the response in JSON format, rather than in Markdown.

We can just hit Run again to see how the response is changing:

1
2
3
4
5
6


2024-07-24 11:45:15.179 [info] Calling https://semantickernel-gpt.openai.azure.com//openai/deployments/gpt-4o/chat/completions?api-version=2024-02-15-preview
2024-07-24 11:45:17.827 [info] ```json
{
 "question": "Which type of services does Contoso Electronics provide?",
 "response": "Hey Matteo! Contoso Electronics provides advanced electronic components for both commercial and military aircraft. Our services include designing and manufacturing cutting-edge systems that are reliable and efficient. We pride ourselves on our commitment to safety, excellence, and continual improvement in our products and services. ✈️🔧"
}

But that’s not enough! We can do more advanced testing thanks to Prompt Flow.

Let’s take a look!

Advanced testing with Prompt Flow

When you worked on the Prompty file, you might have noticed that, at the top, Visual Studio Code was showing up a few options at the top:

All of them are based on Prompt Flow, and the most interesting one is Open test chat page. The Test one, in fact, will simply execute the prompt, like we have done before but with the additional requirement of needing to setup Prompt Flow. Open test chat page, instead, will give us a chat interface that will allow us to do deeper experiments with the prompt.

Before using it, we must setup Prompt Flow however. This technology is based on Python, so the first step is to make sure you have a Python environment installed on your machine. If you don’t have it, you can quickly install it thanks to WinGet if you’re on Windows. Open a terminal and run the following command:

1

winget install Python.Python.3.11

Once you have Python installed, you can install Prompt Flow using the Python package manager, called pip:

1

pip install promptflow

When we use this approach to test our prompt, there’s an important difference: we aren’t executing anymore the prompt using Visual Studio Code (which directly calls the Azure OpenAI APIs), but using the Prompt Flow service. As such, we can’t leverage the model configuration we have previously created in the Visual Studio Code settings, but we must add a new section to our prompty file called model, like in the following example:

1
2
3
4
5
6
7
8
9


model:
 api: chat
 configuration:
 type: azure_openai
 azure_endpoint: <endpoint>
 azure_deployment: gpt-4o
 api_version: 2024-02-15-preview
 parameters:
 max_tokens: 3000

This is the meaning of the various parameters:

api: it’s the type of API you want to use. In our case, we use chat to use the chat completion service.
configuration is used to specify the service configuration with the following parameters:
- type: this is the AI service you want to use. In my case I’m using Azure OpenAI, so I set this value to azure_openai. Based on the type, the other parameters will change, since Azure OpenAI and OpenAI have different connection methods. * azure_endpoint: this is specific for Azure OpenAI and it’s the URL of the service you want to use. * azure_deployment: this is specific for Azure OpenAI and it’s the name of the deployment of the model you want to use. * api_version: it’s the version of the API you want to use.
parameters allows you to specify different parameters to customize the model interaction, like the temperature or (like in this case), the maximum number of tokens to use.

As you can see, unlike we did in the Visual Studio Code settings, we don’t have a way to specify the API key. This is because Prompt Flow requires to use the Microsoft Entra authentication in order to securely connect with Azure OpenAI. For this reason, you will need to have installed on your machine the Azure CLI, which enables a series of commands to work with Azure.

Once you have installed it, run the following command:

1

az login

You will be guided through a series of steps to authenticate with your Azure account and to choose the subscription in which you have deployed the Azure OpenAI service. Once you have gone through this process, you will need to assign a dedicated role to your account to be able to leverage the integrated Microsoft Entra authentication. Open the Azure portal and open the Azure OpenAI instance that you’re using. Click on Access Control (IAM) and then Add -> Add role assignment.

In the first step, in the Role tab, look for a role called Cognitive Services OpenAI User and select it. Then click Next.

In the Members tab, make sure that the option Assign access to is set to User, group or service principal and click on Select members. From there, search for your account and click on Select:

Click on Review + assign two times and wait for the operation to complete.

Now you should be good to go. To quickly test if you did everything properly, you can just click on the Test option at the top of the Prompty file. Visual Studio Code will launch the Prompt Flow service and will execute the prompt. You should get a similar result than the one we have seen before.

However, we said that the most interesting integration is the chat interface, so click on Open test chat page. This option will launch again the Prompt Flow service but, at the end, you should see your browser opening up on a local server that will show you a chat interface like the following one:

On the panel on the left, you can setup the input and outputs of the conversation:

Under Chat input/output field config, the most important parameter to set is Chat input. You must specify, among the parameters supported by your prompt, which is the one mapped with the user input. In my case, it’s the question variable.
Under Prompty inputs, you must assign a value to each of the parameters you have included in your prompt (except the one which you’re going to get from the chat input). In my case, I had to provide a value for context (the paragraph about the fictional company Contoso Electronics) and for firstName (the name of the user).

Now you can just chat using the chat interface on the left. Prompt Flow will send the prompt to the LLM and it will fill all the parameters with the values you have provided, either in the chat or in the Settings panel. Let’s ask, for example, What is Contoso Electronics?

You can notice how the instructions we have provided in the prompt are followed: the LLM is addressing the user with their name and is using the context to provide a more personalized response. If you want, you can click on View trace to see the inner details of the operations, like the raw API calls.

Using Prompty with Semantic Kernel

Now that we have tested our prompt, we are ready to use it in our application. Again, the Visual Studio Code extension simplifies the process. Right click on the Prompty file and you will see the option to generate code for various AI frameworks, including Semantic Kernel:

Click on Add Semantic Kernel code and you will get a new file with the following code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Connectors.OpenAI;

var deployment = "gpt-4o";
var endpoint = "https://semantickernel-gpt.openai.azure.com/";
var key = "aoai_key";
var kernel = Kernel.CreateBuilder()
 .AddAzureOpenAIChatCompletion(deployment, endpoint, key)
 .Build();

 // update the input below to match your prompty
 KernelArguments kernelArguments = new()
 {
 { "question", "what's my question?" },
 };

var prompty = kernel.CreateFunctionFromPromptyFile("basic.prompty");
var result = await prompty.InvokeAsync<string>(kernel, kernelArguments);
Console.WriteLine(result);

If you have already experience with Semantic Kernel, the code will be familiar to you. We are creating a new instance of the Kernel class and we are setting it up with our Azure OpenAI instance. However, we can notice a new method called CreateFunctionFromPromptyFile(), which will create a prompt function starting from a Prompty file. We just need to pass, as parameter, the relative path of the Prompty file we want to use. To use this feature, however, we must install a new NuGet package, which is currently in preview: Microsoft.SemanticKernel.Prompty. As such, in the NuGet Package Manager, you’ll need to enable the Include prerelease option to see it.

The code generated by Visual Studio Code requires an additional change. As you can see, through the KernelArguments collection, we are providing a value for only one of the three parameters used by the prompt, question. As such, we need to enhance the collection to add the missing ones, firstName and context. We also need to wrap the CreateFunctionFromPromptyFile() method in a pragma directive, since the feature is marked as experimental, otherwise the code won’t compile.

This is the final version of the code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


using Microsoft.Extensions.Configuration;
using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Connectors.OpenAI;

var configuration = new ConfigurationBuilder()
 .AddUserSecrets("9aea1645-00e5-48dc-b396-a39b7d6821ca")
 .Build();

string apiKey = configuration["AzureOpenAI:ApiKey"];
string deploymentName = configuration["AzureOpenAI:DeploymentName"];
string endpoint = configuration["AzureOpenAI:Endpoint"];

var kernel = Kernel.CreateBuilder()
 .AddAzureOpenAIChatCompletion(deploymentName, endpoint, apiKey)
 .Build();

KernelArguments kernelArguments = new()
 {
 { "question", "Which type of services does Contoso Electronics provide?" },
 { "firstName", "Matteo" },
 { "context", @"Contoso Electronics is a leader in the aerospace industry, providing advanced electronic components for both commercial and military aircraft.
 We specialize in creating cutting edge systems that are both reliable and efficient. Our mission is to provide the highest quality aircraft components to our customers,
 while maintaining a commitment to safety and excellence. We are proud to have built a strong reputation in the aerospace industry and strive to continually
 improve our products and services. Our experienced team of engineers and technicians are dedicated to providing the best products and services to our customers.
 With our commitment to excellence, we are sure to remain a leader in the aerospace industry for years to come" }
 };

#pragma warning disable SKEXP0040 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
var prompty = kernel.CreateFunctionFromPromptyFile("basic.prompty");
#pragma warning restore SKEXP0040 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
var result = await prompty.InvokeAsync<string>(kernel, kernelArguments);
Console.WriteLine(result);

If you run the application (in my case, it’s a .NET Console application), you will see a similar output than the one we have observed during our tests:

Wrapping up

In this post, we have seen a new way to manage prompts and their execution in our applications, thanks to Prompty, a new solution from Microsoft to standardize prompts and templates. Compared to prompt functions, with Prompty we can use a more powerful syntax, enable more complex scenarios and, most of all, test the quality of our prompt engineering skills right in Visual Studio Code, before writing any code.

Once you have validated your job, you can easily import the Prompty file into a Semantic Kernel based application. And if you are a Python developer, Visual Studio Code gets you covered, since you can quickly generate a code file for LangChain as well.

You can find a Semantic Kernel sample about Prompty in my catalog of samples on GitHub.

Happy coding!

Building a Graph Connector with the TypeScript project generator

Mon, 22 Jul 2024 00:00:00 +0000

Update on 31st July 2024

The workaround of installing the cross-env library to use Graph Connectors with Dev Proxy in a cross-platform way isn’t needed anymore. The latest version of the project (create-graph-connector 0.1.3) uses env-cmd to solve this problem. As such, that part has been removed from the original version of the post.

In another post on this blog we explored a few concepts around Graph Connectors: what they are, which is their value, why they are important for Copilot and, lastly, we focused on the permission model. However, we didn’t spend much time understanding how to implement a Graph Connector from scratch: we started from one of the existing samples, so we have briefly seen concepts like creating a connection, creating a schema and ingesting content, but we didn’t actually implement them. In this post, we’ll take a deeper look, instead, on how to create a Graph Connector, with the help of a new tool released by the Microsoft 365 Advocacy Team: the Graph Connector project generator.

The Graph Connector project generator

The Graph Connector project generator is a new tool developed by Waldek Mastkyarz, a Microsoft 365 Developer Advocate who you might know as one of the creators of the great DevProxy tool (we’ll talk about it a bit later in the post).

The Graph Connector project generator is a CLI tool that allows you to create a new Graph Connector project in a few seconds, with all the boilerplate code you need to start developing your connector. The version we’re going to use today is based on Node.js and TypeScript, so you’ll need a Node.js development environment set up on your machine.

The tool is available on npm, so you can install it with a simple command:

1

npm i create-graph-connector

Once installed, open a terminal and move to the folder where you want to create your Graph Connector project and run the following command:

1

npm create graph-connector

The project generator will start a wizard asking for a few information:

The name of the project, which will be used as name of the folder as well, so make sure to avoid using spaces or weird characters.
The Graph Connector name that will be displayed in Admin Center.
The Graph Connector description.
The connection id. It’s a string between 3 and 32 chars, used to identify the connection. You can’t use spaces and you should avoid using weird characters.
The template you want to use. For the purpose of this post, we’re going to use TypeScript (Basic)

At the end, you will have a Node.js project based on TypeScript, with the following structure:

Here are the key files we’re going to work on in the rest of the project:

src/config.ts: it contains the configuration of the connection and the schema of the data we’re going to import.
src/createConnection.ts: it contains the functions that call the Microsoft Graph APIs to create the connection and the schema.
src/graphClient.ts: we won’t actually need to modify this file, but it’s important to be aware of its existence. It contains the definition of a GraphClient object that the project uses to preform authenticated requests against the Microsoft Graph. It contains also the logic for supporting a proxy (which is needed if you want to use the DevProxy tool) and to manage the delay between the API call and the response returned by the Microsoft Graph.
src/loadContent.ts: it contains the functions that calls the Microsoft Graph APIs to ingest the external content into the Microsoft 365 tenant.
resultLayout.json: it contains the definition of the Adaptive Card that will be used to display the results of the connector.

My advice is to use Visual Studio Code to work on the project, as it has a great support for TypeScript and Node.js projects, and it will help you a lot in understanding the code and in debugging it.

For the purpose of this post, we’re going to create a Graph Connector that connects to RSS feed of my blog, so that all the posts I’ve published are ingested into the Microsoft 365 tenant. We’re going to use the RSS feed of this blog as the source of the data.

Let’s start!

Defining the connection and the schema.

The first step is to define the configuration of the connection and the schema of the data we need to ingest into the tenant. This is done inside the config.ts file. Let’s start with the connection.

The connection

To setup the connection, we don’t have to do much, since the project generator already created a basic configuration for us, using the various information we have provided in the wizard, like the connector’s identifier or the name, as you can see in the following snippet:

1
2
3
4
5


 connection: {
 id: 'devcantina',
 name: 'The Developers Cantina Connector',
 description: 'Imports data from the Developers Cantina blog'
 }

However, there’s an important setting we need to add, which is the urlToItemResolvers property. This property is enables the platform to detect when users share URLs from your external content with each other. It’s especially important when you use Graph Connectors with Copilot, since Copilot for Microsoft 365 has a higher likelihood of displaying content that has been shared with that user. In our case, since we’re talking about ingesting posts from this blog, the hosting platform is the blog itself. As such, we need to provide a way to reconstruct the url of the original article from the identifier of the post. This is what’s done in the following snippet:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


urlToItemResolvers: [
 {
 urlMatchInfo: {
 baseUrls: [
 'https://www.developerscantina.com'
 ],
 urlPattern: '/p/(?<slug>[^/]+)'
 },
 itemId: '{slug}',
 priority: 1
 } as ExternalConnectors.ItemIdResolver
 ]

If you take a look at various posts on this blog, you’ll notice that all of them follows the structure of https://www.developerscantina.com/p/slug, where slug is the identifier of the post (for example, for this post it’s graph-connectors-template). This is what we’re going to use to resolve the original item from the external content.

The schema

The second part of the config.ts file includes the definition of the schema of the data we’re going to ingest. Using JSON format, we simply describe which the properties are of our data and for which purposes they can be used by the Microsoft 365 tenant (like if it’s searchable, retrievable, etc.). The default schema in the template is a good starting point, since it include some critical properties like title, url and the urlIcon. We can add as many we need to match the structure of our external data.

You can review the meaning of all the properties in the official documentation. One of the most important ones is labels, which is used to match one of the properties of the external content with a well known Microsoft 365 property. These are called semantic labels and they greatly improved the search experience in Microsoft 365 for your data. Being Microsoft 365 properties, you will find in the list only generic ones, that are agnostic and that can be applied to almost any data source, like title, url, createdDateTime, etc.

For our scenarios, the properties included in the schema are almost good enough. We will just add a new one called description, which will include the content of the post, following the RSS structure. This is the final schema:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37


schema: {
 baseType: 'microsoft.graph.externalItem',
 // Add properties as needed
 properties: [
 {
 name: 'title',
 type: 'string',
 isQueryable: true,
 isSearchable: true,
 isRetrievable: true,
 labels: [
 'title'
 ]
 },
 {
 name: 'url',
 type: 'string',
 isRetrievable: true,
 labels: [
 'url'
 ]
 },
 {
 name: 'description',
 type: 'string',
 isRetrievable: true
 },
 {
 name: 'iconUrl',
 type: 'string',
 isRetrievable: true,
 labels: [
 'iconUrl'
 ]
 }
 ]
 }

As you can notice, description doesn’t have any semantic label, since we don’t have any Microsoft 365 property that can match it.

Deploy the connector and the schema

Creating the connection and uploading the schema is performed by the createConnection.ts file, which includes two functions for this purpose: createConnection() and createSchema(). Both of them are using the GraphClient object (defined in the graphClient.ts file) to call the Microsoft Graph APIs.

The createConnection() function creates the connection by calling the endpoint /external/connections and passing the connection definition from the config.ts file.

The createSchema() function creates the schema by calling the endpoint /external/connections/{connectionId}/schema and passing the schema definition from the config.ts file. In the case of our sample, the connectionId will be devcantina, as we have defined in the initial wizard.

From a development perspective, we don’t have to change anything in this file. The two functions are implemented, in fact, in a generic way, all the configurations are taken from the config.ts file.

Ingesting the content

Now that we have defined a connection and a schema, we are ready to actually get the list of posts published in the blog and upload them as external items in the Microsoft Graph. The logic for this is contained in the loadContent.ts file, which is also the most complex file of the solution and the one in which we’ll need to most of the changes.

Every data source is unique, with its own logic to retrieve the data. As such, this file contains the basic skeleton of the functions we need to implement, but then it’s up to us to fill the gaps with the actual logic to retrieve the data.

Let’s see them step by step.

The `extract()` function

This function is used to retrieve the list of items we want to ingest. For this purpose, the file includes the definition of a TypeScript interface called Document, which describes an item from our external source. The interface already contains some basic properties, but you can customize it as needed. For my purpose, I don’t need to add anything, since the interface already contains all the properties that describe a blog post:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


interface Document {
 // Document title
 title: string;
 // Document content. Can be plain-text or HTML
 content: string;
 // URL to the document in the external system
 url: string;
 // URL to the document icon. Required by Microsoft Copilot for Microsoft 365
 iconUrl: string;
}

Once we have defined the interface, the goal of the extract() function is to get the list of items from your external data source and convert them into Document objects. In our case, we’re going to use the RSS feed of the blog, which is a XML file that contains all the published posts. To simplify our code, we’re going to use the xml2js library to parse the XML file and extract the information we need.

As first step, open a terminal in the folder that contains the connector and run the following command:

1

npm install xml2js

Then, we must import the following object at the beginning of the file:

1

import { parseStringPromise } from 'xml2js';

Finally, we can implement the extract() method as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


async function extract(): Promise<Document[]> {
 const rssUrl = 'https://www.developerscantina.com/index.xml';
 const response = await fetch(rssUrl);
 const rssText = await response.text();
 const rssJson = await parseStringPromise(rssText);

 const documents: Document[] = rssJson.rss.channel[0].item.map((item: any) => ({
 title: item.title[0],
 content: item.description[0],
 url: item.link[0],
 iconUrl: item['media:thumbnail'] ? item['media:thumbnail'][0].$.url : ''
 }));

 return documents;
}

First, we have changed the signature of the method since we need to execute a few operations (like fetch()), which are asynchronous. As such, the function is marked with the async keyword and, instead of returning an array of Document objects (Document[]), it returns a Promise that will resolve to an array of Document objects (Promise<Document[]>).

Thanks to the xml2js library, it’s fairly easy to parse the XML file and extract the information we need. First we use the fetch() function to retrieve the RSS feed, we parse the response as text using the text() function and then we use the parseStringPromise() function to convert the text into a JSON object. Now we can use the JSON as a dynamic object and, for each post, create a Document object with the information we need: title, description, link and thumbnail.

The `getDocId()` function

The getDocId() function is used to generate a unique identifier for each item we’re going to ingest. This is important because the Microsoft Graph needs a way to uniquely identify each item, so that it can be updated or deleted if needed. Also in this case, we must provide our own logic, because each data source has its own way to identify an item.

In the beginning of the post, we have already said that, for our scenario, the id of a post is its slug. However, the RSS feed doesn’t contain this information, so we must parse the URL to retrieve this information:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


function getDocId(doc: Document): string {
 try {
 const url = new URL(doc.url);
 const pathSegments = url.pathname.split('/');
 const slug = pathSegments[pathSegments.length - 2];
 return slug;
 } catch (error) {
 console.error('Invalid URL:', doc.url);
 return '';
 }
}

In input, we get the usual Document object. We use the URL object to parse the URL and then we split the pathname of the URL to get the segments. The slug is the second to last segment, so we get it and return it.

The `transform()` function

The transform() function is used to convert the collection of Document objects into a collection of ExternalItem objects, which is the type used by the Microsoft Graph to identify an external item. An ExternalItem has the following properties:

id, which is the unique identifier of the item.
properties, which is a collection that contains all the properties of the item, as defined in the schema.
content, which is a string that contains the content of the item. This is used by the Microsoft 365 search engine to index the item and make it searchable.
acl, which defines the permissions of the item. This is used to define who can see the item in the search results. We won’t dive into this property, since we have already covered it in a previous post.

This method receives in input the collection of Document objects that was generated by the extract() method and returns a collection of ExternalItem objects. As such, most of the work here is adjusting the content of properties and content to match the schema we have defined.

For the purpose of my project, I didn’t have to make any change:

id is generated using the getDocId() method we have previously implemented.
The default implementation of properties already includes the information we need to describe a post, like title, url and iconUrl.
content is already set with the actual content of the post.
acl is set so that everyone can access the content of the Graph Connector, which is fine for my scenario.

This is the full implementation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


function transform(documents: Document[]): ExternalConnectors.ExternalItem[] {
 return documents.map(doc => {
 const docId = getDocId(doc);
 return {
 id: docId,
 properties: {
 // Add properties as defined in the schema in config.ts
 title: doc.title ?? '',
 url: doc.url,
 iconUrl: doc.iconUrl
 },
 content: {
 value: doc.content ?? '',
 type: 'text'
 },
 acl: [
 {
 accessType: 'grant',
 type: 'everyone',
 value: 'everyone'
 }
 ]
 } as ExternalConnectors.ExternalItem
 });
}

Putting all this together

Now that we have implemented the extract(), getDocId() and transform() methods, we can put all this together. The file contains another function, called load(), which is the one that actually loads the content into the Microsoft 365 tenant, by calling the Microsoft Graph APIs. However, we don’t need to actually make any change to this code, since the way the API is called doesn’t change based on the data source since, at this stage, we have already converted our items into ExternalItem objects.

The code contains another function that we don’t need to change: it’s called loadContent() and it simply takes care of calling all the functions we have seen before in the right order:

It calls the extract() function to get the list of items from the external data source.
It calls the transform() function to convert the list of Document objects into a list of ExternalItem objects, passing as input the outcome of the extract() function.
It calls the load() method, passing as input the collection of transformed items, so that they can be uploaded to the Microsoft 365 tenant.

Before we actually deploy our Graph Connector, there’s one last step to do.

Setting up the Adaptive Card

If we go back to the createConnection.ts file and we take a deeper look to the createConnection() function, we can notice the following code:

1
2
3
4


 const adaptiveCard = fs.readFileSync('./resultLayout.json', 'utf8');
 if (searchSettings) {
 searchSettings.searchResultTemplates![0].layout = JSON.parse(adaptiveCard);
 }

What it does is reading the content of the resultLayout.json file and setting it as the layout of the search result template. This is an Adaptive Card, which is used to display the results of the search in the Microsoft 365 tenant. The Adaptive Card is a JSON object that describes the layout of the card, which can include text, images, buttons and more. This is required: if we provide an invalid layout, we won’t be able to deploy the connector.

However, the default resultLayout.json file contains an empty card. This is because the layout of the card is highly dependent on the data we’re ingesting. In our case, we’re ingesting blog posts, so we can create a simple card that displays the title of the post, the content and a link to the original post. This is the layout I’ve created:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


{
 "type": "AdaptiveCard",
 "version": "1.6",
 "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
 "body": [
 {
 "type": "TextBlock",
 "text": "${title}",
 "wrap": true,
 "weight": "bolder",
 "size": "large"
 },
 {
 "type": "TextBlock",
 "text": "[${url}](${url})",
 "wrap": true
 },
 {
 "type": "TextBlock",
 "text": "${description}",
 "wrap": true,
 "maxLines": 4
 }
 ]
}

It’s a very simple layout, that generates the following result:

If you want to create a more complex layout, you can use the Adaptive Card Designer, which is a web-based tool that allows you to create Adaptive Cards visually and then it gives you the corresponding JSON.

Managing the authentication

We have mentioned how the Graph Connector uses the Microsoft Graph APIs to create the connection, upload the schema and ingest the content. However, we haven’t talked about how the Graph Connector authenticates against the Microsoft Graph. This is a critical aspect, since the Microsoft Graph requires authentication to be called.

This is achieved by creating a Microsoft Entra app registration in the tenant we want to deploy the connector to. Luckily, we don’t need to do this manually, but the template includes a script that does this for us. We just need to open a terminal in the project’s folder and run the following command if you’re on Windows:

1

./setup.ps1

or this one if you’re on macOS or Linux:

1
2


chmod +x ./setup.sh
./setup.sh

The script will open up your browser, asking you to login with your work account on a Microsoft 365 tenant. Make sure to use an account with admin permissions, since the scopes required to use the Microsoft Graph APIs we need require admin approval. At the end of the process, you will find a new file in the src folder of your project, called env.js. It contains all the information about the app registration (client id, client secret, etc.), which will be used by the graphClient.ts file to create an authenticated HTTP client.

Testing the connector

Now we are ready to deploy and test the connector. As first step, we need to actually build the code. Since we’re using TypeScript, we need to “compile it” so that it can be converted into JavaScript and executed by Node.js. As such, open a terminal on the project’s folder and run the following command:

1

npm run build

Now that the code is built, we have two options:

We deploy the connector to a Microsoft 365 tenant and we test if we are able to find results from our data source (in our case, this blog).
We use DevProxy to simulate the Microsoft Graph API calls. This will enable us to test if the code we have written is correct, without actually needing a tenant where to deploy the connector.

DevProxy is a tool developed by Microsoft that you can use to simulate APIs. It’s very helpful when you’re developing applications that needs to connect to an API, since it allows to simulate API errors, rate limits and throttling, to mock response, etc. In case of our project, it can help us by simulating the usage of the Microsoft Graph APIs to ingest the content, allowing us to test our code without actually deploying the connector.

This option is greatly simplified by the fact that the generated project already includes two configuration files for DevProxy:

One is called graph-connector-mocks.json and you can find it under the .devproxy folder. It contains the mocked responses for the Microsoft Graph APIs we’re going to use.
One is called devproxyrc.json and it’s in the root of the project. It contains the configuration of DevProxy, which enables the plugin to mock the responses and it includes the list of the URLs to intercept (all the Microsoft Graph endpoints).

All you have to do is to install DevProxy (I won’t dive into this, since it’s well documented in the official documentation) and then open two terminals in the project’s folder. In the first terminal, run the following command:

1

devproxy

By launching DevProxy from the project’s folder, it will automatically pick up the mock file and the configuration file and start intercepting requests against the Microsoft Graph.

In the second terminal, instead, run the following command:

1

npm run start:createConnection:proxy

This script will execute the createConnection.ts file, which is going to call the createConnection() and createSchema() functions. However, since we have passed the proxy parameter, the graphClient.ts file will set up the URL http://127.0.0.1:8888 as proxy, which means that all the requests will be sent to DevProxy instead to the real Microsoft Graph endpoints.

You’re going to observe something similar:

In a similar way, you can test the content ingestion by running the following command on the second terminal:

1

npm run:start:loadContent:proxy

This will call the loadContent() function in the loadContent.ts file, which starts the process of downloading the RSS feed of the blog, turning the items into ExternalItem objects and uploading them to the Microsoft 365 tenant.

And what if you want to actually test the code for real and actually deploy the connector? Simply call the scripts without the proxy parameter:

1
2
3


npm run start:createConnection

npm run start:loadContent

After both operations are completed, you will find the connector available in the Search & Intelligence section of the Microsoft 365 Admin Center:

And if you go over to Microsoft365.com and you search for a topic that is covered by one of the posts of this blog, you will see the results of the connector:

Conclusion

Graph Connectors are a very powerful way to bring external data into the Microsoft 365 ecosystem. However, building a custom one might not be the simplest experience, since there are lot of factors to keep in consideration: the connection, the schema, the content ingestion, the results layout, the authentication, etc. Thanks to new project generator we have introduced in this post, now it’s a little bit easier to create a Graph Connector from scratch, since we can focus only the data we want to ingest, rather than on the technical implementations.

You can find the sample connector I’ve created in this post on GitHub.

Happy coding!

Managing permissions with Graph Connectors

Mon, 15 Jul 2024 00:00:00 +0000

The Microsoft 365 ecosystem is, for many companies around the globe, the main productivity hub: it’s used to store documents, communicate via mails or messages, organize meetings, create internal sites. However, it’s very frequent for enterprises to have important data also outside the Microsoft 365 ecosystem: they might have public information published on their website; they might be using a solution to manage knowledge bases and tickets, like Jira or ServiceNow. Wouldn’t be nice if employees didn’t have to struggle to find the right information, but they could search across all the data sources in a single place? Even better, now that we have Copilot for Microsoft 365, if they could just chat with their data, regardless of where they’re located? This is the whole premise of Graph Connectors: they enable brining external data into the Microsoft 365 ecosystem, so that it becomes available for enterprise search and Copilot.

Take a look at this slide, taken from the Build 2024 session titled Developer’s Guide to customizing Microsoft Copilot:

As you can see, Graph Connector actually import the external data into the Microsoft 365 tenant by using the Microsoft Graph, they don’t perform a real time query against the external data source. Once imported, for Microsoft 365 there’s no difference between a Word document stored on OneDrive or a knowledge base article that is coming from ServiceNow. As such, tools like enterprise search and Copilot will be able to leverage them like if it’s 1st party data.

This means that Graph Connectors are ideal for scenarios like knowledge bases, document libraries, etc. where data doesn’t change very often. If you’re looking to integrate real time data into Copilot for Microsoft 365, then you should take a look into plugins.

But what about permissions? When it comes to 1st party data, things are easy. Microsoft 365 has a robust permission platform based on Microsoft Entra, so you can easily control who gets access to what. If you’re searching for information about Project X and they are stored in a Word document on a SharePoint site you don’t have access to, neither the search or Copilot for Microsoft 365 will be able to surface that. When it comes to data that comes from an external source, instead, things are a bit more complicated: the 3rd party platform will likely have its own permission system, which is different than Microsoft Entra. If plugins delegate this problem to the external platform, Graph Connectors must solve it on their own since data is imported into the Microsoft 365 tenant.

In this post, we won’t completely deep dive on how to create a Graph Connector. There are already lot of amazing resources for that, like:

We will focus, instead, on how to set up the permissions even if, while doing so, we’ll also see some basic concepts around Graph Connectors in action.

Let’s start!

Setting up the project

As a starting point, we’re going to use one of the available sample projects on GitHub. The project is based on Node.js and JavaScript and, by using the Microsoft Graph, it supports ingesting into a Microsoft 365 tenant the samples published on the Microsoft 365 Sample Solution Gallery. Thanks to this connector, you’ll be able to search across the available samples, right in your tenant.

If you don’t want to mess up with the production tenant of your organization or you don’t have admin rights, you can create a new tenant for free by following the instructions in the Microsoft 365 Developer Program. This way, you can experiment with the Graph Connector without any risk. Since the content ingested by the Graph Connector is exposed across many sources, you can experiment with it even if you don’t have a Copilot license. You can just use the standard search experience in Microsoft 365.

Let’s see how to set this project up. First, make sure to clone the GitHub repository on your machine or to download the solution as a .ZIP file.

Then, open up a terminal on the samples/nodejs-javascript-solutiongallery folder of the repository and perform the following steps:

Run the .\setup.ps1 script (or ./setup.sh if you’re on MacOS or Linux). The script will ask you to authenticate against your Microsoft 365 account, using the device code approach. You will need an administrator account on the tenant, since the scopes required to register the Graph Connector need admin consent. You will be asked to open a browser on the URL https://microsoft.com/devicelogin, to enter the code displayed in the terminal and then login with your account. Once you’re authenticated, the script will automatically create for you the Microsoft Entra app registration with the correct scopes, which is needed to run the application. The configuration of the application (client id, client secret, etc.) will be stored in a file called env.json
Restore the dependencies of the project by executing npm install.

Now we’re ready to add the Graph Connector to our tenant. This process involves three steps:

Create the connection
Define the schema
Upload the content

Let’s see them in more details.

Create the connection

The first step is to create the connection, which defines the basic properties of the Graph Connector, like its identifier, the public name and the description. The configuration is described using the JSON format. In the case of our sample project, you can find it inside the config.js file. You’ll see a config object with a property called connection, defined as following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


 connection: {
 // 3-32 characters
 id: 'samplesolutiongallery',
 name: 'Sample Solution Gallery',
 description: 'Sample solutions that demonstrate different Microsoft 365 extensibility capabilities.',
 activitySettings: {
 urlToItemResolvers: [
 {
 '@odata.type': '#microsoft.graph.externalConnectors.itemIdResolver',
 urlMatchInfo: {
 baseUrls: [
 'https://adoption.microsoft.com'
 ],
 urlPattern: '/sample-solution-gallery/sample/(?<sampleId>[^/]+)'
 },
 itemId: '{sampleId}',
 priority: 1
 }
 ]
 },
 searchSettings: {
 searchResultTemplates: [
 {
 id: 'samplesolgallery',
 priority: 1,
 layout: {}
 }
 ]
 },

The first elements are used to define the basic properties, like name and description.

The activitySettings node is used to define how to resolve the external item. Content imported from a Graph Connector, in fact, shows up in the search results but, if you try to open it, the operation will be delegated to the original platform the data belongs to. The urlToItemResolvers collection includes the instructions on how to generate the URL of the Sample Solution Gallery website starting from the id of the sample.

The searchSettings node, instead, is used to define how results will be displayed in the Microsoft 365 portal. The layout property is empty, but when the application will be executed will actually load the content of the resultLayout.json file and use it as a layout. This file contains the definition of an Adaptive Card, which allows to greatly customize the look & feel of the result. The following image shows the output of the Adaptive Card defined in the resultLayout.json file:

Once the configuration of the connection is defined, you must supply it to the following endpoint of the Microsoft Graph using a POST request:

1

https://graph.microsoft.com/v1.0/external/connections

In the sample project, this operation is performed with the help of an object defined in the graphClient.js file, which uses the credentials stored in the env.js file to create an authenticated HTTP client that can perform operations with the Microsoft Graph.

The connection is created by the createConnection() function in the createConnection.js file, as in the following snippet:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


async function createConnection() {
 console.log('Creating connection...');

 const { id, name, description, activitySettings, searchSettings } = config.connection;
 const adaptiveCard = fs.readFileSync('./resultLayout.json', 'utf8');
 searchSettings.searchResultTemplates[0].layout = JSON.parse(adaptiveCard);

 await client
 .api('/external/connections')
 .post({
 id,
 name,
 description,
 activitySettings,
 searchSettings
 });

 console.log('Connection created');
}

This method takes care of retrieving the configuration of the connection from the config.js file we have previously seen and then, using the authenticated HTTP client, of performing a POST operation against the correct endpoint, passing the configuration and the layout of the Adaptive Card that will be used to render the results.

Creating the schema

The second step is to define how an external item is defined. In the sample project, this is done, again, in the config.js file, inside the schema property. Here is a short snippet as an example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


schema: [
 {
 name: 'title',
 type: 'String',
 isQueryable: true,
 isSearchable: true,
 isRetrievable: true,
 labels: [
 'title'
 ]
 },
 {
 name: 'description',
 type: 'String',
 isQueryable: true,
 isSearchable: true,
 isRetrievable: true,
 },

Basically, we’re describing the properties of each item of our data source. In our case, every sample published in the Sample Solution Gallery has a title, a description, a list of authors, an image, etc. For each property, the schema describes the name, the type, if it’s searchable, etc.

In a similar way with the connection, the schema is supplied by calling another endpoint of the Microsoft Graph:

1

https://graph.microsoft.com/v1.0/external/connections/{id}/schema

The difference is that, this time, the HTTP operation to perform is PATCH, instead of POST. Additionally, you can notice how the URL contains a placeholder, {id}, which must be replaced with the identifier of the connection you have previously created. In our sample, it would be samplesolutiongallery, as declared in the id property of the connection object.

Also for this scenario, the createConnection.js file includes a function called createSchema(), which uses the authenticated HTTP client to call the Microsoft Graph endpoint passing the schema:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


async function createSchema() {
 console.log('Creating schema...');

 const { id, schema } = config.connection;
 try {
 const res = await client
 .api(`/external/connections/${id}/schema`)
 .header('content-type', 'application/json')
 .patch({
 baseType: 'microsoft.graph.externalItem',
 properties: schema
 });

 const status = res.status;
 if (status === 'completed') {
 console.log('Schema created');
 }
 else {
 console.error(`Schema creation failed: ${res.error.message}`);
 }
 }
 catch (e) {
 console.error(e);
 }
}

Deploy the connector

Now that we have seen how the connection and the schema are configured, we can start to actually deploy the Graph Connector in our tenant.

In order to do that, open a terminal in the folder which contains the nodejs-javascript-solutiongallery project you have cloned and run the following command:

1

npm run createConnection

This command will execute the two functions we have seen: createConnection() and createSchema(). The operation will take a while to be completed. The creation of the connection is almost instant, while uploading the schema will take a few minutes.

After the operation is completed, you can check that everything is fine by opening the Microsoft 365 Admin Center. Under Settings, click on Search & Intelligence, then Data sources. You should see your Graph Connector listed there:

Under the Required actions column you will find a link labelled Include in results. This will enable the content imported by the Graph Connector to be included in search results and in Copilot responses.

However, at the moment, we haven’t imported any content, we have just configured the connection and the schema. You’ll notice, in fact, that the number of indexed items is 0. Let’s see how the content is imported, which is the key of this article, since it’s also where you’re going to define permissions.

Upload the content

The logic for uploading the content is, potentially, the most complex one, since it needs to take into account many factors. For example, a good crawl engine shouldn’t try to upload all the content each time, but only what’s changed. We won’t dig into the complexity of this implementation into this post, but we’ll focus on the content upload. The sample project itself uses a simplified approach: when a sync is performed, it stores the last sync date and time into a text file called latestChange.txt. Then, every time you trigger a new sync, it will check if the gallery has any sample that was published after this date and, if yes, it’s going to push it into the tenant.

This operation is performed by the loadContent.js file, which contains the definition of 3 key functions:

extract(), which uses an API to retrieve the list of the samples from the Sample Solution Gallery in JSON format.
transform(), which takes the samples in JSON format and turns them into externalItem objects, which is the type used by the Microsoft Graph to map content imported by a Graph Connector. This method takes all the properties returned by the API and converts them into the properties that we have declared in the schema.
pushItem(), which actually uploads the items into the tenant. This operation is similar to the one we have seen to create the connection and the schema, since it uses the authenticated client to perform a HTTP operation against the Microsoft Graph. The endpoint used for this scenario is the following one:
1

https://graph.microsoft.com/v1.0/external/connections/{id}/items/{itemId}
In this case, we have two placeholders: the first one is the identifier of the connection (in our case, it would be samplesolutiongallery); the second one is the identifier of the item we’re uploading (in our case, it’s the slug of the sample that it’s used in the URL. For example, the identifier of the following sample https://adoption.microsoft.com/sample-solution-gallery/sample/pnp-devproxy-northwinddb/ would be pnp-devproxy-northwinddb).

The operation to perform uses the PUT verb of the HTTP protocol.

Let’s focus on the second function, transform():

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39


function transform(samples) {
 return samples.map(sample => {
 // Date must be in the ISO 8601 format
 const createdDateTime = new Date(sample.creationDateTime).toISOString();
 const lastModifiedDateTime = new Date(sample.updateDateTime).toISOString();
 const imageUrl = sample.thumbnails.length > 0 ? sample.thumbnails[0].url : '';
 return {
 id: sample.sampleId,
 properties: {
 title: sample.title,
 description: sample.shortDescription,
 'authors@odata.type': 'Collection(String)',
 authors: sample.authors.map(author => author.displayName),
 'authorsPictures@odata.type': 'Collection(String)',
 authorsPictures: sample.authors.map(author => author.pictureUrl),
 imageUrl,
 iconUrl: 'https://raw.githubusercontent.com/pnp/media/master/pnp-logos-generics/png/teal/300w/pnp-samples-teal-300.png',
 url: `https://adoption.microsoft.com/sample-solution-gallery/sample/${sample.sampleId}/`,
 createdDateTime,
 lastModifiedDateTime,
 'products@odata.type': 'Collection(String)',
 products: sample.products,
 'metadata@odata.type': 'Collection(String)',
 metadata: sample.metadata.map(m => `${m.key}=${m.value}`)
 },
 content: {
 value: sample.shortDescription,
 type: 'text'
 },
 acl: [
 {
 accessType: 'grant',
 type: 'everyone',
 value: 'everyone'
 }
 ]
 }
 });
}

The method is fairly simple: it takes the samples collection as input (which is the JSON data downloaded from the API) and then, for each item, it transforms it into an externalItem object, by setting the value of the various properties we have defined in the schema (title, description, authors, etc.).

The most interesting part for the purpose of this post is the acl property, which is the one used to define the permissions. The default value in the sample project is the following one:

1
2
3
4
5
6
7


acl: [
{
 accessType: 'grant',
 type: 'everyone',
 value: 'everyone'
}
]

This is the most permissive approach: everyone in the tenant will get access to this item.

If you want to test this behavior, you just need to open again a terminal on the sample folder and run the following command:

1

npm run loadContent

This command will execute the various functions we have just seen and that are declared in the loadContent.js file, which will actually upload the content from the Samples Solution Gallery into our tenant.

The command will have two outcomes:

In the Admin Center, you will see that the number of indexed items won’t be 0 anymore:
You’ll be able to search for content published on the Sample Solution Gallery in your tenant. For example, go to Office.com and, using the search bar at the top, search for the keyword http cats. You will see a result coming from the connector, which uses the layout of the Adaptive Card included in the project:

Be aware that, even if the command doesn’t take a long time to complete, the content might take a while to be available (even a few hours). So don’t be surprised if you won’t be able to find any result, just wait a bit and try again!

Now that the content is available in our tenant, we can experiment with the different ways to control permissions.

Controlling permissions at user level

The simplest approach is to define the users who can access the content at personal level. Let’s take a look at the following setup for the acl property:

1
2
3
4
5
6
7
8


acl: [
{
 accessType: 'grant',
 type: 'User',
 value: '17aba01e-8833-4041-a608-ff6ad5506d72',
 identitySource: 'azureActiveDirectory'
}
]

The three key properties are:

type, which is User
identitySource, which is the source of the user, in our case azureActiveDirectory
value, which is the GUID that identifies the user in Microsoft Entra. To get it, you must go to the Azure portal, head to the Microsoft Entra section and select Users:

Then, click on the user you are interested to and, in the detail page, you’ll find the unique identifier, under the Object ID section:

When you use the grant value for the accessType property, like in this case, it means that everyone in the tenant won’t be able to see this content, except for the ones listed in the acl property. If you want to set it up the other way around, you can use instead the deny value. Let’s consider the following acl:

1
2
3
4
5
6
7
8


acl: [
{
 accessType: 'deny',
 type: 'User',
 value: '17aba01e-8833-4041-a608-ff6ad5506d72',
 identitySource: 'azureActiveDirectory'
}
]

In this case, every user will be able to see the content, except for the one you have specified in the property.

If you want to test the feature in action, just change the acl property in the loadContent.js file like in the previous snippet, just remember to set as value the id of a real user in your tenant. Then delete the latestChange.txt file (otherwise the application won’t find any new item and it won’t trigger the sync) and then run again the npm run loadContent command.

Once the operation is completed, try to repeat the search for the http cats keyword. You should be able to observe that only the user you have added in the acl property will be able to find the sample from the Sample Solution Gallery. Every other user will get the message We couldn’t find any results for http cats.

Also in this case, the tenant might take a while to update the permissions, so you might need to wait a few hours before seeing the behavior changing.

Using a Microsoft 365 Dev tenant is really helpful in this scenario, since you can easily login with different users and test the behavior.

Controlling permissions at group level

In most of the cases, setting the permissions one by one for each user isn’t very effective. As such, you can use the acl property also to grant or deny access to a Microsoft Entra group, as in the following example:

1
2
3
4
5
6
7
8


acl: [
 {
 accessType: 'grant',
 type: 'Group',
 value: 'da02f39f-da22-435a-a9c2-7bec4895296c',
 identitySource: 'azureActiveDirectory'
 }
 ]

The configuration is very similar, except that the type property must be set to Group. Also in this case, as value, we must supply the Object ID from Microsoft Entra of the group. To find it, you can follow the same approach we used for the user, but this time, in the Azure portal, you need to choose Groups instead of Users under Microsoft Entra -> Manage.

Controlling permissions using external groups

All the previous scenarios work well when your external data source has a permission model that matches your Microsoft Entra organization. However, it’s very frequent that the data source is actually using their own permission model. For example, if you are using ServiceNow to manage tickets and knowledge bases, the permissions will be regulated by the account that employees have on the ServiceNow platform.

To manage this scenario, the Microsoft Graph supports the option to create external groups, which are special groups that exists only in the context of the connector. Inside an external group, you can add one or more Microsoft Entra users or groups. Thanks to this feature, you can define your own logic to keep the users from the external platform in sync with the users in Microsoft Entra. For example, let’s say you’re using the same group names across both platforms. You could build a sync solution that enables the following flow:

You create multiple external groups in the connector, one for each group you’re using in the external platform. For example, if you have a HR group in ServiceNow, you create an external group called HR for the Graph Connector.
For every user in the external group, you use the alias to find the matching user in Microsoft Entra.
You add the user to the proper external group.

Thanks to this approach, you are able to maintain in sync the permissions between the external platform and the Microsoft Entra organization.

Let’s see the steps to follow this approach. To perform them, we’ll need to call a series of Microsoft Graph APIs which, however, aren’t part of the sample project we have used so far. If you don’t want to add the code on your own, you can use the Graph Explorer tool. Just make sure to sign in with a work account on your tenant with Admin rights, then you’ll be able to easily call the APIs I’m going to describe in the next sections.

Graph Explorer can help you also in getting access to the right scopes. All the APIs we’re going to see, in fact, requires specific scopes to be called. You can request them by clicking on the Modify permissions button on the top right corner of the tool, after you have added the endpoint and the body of the request. Just click Consent near the permission you need and the tool will take care of the rest.

The Microsoft Graph APIs to manage external groups are still in beta, so you must use the https://graph.microsoft.com/beta endpoint to call them.

Create the external group

The first step is to create the external group. We do this performing a HTTP POST request to the following endpoint:

1

https://graph.microsoft.com/beta/external/connections/{connectionId}/groups

Using this API requires requesting access to the scope ExternalConnection.ReadWrite.All.

The connectionId is the identifier of the connection we have created at the beginning of the article. In case of our sample project, that would be samplesolutiongallery. The body of the request must contain the id, name and description of the group, as in the following:

1
2
3
4
5


{ 
 "id": "contoso", 
 "displayName": "Contoso", 
 "description": "Contoso employees"
} 

Unfortunately, this API doesn’t support listing all the external groups which are available for a specific connection. This is an information that you must track on your own, for example by storing it in a database or in a file.

Adding users or groups to an external group

The next step is to add users or groups to the external group. This operation is performed by calling the following endpoint:

1

https://graph.microsoft.com/beta/external/connections/{connectionId}/groups/{groupId}/members

In this case, we have two placeholders:

connectionId, which is the identifier of the connection (again, in our sample project it’s samplesolutiongallery).
groupId, which is the identifier of the group we have created in the previous step, so in the example above it would be contoso.

To call this API, you’ll need to request the ExternalItem.ReadWrite.All scope.

The API supports both to add single users or, in case you already have a group in Microsoft Entra that matches the group in the external platform, you can add the group itself.

Regardless of the choice, we must supply as body of the POST request a JSON payload with the Object Id of the Microsoft Entra user or group and, as identitySource, the value azureActiveDirectory. What changes is the value of the type property, which could be either user or group.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


{
 "id": "25f143de-be82-4afb-8a57-e032b9315752",
 "type": "user",
 "identitySource": "azureActiveDirectory"
}

{
 "id": "99a3b3d6-71ee-4d21-b08b-4b6f22e3ae4b",
 "type": "group",
 "identitySource": "azureActiveDirectory"
}

Also in this case, unfortunately, the API doesn’t support returning a list of the users or groups assigned to an external group, so it’s up to you to store this information. However, if you’re building a solution (like a service) to keep in sync users across Microsoft Entra and the external platform, you can easily keep track of this information.

To properly build a sync service, you’ll probably need also a way to remove users or groups, in case for example one of the employees leaves the company. In this case, you’ll need to use the same endpoint as before but:

With the DELETE verb
With the id of the user or group you want to remove in the URL

For example, if you want to delete the user with the id 25f143de-be82-4afb-8a57-e032b9315752 from the group with the id contoso, you’ll need to perform a HTTP DELETE request against the following endpoint:

1

https://graph.microsoft.com/beta/external/connections/samplesolutiongallery/groups/contoso/members/25f143de-be82-4afb-8a57-e032b9315752

You’ll get back a 204 status code without any content, if the operation is successful.

Setting up the acl property

Once you have created the external group and added one or more user or group to it, it’s time to configure the external items you import using the Graph Connector in the right way. Again, we’re going to change the value of the acl property:

1
2
3
4
5
6
7
8


acl: [
 {
 accessType: 'grant',
 type: 'Group',
 value: 'Contoso',
 identitySource: 'External'
 }
 ]

We’re using, again, Group as type. However, this time we set:

As identitySource, the value External
As value, the name of the group we have created in the previous steps.

That’s it! If you force a sync of the content with the command npm run loadContent, you should be able to see that only the users you have added to the group Contoso will be able to find the samples from the Sample Solution Gallery.

Just remember that permissions might need some time to propagate, so you might have to wait a bit to see the changes.

Wrapping up

In this post, we have explored how to manage permissions for data that lives outside the Microsoft 365 ecosystem, but that you want to import into your tenant using a Graph Connector. We have seen how to set up the connection, how to define the schema and how to upload the content. We have also seen how to control permissions at user level, at group level and how to use external groups to manage permissions when the external platform has its own permission model.

Remember that you can find the sample project use in this blog on GitHub.

Build a Blazor application to manage a custom Viva Learning provider - Part 3

Thu, 16 May 2024 00:00:00 +0000

Welcome to the last part of this series about building a web application with Blazor to manage a custom provider for Viva Learning! In part 1, we built the base architecture and we introduced the option to create and list custom providers. In part 2, we made our custom provider to something actually useful, by adding content that the employees can read or watch. In this post, we’re going to introduce assignments.

Viva Learning, in fact, isn’t just a platform that offers content that employees can leverage to learn new things. It also allows managers to create assignments, which are courses that employees must complete. Think, for example, of compliance training about security or privacy. Or a new selling approach that all your sales team must learn. Assignments are a powerful tool to ensure that your employees are up-to-date with the latest information and that they are compliant with the company’s policies.

Let’s start writing the code to manage assignments in our Blazor application.

Adding an assignment

This post will be much shorter compared to the previous ones. Thanks to the building blocks we have set in the first post and the implementation we did to include a GraphServiceClient instance configured to use application permissions in the second post, we already have everything we need. The Microsoft Graph API we must use to submit assignments is the following one:

1

POST https://graph.microsoft.com/v1.0/employeeExperience/learningProviders/{learningProviderId}/learningCourseActivities

This is translated, in the Microsoft Graph .NET SDK, to the following code, which we must add to our CustomGraphService class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


public async Task AddAssignment(string providerId, string contentId, string userId, DateTimeTimeZone dueDateTime)
{
 LearningAssignment assignment = new LearningAssignment
 {
 LearningProviderId = providerId,
 LearningContentId = contentId,
 LearnerUserId = userId,
 Status = CourseStatus.NotStarted,
 AssignmentType = AssignmentType.Required,
 DueDateTime = dueDateTime,
 AssignedDateTime = DateTimeOffset.Now,
 OdataType = "#microsoft.graph.learningAssignment"
 };

 await applicationClient.EmployeeExperience.LearningProviders[providerId].LearningCourseActivities.PostAsync(assignment);
}

The assignment is mapped with the LearningAssignment class, which has a few key properties we must define:

LearningProviderId: the ID of the provider that offers the content.
LearningContentId: the ID of the content.
LearnerUserId: the ID of the user that must complete the assignment.
Status: the status of the assignment. We set it to NotStarted because we’re creating a new assignment.
AssignmentType: the type of the assignment. We set it to Required because we want the user to complete it.
DueDateTime: the date and time when the assignment must be completed.
AssignedDateTime: the date and time when the assignment was assigned.

Then, we just call the PostAsync() method on the LearningCourseActivities property of the selected provider, passing the assignment object. The Microsoft Graph SDK will take care of the rest.

Since we’re already working with the CustomGraphService class, let’s add another method that we’re going to need later:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


public async Task<string> GetUserId(string usermail)
{
 var users = await applicationClient.Users.GetAsync(requestConfig =>
 {
 requestConfig.QueryParameters.Filter = $"mail eq '{usermail}'";
 requestConfig.QueryParameters.Select = ["id"];
 });

 return users.Value.FirstOrDefault().Id;
}

This method uses the Microsoft Graph APIs to search for a specific user in the tenant given their mail address and to retrieve their ID. This is useful because, when we create an assignment, we must specify the ID of the user that must complete it. However, from a user experience perspective, it’s much easier to ask the user to specify their mail address rather than their ID, which is a GUID. The method requires using the application permissions to call the /users endpoint of the graph. To improve the performance, we provide a requestConfig parameter to filter the results by the mail address we’re looking for and to select only the id property. Thanks to this method, when it will come to build the UI to create an assignment, we’ll be able to simply ask to the user the mail address of the person who must complete the assignment, and then we’ll be able to retrieve the ID of the user using this method.

Adding the user interface

Now that we have our method, we just need to update the UI so that we can enable the user to create assignments. Assignments are mapped to a specific content, so let’s add a new column in the content table in the LearningProviders.razor component. This column will contain a button that, when clicked, will redirect to a new page to assign the selected content to a user.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


@if (contents != null)
{
 <table class="table">
 <thead>
 <tr>
 <th>Title</th>
 <th>Url</th>
 <th>Assignments</th>
 </tr>
 </thead>
 <tbody>
 @foreach (var content in contents)
 {
 <tr>
 <td>@content.Title</td>
 <td>@content.ContentWebUrl</td>
 <td><button type="button" @onclick="(() => AddAssignment(content.Id))">Add assignment</button></td>
 </tr>
 }
 </tbody>
 </table>
}

The button we have just added will call a method in code called AddAssignment(), passing as parameter the id of the selected content. Let’s implement this method in the LearningProviders.razor.cs file:

1
2
3
4


public void AddAssignment(string learningContentId)
{
 navigationManager.NavigateTo($"/addAssignment?learningContentId={learningContentId}&learningProviderId={providers?.FirstOrDefault().Id}");
}

We’re going to use, again, the navigationManager object to trigger the navigation to a new page, called AddAssignment, which we’re going to create as next step. We’re going to implement the navigation in a slightly different way than what we did to enable the creation of content. This is because, to manage an assignment, we need to carry over two properties: the id of the content and the id of the provider. As such, the best way to do that is by using query strings.

Now let’s create a new component under the Pages folder called AddAssignment.razor. This component will contain a form to create a new assignment, using the same approach we have used so far based on the EditForm component. First, let’s create a model class to represent the assignment:

1
2
3
4
5
6
7
8


public class LearningAssignmentModel
{
 [Required]
 public string UserMail { get; set; }

 [Required]
 public DateTime DueDate { get; set; }
}

For the sake of simplicity, in this sample we’re going to allow the user to customize only two key properties: the assigned user and the due date. In a real-world scenario, you would probably want to add more properties, such as the assignment type or the status. Now, we can add to our Razor component a form mapped to this new model class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


<EditForm Model="@learningAssignmentModel" OnValidSubmit="@HandleSubmit">
 <DataAnnotationsValidator />
 <ValidationSummary />

 <p>
 <div>User mail</div>
 <div><InputText id="userMail" @bind-Value="learningAssignmentModel.UserMail" /></div>
 </p>

 <p>
 <div>Due date</div>
 <div><InputDate id="dueDate" @bind-Value="learningAssignmentModel.DueDate" /></div>
 </p>

 <p>
 <button type="submit">Assign</button>
 </p>

</EditForm>

The form is very simple: it contains two fields, one for the user’s email and one for the due date. When the form is submitted, the HandleSubmit() method is called. Let’s take a look now at the AddAssignment.razor.cs file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


public class AddAssignmentBase: ComponentBase
{
 [SupplyParameterFromQuery(Name = "LearningContentId")]
 public string LearningContentId { get; set; }

 [SupplyParameterFromQuery(Name = "LearningProviderId")]
 public string LearningProviderId { get; set; }

 [Inject]
 ICustomGraphService graphService { get; set; }

 public LearningAssignmentModel learningAssignmentModel = new();

 public async Task HandleSubmit()
 {
 if (learningAssignmentModel != null)
 {
 var userId = await graphService.GetUserId(learningAssignmentModel.UserMail);
 DateTimeTimeZone dueDate = new()
 {
 DateTime = learningAssignmentModel.DueDate.ToString("yyyy-MM-ddTHH:mm:ss"),
 TimeZone = "UTC"
 };

 await graphService.AddAssignment(LearningProviderId, LearningContentId, userId, dueDate);
 }
 }
}

First, in a similar way we did for the content creation, we’re going to retrieve the parameters we need from the LearningProviders.razor page. The difference is that, since they will be retrieved from the query string and not from the Uri, we need to use a different attribute to decorate our properties, called [SupplyParameterFromQuery]. The outcome, however, will be the same: the two properties will be automatically populated with the values from the query string, without needing to perform any manual parsing of the URL.

Then, we have the HandleSubmit() method, which is called when the form is submitted. As the first step, we use the GetUserId() method we have implemented before to retrieve the ID of the user that must complete the assignment from their mail address. Then, we can simply call the AddAssignment() method we have implemented, passing the id of the content, the id of the provider, the id of the user, and the due date. The only thing you need to keep in mind is that the due date is specified using a DateTimeTimeZone object, so you need to create an instance of this class starting from the DateTime property that is returned by the InputDate Blazor component.

Testing the code

That’s it! Now we’re ready to test the code. Press F5 to start the application and then:

Navigate to the Learning Providers page.
Click on the Assign button next to one of the available contents.
In the new form, specify the email address of the user who must complete the assignment and the due date. For simplicity, you can use your own email address.
Click on the Assign button.

To test that the operation completed successfully, you can just login to the Viva Learning portal in Microsoft Teams. You should see that now you have some pending assignments right from the main page:

If you click on Assigned to you, you should see the content from your custom provider being assigned to you with the due date you have specified:

Wrapping up

In this post, we have concluded our journey of building a web application with Blazor to manage a custom provider for Viva Learning. We have added the ability to create assignments, which are mandatory courses for employees. Now we have a full application that we can use to manage all the basic operations that you can perform with Viva Learning to integrate your content catalog into the platform.

You can find the full sample application on GitHub.

Happy coding!

Build a Blazor application to manage a custom Viva Learning provider - Part 2

Wed, 15 May 2024 00:00:00 +0000

In the previous post, we began building a Blazor web application for managing a custom Viva Learning provider. We created the project, implemented the authentication flow, and added the initial operations: creating and listing custom content providers. Today, we’ll explore how to add new content.

Managing the authentication

You might be wondering, “Why are we discussing authentication again? Didn’t the Microsoft Identity Web library handle everything for us?” Almost 😊 The challenge comes from the fact that the authentication process we have implemented is based on the user’s identity. This was fine when it comes to create and list learning providers, since these APIs required delegated permissions. Delegated permissions allow the app to act on behalf of a user, whereas application permissions enable the app to act independently. This is where the concept of application permissions comes into play.

If you want to learn more, you can read the official documentation on the topic.

This means that the delegatedClient object that was injected into the CustomGraphService class and that we have been using so far isn’t enough. We need to create a new instance of the GraphServiceClient class that will use application permissions. In order to do this, we need to manually create a new instance of the GraphServiceClient class, thus we need to read the Microsoft Entra configuration we have stored in the appsettings.json file. As such, we need a new IConfiguration parameter in the constructor of the CustomGraphService class to easily get access to the content of the appsettings.json file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


private readonly IConfiguration configuration;
private GraphServiceClient delegatedClient;
private readonly MicrosoftIdentityConsentAndConditionalAccessHandler consentHandler;

public CustomGraphService(GraphServiceClient delegatedClient, IConfiguration configuration, MicrosoftIdentityConsentAndConditionalAccessHandler consentHandler)
{
 this.configuration = configuration;
 this.consentHandler = consentHandler;
 this.delegatedClient = delegatedClient;
}

Now we can add a new method in the service that will create the application client.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


private GraphServiceClient applicationClient;

public void AcquireApplicatonAccessToken(string tenantId)
{
 var aadConfig = configuration.GetSection("AzureAd");
 var clientId = aadConfig["ClientId"];
 var clientSecret = aadConfig["ClientSecret"];

 var clientSecretCredential = new ClientSecretCredential(tenantId, clientId, clientSecret);

 applicationClient = new GraphServiceClient(clientSecretCredential);
}

The Microsoft Entra application we’re going to use to authenticate is the same, so we retrieve the content of the AzureAD section of the configuration file, in which we stored the client id, the redirect URL, etc. However, we need to authenticate in a different way, since application permissions don’t require any user interaction. For this purpose, we create a ClientSecretCredential object passing, as parameters, the client ID and the client secret. There’s an extra parameter we need to supply, which is the tenant ID: we get it as a parameter from the method, so that we can enable the usage of our application with any tenant in a multi-tenant scenario. Then, we create a new instance of the GraphServiceClient class, passing the ClientSecretCredential object as a parameter. We store this instance in a property of the class, so that we can use it in the methods that require application permissions.

Adding new content to a custom provider

The first operation we’re going to implement is the ability to add new content to a custom provider. We’re going to use articles from this blog as sample content to load in our provider. The API to do this is a bit peculiar. We’ll have to perform, in fact, a PATCH operation against the following URL:

1

https://graph.microsoft.com/v1.0/employeeExperience/learningProviders/{learningProviderId}/learningContents(externalId='{externalId}')

As you can see, the URL is a bit peculiar because we need to supply two different parameters in the URL itself:

The ID of the learning provider we want to add the content to. This is the learningProviderId parameter.
We need to supply the ID that identifies the content we want to add. This ID is the externalId property of the content and it’s called external because this is the ID how your system identifies that content. For example, in case of this blog, it could be the slug of the post. We’ll have to provide this value regardless if we’re performing the operation to create a new content or to update an existing one.

Luckily, the Microsoft Graph library for .NET makes this operation really simple, as you can see from the following snippet:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


public async Task AddLearningContent(string providerId, string contentId, string title, string contentUrl, string language)
{
 var learningContent = new LearningContent
 {
 ExternalId = contentId,
 Title = title,
 ContentWebUrl = contentUrl,
 LanguageTag = language
 };

 await applicationClient.EmployeeExperience.LearningProviders[providerId].LearningContentsWithExternalId(contentId).PatchAsync(learningContent);
}

First, we create a new LearningContent object, which maps a content for Viva Learning. There are many properties available that you can explore in the official docs, in this sample you can see the four required ones:

ExternalId: the ID that identifies the content in your system. It must be the same you’re going to supply also in the URL.
Title: the title of the content.
ContentWebUrl: the URL where the content is available.
LanguageTag: the language of the content.

Now we can call the Microsoft Graph APIs. Whenever they require a parameter in the URL to identify a specific content (like, in this case, the identifier of the learning provider), the .NET library translates this by using the array syntax. This is why we have the LearningProviders[providerId] syntax in the method. The complex second parameter implementation is made easier by the LearningContentsWithExternalId() helper method, which allows us to specify the external ID of the content we want to add. The PatchAsync() method, finally, is the one that actually performs the PATCH operation and adds the content to our provider.

Since we’re already working on the CustomGraphService class, we can add a new method to retrieve the list of available content for a given provider. The implementation is simple:

1
2
3
4
5


public async Task<IList<LearningContent>?> GetLearningContentAsync(string providerId)
{
 var response = await applicationClient.EmployeeExperience.LearningProviders[providerId].LearningContents.GetAsync();
 return response.Value;
}

Now that we have both methods, we can update the UI to support this new feature. We’re going to make three changes:

Initialize the GraphServiceClient which supports application permissions.
Update the LearningProviders.razor component to show the list of contents for a provider.
Create a new component to add content to a provider.

Initializing the application client

At the beginning of the post, we have added a new method to our CustomGraphService class to create an instance of the Microsoft Graph client which uses application permissions. Now it’s time to use it! Let’s change the OnInitializedAsync() method in the LearningProviders.razor.cs file to call this method:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


[Inject]
AuthenticationStateProvider AuthenticationStateProvider { get; set; }

protected override async Task OnInitializedAsync()
{
 var state = await AuthenticationStateProvider.GetAuthenticationStateAsync();
 var user = state.User;

 string tenantId = user.FindFirst("http://schemas.microsoft.com/identity/claims/tenantid")?.Value;
 graphService.AcquireApplicatonAccessToken(tenantId);
 providers = await graphService.GetLearningProvidersAsync();
}

As you can notice, we’re not just calling the method, but there’s more to it. The reason is that we built the AcquireApplicationAccessToken() method to support multi-tenant scenarios, so we must provide as parameter the id of the tenant we want to interact with. We can retrieve this information from the user’s claims, which contains the tenant ID. To do that, we need to:

Inject the AuthenticationStateProvider service, which is automatically registered in the dependency injection container when we initialize the authentication support.
Call the GetAuthenticationStateAsync() method to retrieve the current user’s authentication state.
Retrieve the user’s claims from the AuthenticationState object.
Extract the tenant ID from the claims.

Now that we have the identifier of the tenant, we can finally call the AcquireApplicationAccessToken() method we have previously implemented. This will create the GraphServiceClient object that uses application permissions, which will be needed to call the Microsoft Graph APIs to work with content.

Listing the content for a given provider

Now that we have setup the code so that we can use an application client, we’re going to change the column which contains the provider’s identifier to become actionable. By clicking on the id, we’re going to load the available content for that provider. First, we’re going to add in the LearningProviders.razor.cs file a new method to load the content given a provider’s ID:

1
2
3
4
5
6
7


public IList<LearningContent> contents = null;

public async Task LoadContent(string providerId)
{
 contents = await graphService.GetLearningContentAsync(providerId);

}

We have created a new property called contents, which will store the list of content for a given provider. The LoadContent() method receives, as input, the id of the provider and it uses it to call the GetLearningContentAsync() method we have previously implemented.

Now update the table in the LearningProviders.razor component in the following way:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


<table class="table">
 <thead>
 <tr>
 <th>Id</th>
 <th>Name</th>
 <th>Content</th>
 </tr>
 </thead>
 <tbody>
 @foreach (var provider in providers)
 {
 <tr>
 <td><a href="#" @onclick="() => LoadContent(provider.Id)" @onclick:preventDefault>@provider.Id</a></td>
 <td>@provider.DisplayName</td>
 </tr>
 }
 </tbody>
</table>

Instead of just displaying the value of the Id property, we’re now wrapping it in an anchor tag. We’re also adding an @onclick directive to the tag, which will call the LoadContent() method when the user clicks on it, passing as parameter the id of the current row’s provider. We’re also adding the @onclick:preventDefault directive to prevent navigating to the URL specified in the href attribute. We just want to call, in fact, the LoadContent() method.

As the second step, we need to add a new table which is actually going to display the items in the contents collection. Here is the code to add below the previous table:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


@if (contents != null)
{
 <table class="table">
 <thead>
 <tr>
 <th>Title</th>
 <th>Url</th>
 </tr>
 </thead>
 <tbody>
 @foreach (var content in contents)
 {
 <tr>
 <td>@content.Title</td>
 <td>@content.ContentWebUrl</td>
 </tr>
 }
 </tbody>
 </table>
}

The implementation is very similar to the one for the providers. We add a table with two columns, one for the tile of the content and one or its URL. Then, using a @foreach statement, we iterate over the contents collection and add a new row to the table for each item. Now, every time you click on the id of the provider, the list of content for that provider will be loaded and displayed in the table.

That’s it! However, it’s a bit early to test the work, because the list of content will be actually empty. Let’s add a new component to create one!

Adding content to a provider

As the first step, we’re going to create a new component called NewLearningContent.razor inside the Pages folder. The approach we’re going to use it the same one we have used for the NewLearningProvider component. As such, first, we need to create a new class to hold the model of the content we’re going to add. This class will be called LearningContentModel and it will contain the following properties:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


public class LearningContentModel
{
 [Required]
 public string? ExternalId { get; set; }

 [Required]
 public string? Title { get; set; }

 [Required]
 public string? ContentWebUrl { get; set; }

 [Required]
 public string? LanguageTag { get; set; }
}

The properties are the same ones we have seen in the LearningContent class provided by the Microsoft Graph SDK. We’re also adding the Required attribute to all the properties, to ensure that the user provides a value for each of them.

Now we can build the UI using an EditForm component to request the input from the user. Here is how it looks like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


<EditForm Model="@learningContentModel" OnValidSubmit="@HandleSubmit">
 <DataAnnotationsValidator />
 <ValidationSummary />

 <p>
 <div>External Id</div>
 <div><InputText id="externalId" @bind-Value="learningContentModel.ExternalId" /></div>
 </p>
 <p>
 <div>Title</div>
 <div><InputText id="title" @bind-Value="learningContentModel.Title" /></div>
 </p>

 <p>
 <div>Content URL</div>
 <div><InputText id="contentUrl" @bind-Value="learningContentModel.ContentWebUrl" /></div>
 </p>

 <p>
 <div>Language</div>
 <div><InputText id="longLogoLight" @bind-Value="learningContentModel.LanguageTag" /></div>
 </p>

 <p>
 <button type="submit">Create</button>
 </p>

</EditForm>

Let’s see now the implementation of the NewLearningContent.razor.cs file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


public class NewLearningContentBase: ComponentBase
{
 [Parameter]
 public string? LearningProviderId { get; set; }

 [Inject]
 ICustomGraphService graphService { get; set; }

 public LearningContentModel learningContentModel = new();

 public async Task HandleSubmit()
 {
 if (learningContentModel != null)
 {
 await graphService.AddLearningContent(LearningProviderId, learningContentModel.ExternalId, learningContentModel.Title, learningContentModel.ContentWebUrl, learningContentModel.LanguageTag);
 }
 }
}

The code should be familiar. Also in this case, we’re injecting our instance of the CustomGraphService class, which we’re going to use inside the HandleSubmit() method to call the AddLearningContent() method we have previously implemented. The method will pass the values provided by the user in the form to the service.

There’s only an extra thing we need to take care of. As you can see, we have a property called LearningProviderId, which is decorated with the [Parameter] attribute. This is because we’re going to get the id of the provider from the previous page by passing it as a navigation parameter (for example, /newLearningContent/47ad9681-2206-4599-89c5-810252d7b81a). By using this special attribute, its value will be automatically populated based on the URL. To make this working, we need to change the routing of the page in the NewLearningContent.razor file so that it can accept the id of the provider as a parameter:

1
2


@inherits NewLearningContentBase
@page "/newLearningContent/{learningProviderId}"

This way, since the name of the parameter in the URL matches the name of the property in code, it will be automatically populated.

As the last step, we need to trigger the navigation to the new page when from the LearningProviders.razor component, so that we pass the id of the provider. First, let’s add a new column to our table which will be populated by a button that will trigger the navigation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


<table class="table">
 <thead>
 <tr>
 <th>Id</th>
 <th>Name</th>
 <th>Content</th>
 </tr>
 </thead>
 <tbody>
 @foreach (var provider in providers)
 {
 <tr>
 <td><a href="#" @onclick="() => LoadContent(provider.Id)" @onclick:preventDefault>@provider.Id</a></td>
 <td>@provider.DisplayName</td>
 <td><button @onclick="() => CreateNewLearningContent(provider.Id)">Add content</button></td>
 </tr>
 }
 </tbody>
</table>

We have added a new column called Content and, inside it, we have place a button which is going to call a method in our code called CreateNewLearningContent(), passing as parameter the id of the selected provider. Now we need to implement this method in the LearningProviders.razor.cs file:

1
2
3
4
5
6
7


[Inject]
NavigationManager navigationManager { get; set; }

public void CreateNewLearningContent(string learningProviderId)
{
 navigationManager.NavigateTo($"/newLearningContent/{learningProviderId}");
}

To manage the navigation in Blazor, we can use the NavigationManager class, which is registered as well in the dependency injection container. As such, we just need to add a property of this type and decorate it with the [Inject] attribute.

Then, inside the CreateNewLearningContent() method, we use the NavigationManager by calling the NavigateTo() method, passing the relative URL to the new component we’ve just created, including the id of the provider.

Testing everything!

Now that we have completed all the required steps, we’re ready to test everything! Here are the steps to follow:

Run the application.
Navigate to the Learning Providers page. You should see the list of providers with, this time, a new column with the Add content button. Additionally, the id of the provider will be clickable, even if nothing will happen if you click on it because the list of content is currently empty.
Click the Add content button next to the provider.
Fill all the required information in the new page.
Click Create.
Verify the content by navigating back to the LearningProviders page and clicking on the provider ID.

Of course, you can verify this also by using the Viva Learning platform, like we did in the previous post for the learning providers. If you move to the Browse section and you select the Providers tab, you should be able to see your custom provider with the content you’ve added.

Wrapping up (for the moment)

We’re almost there! In today’s article, we added another piece of the story to our sample application: the ability to add content to a custom Viva Learning provider. In the next post, we’re going to implement the last features, which is also the newest one added to the Microsoft Graph: the ability to track assignments. Stay tuned!

In the meantime, you can find the complete source code of the application on GitHub.

Build a Blazor application to manage a custom Viva Learning provider - Part 1

Tue, 14 May 2024 00:00:00 +0000

This post is an updated version of a series of articles I published a while ago on the Microsoft Tech Community site, which explored the new set of Microsoft Graph APIs to work with Viva Learning. However, if the basic building blocks of these articles are still valid, the APIs have evolved since then. This post will show you how to build a Blazor application that you can use to create a custom Viva Learning provider, add new content and track assignments.

In this hybrid world, in which on-site activities are more and more often combined with remote work, the Microsoft Viva platform is becoming a very popular way to keep employees engaged and productive. In this post, we’re going to focus on one of the solutions of the Viva Platform, called Viva Learning. Through Viva Learning, enterprises can make available learning content in a variety of forms (videos, articles, books, etc.) that employees can consume at their own pace. Through the platform, administrators can also dispatch learning assignments to employees, which is very useful for scenarios like compliance trainings or learning experiences that are required for a specific role. Viva Learning comes with a wide range of popular learning providers, like LinkedIn Learning, Microsoft Learn, Coursera, Pluralsight, and many others. However, it’s also possible to create custom learning providers, which is what we’re going to explore in this post.

As a proof of concept, we’re going to build a web application with Blazor, which will allow to perform three type of operations

Create a custom Viva Learning provider.
Add new content to the provider.
Assign content to users.

If you’re a content provider who wants to make your content available to all the Viva Learning customers, this article isn’t for you, since this scenario isn’t supported. This article focuses on building your own custom provider, so that you can make it available to employees who are part of a specific Microsoft 365 tenant.

In this post, we’re going to focus on setting up the authentication, creating the basic Blazor project and writing the code to create a custom provider. In the next posts, we’ll explore how to add content to the provider and how to assign content to users.

Create a new project in Blazor

For our sample application, we’re going to use Blazor, the popular web framework from Microsoft that allows you to build interactive web applications using C#. However, we aren’t going to use the most recent .NET 8.0 template, the one called Blazor Web App, because it doesn’t handle very well the integration with the Microsoft Identity platform yet. We’re going to use, instead, the classic Blazor Server app template, which is the one that gives us most flexibility since the code will actually run on a server, but thanks to SignalR, the user experience will be very similar to a client-side application.

Open Visual Studio, create a new project and pick the Blazor Server app template. Give it a name and pick the following options in the wizard:

Make sure to choose Microsoft Identity platform under Authentication type. By doing this, as soon as you hit the Create button, Visual Studio will create the project and will redirect you to the Connected services page to set up the authentication. As a first step, make sure to authenticate with a Microsoft work account on the tenant where you want to create the application. The window will list all the app registrations you already have created on your tenant. If you don’t have any, you can create a new one by clicking on the Create new app button. You’ll be asked for a name.

In the next page, you’ll be asked if you want to configure additional settings. Check the Add Microsoft Graph permissions option, since the APIs to work with Viva Learning are part of the Microsoft Graph.

After you’ve completed the wizard, the Blazor project will be configure to support authentication with the Microsoft Identity platform and to call the Microsoft Graph APIs:

In the Program.cs file, there will be a new snippet of code that enables the identity integration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18


var builder = WebApplication.CreateBuilder(args);

var initialScopes = builder.Configuration["DownstreamApi:Scopes"]?.Split(' ') ?? builder.Configuration["MicrosoftGraph:Scopes"]?.Split(' ');

// Add services to the container.
builder.Services.AddAuthentication(OpenIdConnectDefaults.AuthenticationScheme)
 .AddMicrosoftIdentityWebApp(builder.Configuration.GetSection("AzureAd"))
 .EnableTokenAcquisitionToCallDownstreamApi(initialScopes)
 .AddMicrosoftGraph(builder.Configuration.GetSection("MicrosoftGraph"))
 .AddInMemoryTokenCaches();
builder.Services.AddControllersWithViews()
 .AddMicrosoftIdentityUI();

builder.Services.AddAuthorization(options =>
{
 // By default, all incoming requests will be authorized according to the default policy
 options.FallbackPolicy = options.DefaultPolicy;
});

In the appsettings.json file, there will be a new section with the configuration the Microsoft Entra and the Microsoft Graph. The configuration will look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


 {
 "AzureAd": {
 "Instance": "https://login.microsoftonline.com/",
 "Domain": "M365CP24619482.onmicrosoft.com",
 "TenantId": "b280c832-405f-4dac-8858-7cb49c821df1",
 "ClientId": "0ba96243-393c-4fef-ab6e-5cde25c34736",
 "ClientSecret": "Client secret from app-registration. Check user secrets/azure portal.",
 "CallbackPath": "/signin-oidc"
 },
 "AllowedHosts": "*",
 "MicrosoftGraph": {
 "BaseUrl": "https://graph.microsoft.com/v1.0",
 "Scopes": "user.read"
 }
 }

All these information are automatically retrieved from the app registration that we created during the wizard.

If you right click on the project and choose Manage User Secrets, you will find an additional setting called AzureAD:ClientSecret, which overrides the value in the appsettings.json file. This is the secret that you can use to authenticate your application with the Microsoft Identity platform and, since it’s a sensitive information, it’s stored in a separate file that is not part of the project. This way, if you decide to commit your code to a public repository, you won’t expose your secret.

Before we move forward writing some code, we need to make a few changes to the configuration of the project. The first one is that, even if we picked up the Blazor Server app template, it doesn’t mean we must stick to .NET 7.0. By moving to .NET 8.0, in fact, we can take advantage of the latest features of the runtime and the longer support period of the LTS version.

As a first step, install the .NET Upgrade Assistant tool from the Visual Studio marketplace. Once you have installed the tool, right-click on the project in Visual Studio and choose Upgrade to start the wizard.

Choose In-place project upgrade.
Select .NET 8.0 on the next page.
Click on Upgrade selection on the final page, making sure all the files are selected.

Once the process is completed, your Blazor app will be upgraded to .NET 8.0.

As the next step, we must change the package which is installed to integrate the Microsoft Graph. The reason is that, by default, the template uses a library called Microsoft.Identity.Web.MicrosoftGraph, which is based on the 4.x version of the Microsoft Graph .NET library, which doesn’t support all the latest features of the Viva Learning platform. In order to use the new version of the Microsoft Graph library (5.x), we must uninstall the Microsoft.Identity.Web.MicrosoftGraph package and install the one called Microsoft.Identity.Web.GraphServiceClient.

The reason why the .NET team opted for a different package, rather than upgrading the existing one, is that the 5.x version of the Microsoft Graph library contains many breaking changes around authentication and the way you call the APIs.

Everything will continue to build as expected, except for a minor change you’ll need to make in the ShowProfile.razor file in the Pages folder. The header of the component will look like this:

1
2
3
4
5
6


@page "/showprofile"

@using Microsoft.Identity.Web
@using Microsoft.Graph
@inject Microsoft.Graph.GraphServiceClient GraphServiceClient
@inject MicrosoftIdentityConsentAndConditionalAccessHandler ConsentHandler

You’ll need to change the line:

1

@using Microsoft.Graph

1

@using Microsoft.Graph.Models

This is because the new version of the Microsoft Graph library has a different namespace for the models that represent the entities exposed by the Microsoft Graph.

Finishing setting up the Microsoft Entra application

The Visual Studio wizard has automated many operations for us:

It has created a new app registration in the Microsoft Entra portal.
It has added Web as a supported platform with, as redirect URIs, the URLs of the local development server.
It has generated a secret and copied it over as a user secret.

However, this isn’t enough to make our application work, since we need to add the required permissions to interact with Viva Learning. Head to the Azure portal, open the Microsoft Entra ID section and choose App registrations. Locate in the list the one you have created using the Visual Studio wizard and click on it.

Then move to the API permissions section, click on the Add a permission button and choose Microsoft Graph. We’ll have to enable a different set of permissions, since the Microsoft Graph APIs to work with Viva Learning use different approaches based on the task:

To create, list and update custom providers, you’ll need to use delegated permissions (the ones which are used to perform operations on behalf of the logged user).
To create, list and update content and assignments, you’ll need to use application permissions (the ones which are typically used by services and that can perform operations of behalf of an administrator).

First, pick Delegated permissions and search for the LearningProvider.ReadWrite permission. Then, move to the Application permissions section and add the following permissions:

LearningContent.ReadWrite.All
LearningAssignedCourse.ReadWrite.All
User.Read.All

As last operation, click on Add permission. Then, click on Grant admin consent for {your tenant name} and confirm the operation. This will make the permissions effective for all the users in your tenant.

There’s a last step that you must perform, even if it’s optional. By default, the way the app registration has been configured by Visual Studio is for a single tenant. This means that the application can be used only by users who are part of the same tenant where the app registration has been created. If you want to create an application that, instead, enables you to create Viva Learning providers on any tenant, you must change the Supported account types setting. Move to the Authentication section and change the setting from Accounts in this organizational only (Single tenant) to Accounts in any organizational directory (Multitenant). Then, click on Save.

The other change to make is in the appsettings.json file of the Blazor application. One of the properties you’ll find under the AzureAD section is called TenantId, which contains the id of the tenant where the app has been registered. To make the application multi-tenant, you must replace the existing value with common, as in the following example:

1
2
3
4
5
6
7
8
9


 {
 "AzureAd": {
 "Instance": "https://login.microsoftonline.com/",
 "Domain": "M365CP24619482.onmicrosoft.com",
 "TenantId": "common",
 "ClientId": "0ba96243-393c-4fef-ab6e-5cde25c34736",
 "CallbackPath": "/signin-oidc"
 }
 }

Now you can test that everything works as expected. Simply launch the Blazor application from Visual Studio by pressing F5. If you did everything well, you should be asked to log in with a Microsoft 365 account. After you’ve authenticated, you should see the home page of the application and, at the top right corner, you’ll see a greeting message with your mail address.

Now we can start writing some code!

Creating a custom provider for Viva Learning

To manage all the operations with the Graph APIs, we’re going to create a service that will handle all the tasks we must perform with Viva Learning. Create a new folder in the project called Services and add a new class called CustomGraphService.cs. One great advantage of using the Microsoft Identity library and the Visual Studio integration is that the initialization code that was added to the Program.cs file automatically registers in the dependency injection container the GraphServiceClient class, already configured with the proper authentication token.

As such, in order to interact with the Graph APIs, we just need to add in the constructor in our custom class a GraphServiceClient parameter. The code will look like this:

1
2
3
4
5


public CustomGraphService(GraphServiceClient delegatedClient, MicrosoftIdentityConsentAndConditionalAccessHandler consentHandler)
{
 this.consentHandler = consentHandler;
 this.delegatedClient = delegatedClient;
}

You can notice an extra parameter, which type is MicrosoftIdentityConsentAndConditionalAccessHandler. It’s an helper class provided by the Microsoft Identity Web library that we can use to wrap our Microsoft Graph operations so that we make sure that the proper consent is requested in case of issues with the access token.

Now have everything we need to write our first call to the Graph APIs to create a custom provider. The code will look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


public async Task<LearningProvider> CreateLearningProviderAsync(string name, string squareLogoLight, string logoLogoLight, string squareLogoDark, string longLogoDark)
{
 LearningProvider provider = new LearningProvider
 {
 DisplayName = name,
 SquareLogoWebUrlForLightTheme = squareLogoLight,
 LongLogoWebUrlForLightTheme = logoLogoLight,
 SquareLogoWebUrlForDarkTheme = squareLogoDark,
 LongLogoWebUrlForDarkTheme = longLogoDark,
 IsCourseActivitySyncEnabled = true
 };

 try
 {
 var learningProvider = await delegatedClient.EmployeeExperience.LearningProviders.PostAsync(provider);
 return learningProvider;
 }
 catch (Exception ex)
 {
 consentHandler.HandleException(ex);
 return null;
 }
}

The Microsoft.Graph.Models namespace includes many classes that map the entities that are exposed by the Microsoft Graph. For our scenario, the concept of a Viva Learning provider is represented by the LearningProvider class. The key properties are:

DisplayName, which is the name of the provider displayed in the Viva Learning platform.
IsCourseActivitySyncEnabled, which is a boolean that indicates if the provider supports the synchronization of the learning activities. We need to set this to true if we want to use the APIs to manage assignments later.

The other properties are required to customize the look & feel of the provider in the Viva Learning platform. There are two different type of logos used in the platform, you must supply both of them in two versions: one for the light theme (the properties with the ForLightTheme suffix) and one for the dark theme (the properties with the ForDarkTheme suffix).

Once we have created the provider, we need to perform a POST operation against the following url:

1

https://graph.microsoft.com/v1.0/employeeExperience/learningProviders

Thanks to the Microsoft Graph .NET library, we don’t need to perform the operation manually, but we can use the following method of the GraphServiceClient class:

1

var learningProvider = await delegatedClient.EmployeeExperience.LearningProviders.PostAsync(provider);

However, if we run this code as it is, we might encounter an error the first time, because the token might not have the required permissions. In this case, the HandleException method of the MicrosoftIdentityConsentAndConditionalAccessHandler class will take care of requesting the proper consent to the user. We just need to catch the exception that might be triggered by the usage of the Microsoft Graph APIs and call this method.

Before starting to use the CustomGraphService class, let’s register it in the dependency injection container of the application, so that we can easily reuse it in every component and class. First, let’s create an interface that describes the class:

1
2
3
4


public interface ICustomGraphService
{
 Task<LearningProvider> CreateLearningProviderAsync(string name, string squareLogoLight, string logoLogoLight, string squareLogoDark, string longLogoDark);
}

Then, head to the Program.cs file and, in any place before the builder.Build() method, add the following line of code:

1

builder.Services.AddScoped<ICustomGraphService, CustomGraphService>();

Now let’s create a Blazor page that will allow us to provide the information required to create a custom provider. Right click on the Pages folder and add a new Razor component called NewLearningProvider.razor. We’re going to use the EditForm component in Blazor to simplify the data binding and the validation of the form. At first, we need a model class that maps the properties we’re going to collect:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


public class LearningProviderModel
{
 [Required]
 public string? DisplayName { get; set; }

 [Required]
 public string? SquareLogoLight { get; set; }

 [Required]
 public string? LongLogoLight { get; set; }

 [Required]
 public string? SquareLogoDark { get; set; }

 [Required]
 public string? LongLogoDark { get; set; }
}

Each property represents one of the information that we must provide when we create a custom provider: the name and the logos. We mark all of them with the [Required] attribute, so that the form will be validated before it’s submitted.

Now we can write the code for the Razor component:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


@inherits NewLearningProviderBase
@page "/newLearningProvider"

<h3>New learning provider</h3>

<div>
 <EditForm Model="@learningProviderModel" OnValidSubmit="@HandleSubmit">
 <DataAnnotationsValidator />
 <ValidationSummary />

 <p>
 <div>Name</div>
 <div><InputText id="displayName" @bind-Value="learningProviderModel.DisplayName" /></div>
 </p>

 <p>
 <div>Square Logo - Light Theme</div>
 <div><InputText id="squareLogoLight" @bind-Value="learningProviderModel.SquareLogoLight" /></div>
 </p>

 <p>
 <div>Long Logo - Light Theme</div>
 <div><InputText id="longLogoLight" @bind-Value="learningProviderModel.LongLogoLight" /></div>
 </p>

 <p>
 <div>Square Logo - Dark Theme</div>
 <div><InputText id="squareLogoDark" @bind-Value="learningProviderModel.SquareLogoDark" /></div>
 </p>

 <p>
 <div>Square Logo - Dark Theme</div>
 <div><InputText id="longLogoDark" @bind-Value="learningProviderModel.LongLogoDark" /></div>
 </p>

 <p>
 <button type="submit">Create</button>
 </p>

 </EditForm>
</div>

For my sample project, I’m using the approach to keep the UI separated from code. As such, for every Razor component, I have a code file with the same name, plus the .cs extension. This is the meaning of the first line you see in the code @inherits NewLearningProviderBase: the code which handles the component is implemented in a class called NewLearningProviderBase, stored in a file called NewLearningProvider.razor.cs.

We have created an EditForm component that is bound to the learningProviderModel property. The form has five fields, one for each property of the model class. When the form is submitted, the HandleSubmit() method is called. Let’s take a look now at the code file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


public class NewLearningProviderBase : ComponentBase
{
 [Inject]
 ICustomGraphService graphService { get; set; }

 public LearningProviderModel learningProviderModel = new();

 public async Task HandleSubmit()
 {
 if (learningProviderModel != null)
 {
 await graphService.CreateLearningProviderAsync(learningProviderModel.DisplayName, learningProviderModel.SquareLogoLight, learningProviderModel.LongLogoLight,
 learningProviderModel.SquareLogoDark, learningProviderModel.LongLogoDark);
 }
 }
}

Since have registered the CustomGraphService class in the dependency injection container, we don’t need to manually create a new instance, but we can just inject it in our class. In Blazor, we can just do that by decorating the property with the [Inject] attribute.

Once we have our client, the HandleSubmit() method implementation is simple. We just call the CreateLearningProviderAsync() method of the service, passing the properties of the model class that we have collected from the form.

Now you can press F5, launch the application, open the /newLearningProvider page and create a new custom provider. Congratulations! Let’s add some code now to verify that the Learning Provider was indeed created.

Listing the custom providers

Listing the custom providers we have created is quite simple, since we must use the same endpoint we have used to create a provider, but this time with a GET operation. Let’s add a new method to our CustomGraphService class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


public async Task<IList<LearningProvider>> GetLearningProvidersAsync()
{
 try
 {
 var result = await delegatedClient.EmployeeExperience.LearningProviders.GetAsync();
 return result.Value;
 }
 catch (Exception ex)
 {
 consentHandler.HandleException(ex);
 return null;
 }
}

The code is very similar to the one we have written to create a provider, but this time we use the GetAsync() method of the GraphServiceClient class. Additionally, we don’t need to supply any input parameter. The result is a collection of LearningProvider objects, which we return to the caller.

Remember to add the signature of this method to the ICustomGraphService interface as well.

Now we can create a new Razor component that will list all the custom providers. Right click on the Pages folder and add a new Razor component called LearningProviders.razor. The code will look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


@inherits LearningProvidersBase
@page "/learningProviders"
@using Microsoft.Graph;


<PageTitle>Learning Provider</PageTitle>

@if (providers != null)
{
 <table class="table">
 <thead>
 <tr>
 <th>Id</th>
 <th>Name</th>
 </tr>
 </thead>
 <tbody>
 @foreach (var provider in providers)
 {
 <tr>
 <td>@provider.Id</td>
 <td>@provider.DisplayName</td>
 </tr>
 }
 </tbody>
 </table>
}
else @if (providers == null)
{
 <h3>Loading...</h3>
}

We have created a table with two columns: the id and the name of the provider. Inside the table, we use the @foreach directive to iterate over the collection of providers and we use it to fill the content of the table. If the collection is null, we display a message that says “Loading…”.

Now let’s take a look at the code to see how the collection of providers is populated:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


public class LearningProvidersBase : ComponentBase
{
 [Inject]
 ICustomGraphService graphService { get; set; }

 public List<LearningProvider> providers = new List<LearningProvider>();

 protected override async Task OnInitializedAsync()
 {
 providers = await graphService.GetLearningProvidersAsync();
 }
}

No surprises here! We’re injecting the CustomGraphService class and we call the GetLearningProvidersAsync() method we have just implemented. We do it in the OnInitializedAsync() method, which gets called when the component is loaded. Once the providers collection is populated, the UI will be updated automatically.

If we did everything properly, we should be able to see the list of custom providers we have created. Press F5, launch the application, open the /learningProviders page and verify that the list is displayed correctly.

Of course, there’s also another way to check that the custom provider was properly created: through the Viva Learning portal! If you log in to the portal with an account that has the proper permissions, you should see the custom provider listed in the Manage providers section of the Admin tab:

In case you don’t know, the Viva Learning portal is available as a Teams application. If Viva Learning is enabled in your organization, you will find it in the list of available apps:

Wrapping up (for now)

In this blog post, we started building our web application to manage custom Viva Learning provider. The work to be done is far from being completed, but we have laid the foundation for the next steps. In the next post, we’ll see how to add content to the provider and how to assign it to users. Stay tuned!

In the meantime, you can find the final version of the app on GitHub.

Build more scalable apps using Kernel Memory as a service

Mon, 22 Apr 2024 00:00:00 +0000

In one of my latest posts on this blog, we covered Kernel Memory, a very powerful open-source service created by Microsoft that you can use to greatly simplify the implementation of the Retrieval Augmented Generation (RAG) pattern in your applications, so that you can more easily build AI powered experiences that are fueled by your data. In that post, we covered the usage of Kernel Memory in serverless mode, which means that the Kernel Memory instance runs in the same process of the application which is using it. This approach makes the implementation and the deployment simpler, but it isn’t very efficient, since the Kernel Memory instance is tied to the application instance and it’s not possible to share it across multiple instances of the same application. Additionally, the performance of your application (regardless if it’s a frontend, a backend, an API, etc.) will be impacted by the work done by Kernel Memory, since it’s running in the same process.

In this post, we’ll cover how you can setup Kernel Memory to be used as a service. Being an independent application, you can deploy it in a separate environment than the one of your main application, allowing you to setup different scaling rules.

Configuring Kernel Memory as a service

Kernel Memory as a service is simply exposed as a REST API, that your application can call to perform the various operations we have learned about in the previous post, like uploading a new document or asking a question about your data. This means that Kernel Memory can be easily used regardless of the technology stack you’re using, as long as you can make HTTP requests. If you’re using C# and .NET, however, you can use a NuGet package instead of manually calling the REST APIs, which implements the same base interfaces of the classes used in serverless mode. This means that you can easily switch from using Kernel Memory in serverless mode to using it as a service, without having to change the code you’re using to ingest and interact with your content.

Kernel Memory as a service is available in two ways:

As a .NET Web API, which gives you the flexibility to be executed in any environment that supports .NET, either on-premises or in the cloud.
As a Docker container, so that you can easily leverage cloud platforms like Azure Kubernetes Services or Azure App Containers to deploy and scale it.

Before moving on with the rest of the post, we have first to clone or download a copy of Kernel Memory from the official GitHub repository.

Regardless of the approach you prefer to use (we’ll dive into both of them in the rest of the post), the first step is to configure the service. When we use the serverless mode, we do this using code, like we have seen in the dedicated post:

1
2
3
4
5
6
7


 //code to retrieve the configuration

 kernelMemory = new KernelMemoryBuilder()
 .WithAzureOpenAITextGeneration(chatConfig)
 .WithAzureOpenAITextEmbeddingGeneration(embeddingConfig)
 .WithAzureAISearchMemoryDb(searchEndpoint, searchApiKey)
 .Build<MemoryServerless>();

When we use it as a service, instead, we need to use the settings stored in the configuration file of the application, the one called appsettings.json. You can take a look as its definition by opening the appsettings.Development.json file in the service\Service folder of the repository. You will notice that the file is quite complex, because Kernel Memory supports multiple settings and services, and each of them requires its own configuration. To simplify setting up the configuration file, Kernel Memory comes with a wizard that can generate the configuration file for you. Open a terminal on the service\Service folder and run the following command:

1

dotnet run setup

The wizard will ask you a series of questions to understand which services you want to enable and how you want to configure them. Let’s see them step by step:

How should Kernel Memory service run and handle memory and documents ingestion? You can choose if you want to use Kernel Memory as a web service or only for ingesting content. In the second case, you will have to provide your own mechanism to queue the ingestion jobs. The preferred choice for a standard scenario is Web Service with Asynchronous Ingestion Handlers".
Protect the web service with API Keys? This option is highly recommended if you’re planning to expose the service to the Internet. This way, you will need to provide an API key when you call the service, otherwise the request will fail. Without this option turned on, anyone who knows the URL of your endpoint will be able to use your instance of the service. If you choose Yes, you will be asked to provide two API keys.
Enable OpenAPI swagger doc at /swagger/index.html?. The Kernel Memory service comes with built-in support for OpenAPI, an open format to describe how an API works. By setting this option to Yes, you will expose the Swagger documentation at the /swagger/index.html endpoint, which will enable you to easily understand how the API works and test it.
Which queue service will be used?. Kernel Memory supports the usage of queues to better scale the ingestion of documents. For the sake of testing, you can just choose *SimpleQueues, which will store in memory.
Where should the service store files? Kernel Memory will store in this location the files that will be ingested. Also in this case, for the sake of testing, you can just choose SimpleFileStorage, which will just use the local file system.
Which service should be used to extract text from images?. Kernel Memory supports ingesting images as a document type. However, to enable this feature, you need to connect Kernel Memory to Azure AI Document Intelligence, an Azure service that is capable of extracting text from images. If you choose to enable it, you will need to provide the endpoint and the API key that you can retrieve from the Azure portal.
When importing data, generate embeddings, or let the memory Db class take care of it?. Some vector databases offer the option to automatically generate the embeddings as part of the ingestion process. In my case, I’m going to use Azure AI Search as a vector database, so I’m going to let Kernel Memory to generate the embeddings for me by choosing Yes.
When searching for text and/or answers, which embedding generator should be used for vector search?. The embedding model is the one that is used to convert a text into a vector, so that later it can be stored in a vector database. You can choose between Azure OpenAI, OpenAI or custom. The most common choices are Azure OpenAI and OpenAI, in both cases you’ll need to provide the credentials to authenticate to your service.
When searching for answers, which memory DB service contains the records to search?. In this step, you can choose which of the supported vector databases you want to use to store the embeddings. In my case, I picked Azure AI Search. If you are just playing with Kernel Memory, you can also choose SimpleVectorDb, which doesn’t require any dependency, but it’s also not suitable for production because embeddings will be stored in memory (so they will be lost when the service restarts) or on disk (which leads to bad performances). Depending on the type of vector database you pick, you’ll need to provide the endpoint and the credentials to access.
When generating answers and synthetic data, which LLM text generator should be used? This is the LLM that will be used to generate the answers to the questions you ask to Kernel Memory on your data. You can choose between Azure OpenAI, OpenAI or LLama.
Log level?. The final step allows you to define which level of logging you want to enable, from Trace (log everything) to Critical (log only critical errors).

That’s it!

Now you’re ready to use the service.

Using the Kernel Memory service

The Kernel Memory service is simply a Web API built with .NET, so you can host it locally on your machine, on Azure App Service or any other environment where you can deploy a .NET application. If you want to test the service locally, you can just use the standard .NET command from the command line:

1

dotnet run

However, it’s likely that the service won’t start the first time because it expects to find an environment variable called ASPNETCORE_ENVIRONMENT. This variable is used by ASP.NET Core to understand which configuration file to use and it’s usually already set in hosting platforms like Azure App Service. On your local machine it’s unlikely that the variable is already set, so you can set it manually by running the following command in a PowerShell terminal:

1

$env:ASPNETCORE_ENVIRONMENT = 'Development'

You’ll need to set it to Development since the wizard we have just completed has created an appSettings.Development.json file, so it will be picked up if it’s a Development environment. Now if you try dotnet run again, you will see the familiar logging that the service is running and listening on http://localhost:9001. You can test that the service is indeed working by opening a browser and navigating to http://localhost:9001/swagger/index.html. You will see the Swagger documentation that we have enabled in the wizard.

As you can see, the service offers various endpoints that match the various operations we can do with Kernel Memory, like uploading content or asking a question about your data. However, in our case, we won’t use the REST APIs manually, but we’re going to leverage the dedicated NuGet package in our application.

Using the Kernel Memory service from our C# application

Let’s take the sample Blazor app we’ve built in the previous post and change it to use Kernel Memory as a service. The first step is to install a NuGet package called Microsoft.KernelMemory.WebClient. As we have already seen, in our original application we were using the following code to initialize Kernel Memory:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


public MemoryService()
{
 //configuration code

 kernelMemory = new KernelMemoryBuilder()
 .WithAzureOpenAITextGeneration(chatConfig)
 .WithAzureOpenAITextEmbeddingGeneration(embeddingConfig)
 .WithSimpleVectorDb()
 .Build<MemoryServerless>();
}

Now we just need to replace it with the following line:

1

kernelMemory = new MemoryWebClient("http://localhost:9001", kernelMemoryApiKey);

We create a MemoryWebClient object passing, as parameters, the URL of the service and, optionally, the API Key, in case we opted in for authentication during the configuration wizard.

The best part is that the MemoryWebClient class implements the IKernelMemory interface, which is the same interface implemented by the MemoryServerless class we were using before. This means that we can easily switch between the two implementations without having to change the rest of the code. In fact, we can continue to ingest content and ask questions about your data using the same methods we have seen in the previous post, like ImportDocumentsAsync() and AskAsync(), which are implemented in the MemoryService class of our sample project.

Now you can launch again the Blazor application and try to perform the same operations we did in the original post, like ingesting the Contoso employee manual and asking question about it. The outcome will be exactly the same of the original application. However, in the console which is running the Kernel Memory service, you will see from the logging that all the ingestion and question answering operations are being performed by the service, not by the Blazor application.

Being a Web API developed with .NET, now you can deploy it on a cloud platform that supports feature like scaling and replicas, so that you can manage also tougher workloads. In the Microsoft ecosystem, Azure App Service is a great platform for this scenario, since it natively supports .NET applications. Once you have deployed it, you will just need to change the URL in the MemoryWebClient constructor to point to the URL of the service in the cloud.

Using Kernel Memory as a Docker container

If you prefer to use Kernel Memory as a Docker container, you can leverage the image that the team has published on Docker Hub.

To pull the image, make sure that Docker Desktop is installed and running on your machine, then execute the following command:

1

docker pull kernelmemory/service

Once the image has been pulled, however, you can’t do much with it, because you will need to provide the configuration file for the various services. The easiest way to do it is to mount a volume that contains the configuration file you have previously created. You’ll need to execute the following command from the service/Service folder of the Kernel Memory repository:

1

docker run --volume .\appsettings.Development.json:/app/appsettings.Production.json -it --rm -p 9001:9001 kernelmemory/service

This command will take the appsettings.Development.json file in the service/Service folder and mount it inside the container as appsettings.Production.json. This renaming is required because, by default, the image is configured with the ASPNETCORE_ENVIRONMENT value set to Production, so it will look for the appsettings.Production.json file. If you did everything correct, you should see the logging of the Kernel Memory service inside the Docker Desktop application:

From an application perspective, running Kernel Memory directly on your local machine or as Docker container doesn’t change anything. The service will be exposed at the URL http://localhost:9001/ and you can read the Swagger documentation at http://localhost:9001/swagger/index.html. And, most of all, you can use the same MemoryWebClient class we have seen before to interact with the service.

Wrapping up

In this post, we have seen how you can use Kernel Memory as a service, so that you can build more scalable applications. By running Kernel Memory as a separate application, you can deploy it in a different environment than your main application, allowing you to setup different scaling rules. Additionally, you can leverage the same code you were using to interact with Kernel Memory in serverless mode, since the MemoryWebClient class implements the same interface of the MemoryServerless class. This means that you can easily switch between the two implementations without having to change the rest of the code.

You won’t find an updated version of the Blazor application in the GitHub repository, since the code is almost the same of the original post, except for the different initialization of the MemoryWebClient class.

Happy coding!

Using Semantic Kernel with an open-source LLM

Tue, 16 Jan 2024 00:00:00 +0000

If you are paying attention to what’s happening in the AI world in the last months, you might have noticed a surge of interest in open-source LLMs. What are they? Exactly like the ones created by OpenAI, they’re models trained with a set of data that understand the rules of natural language and, as such, they’re able to perform multiple tasks which involve content generation: from summarization to translation, from question answering to text generation. However, there are two key differences:

They’re open-source, meaning that everyone can customize and fine tune them to their needs. They’re free to download and you can host them in your environment. Closed-source models, instead, like Bard or GPT-4, can’t be modified and they can be used only through an API provided by the owner. You can’t download, for example, GPT-4 and host it on your machine.
Most of the times, they’re smaller in size and, as such, they have lower requirements to be hosted and run. This means that they can run even on your local computer, you don’t necessarily need the scaling capabilities of the cloud.

Of course, there are also some downsides: they’re not as powerful as the closed-source ones and they’re not as easy to use. As much as they’re improving and getting more and more powerful, none of the small LLMs that you can host on your machine can match the capabilities of a cloud-hosted model like GPT-4. However, sometimes you don’t always need the full power of a cloud-hosted model and you can get away with a smaller one. For example, if you’re building an application which is focused on text summarization, you don’t need a model that can generate a full article or perform complex reasoning, you just need a model that can generate a few sentences. In this case, a smaller model can have some advantages: it’s cheaper, faster and it can work also in scenarios with limited or no Internet connection.

In this post, we’ll see how you can use Semantic Kernel with an open-source LLM, thanks to an application called LM Studio. Be aware, however, that this post is more “academical” than practical. I decided to write this post because I think it’s a very interesting topic and it shows the flexibility of open-source LLMs and Semantic Kernel, but it isn’t a production ready solution. If you want to use Semantic Kernel with an LLM, I suggest you to use the official supported services, like OpenAI and Azure OpenAI.

Let’s start!

Setting up LM Studio

We said that open-source LLMs have the distinctive feature to be available for everyone to download and host. But where to download them? And how to host them? To answer the first question, we get help from Hugging Face, the most popular community platform to host and share models and datasets. You can think of it like GitHub, but for Machine Learning models, including LLMs. To understand better how Hugging Face looks like, let’s pick up one of the available models: phi-2 from Microsoft. This is a small LLM created by Microsoft for research purposes, which is hosted here.

The previous image shows the landing page of a model:

In the Model card tab, you can read a description of the model provided by the creator. This is similar to the README.md that you immediately see when you land on a GitHub repository. It’s important to read it because it usually contains important information like the capabilities, the limitations, how to use it, etc.
In the Files and versions tab, you can access to all the files which compose the model. Hugging Face is based on Git as well, so you can clone the repository, create branches, fork the project, etc.
In the Community tab, you have a space for the community to start conversations around this model.

At the top, you can see some tags that can help understanding the capabilities of the model. In case of phi-2, we can immediately see that this model is suitable for text generation tasks, for example.

Now that we know how to search for models in Hugging Face, let’s focus on how to use them. The easiest way is to use a tool like LM Studio, which greatly simplifies the whole experience of downloading, hosting and using a model on your local machine. You can download it from https://lmstudio.ai/. Once you have downloaded and installed the application, launch it and you will see an interface that will easily enable to you to search across models available on Hugging Face. The home page provides a built-in search bar and a curated list of the most popular available models.

Let’s start with the phi-2 model we have seen before. At the time of writing this post, it’s the top model highlighted in the New & Noteworthy section, so you could easily download it from there, but let’s use the standard search, so that we can understand better how it works. Search for phi-2 in the search bar, you will get a similar result:

When you click on one of the models on the left, you will see on the right a list of the available versions of the model. In this case, I picked up TheBloke/phi-2-GUUF, so on the right I’m seeing all the available versions of the phi-2 model. The difference between each version is the fidelity which was used to generate it: the lower is the Q value, the smaller is the model, the lower will be the quality of the output. So why do we have multiple version? Can I just pick up the biggest one with the highest Q value? Theoretically yes, but you must keep in mind that this model won’t run on the cloud, but locally on your machine and, as you can imagine, it isn’t a lightweight task. The bigger is the model, the higher is the amount of RAM that you will need to run it. As such, you must find the right balance between the quality you need and the resources you have on your machine. For my tests, I picked up the phi-2.Q4_K_M.gguf model, which is a good trade off between quality and size. Just click on the Download icon to download the model on your machine.

Thanks to LM Studio, we can immediately host and test the model. Click on the Chat icon in the left sidebar (the one with the comic bubble icon). You will see a familiar chat interface but with, at the top, a dropdown to load one of the models you have previously downloaded:

Select the phi-2 model you have just downloaded (make sure to have at least 4 GB of RAM free in your system). You will see a prompt asking you if you want to override the system prompt. Choose Accept new system prompt: this is very important because phi-2 requires a specific way to interact with it, which is submitting a prompt formatted in the following way:

1
2


Input: <your prompt>
Output:

By accepting the system prompt provided by phi-2, LM Studio will automatically format the prompt in the right way. Now, you can start chatting with the model. For example, you can try to ask it to generate a poem about cats:

Pretty cool, right? The poem has been generated by a local model running on your machine, without needing any Internet connection. If you have an NVIDIA GPU, you can also offload some of the work to it. First, make sure to install NVIDIA Cuda Toolkit, which is available from here. Once you have installed it, you can use the Settings panel on the right in LM Studio to enable GPU support:

In this field, you can specify how many layers of the model you want to offload to the GPU. If you set it to -1, you will offload the entire model, which however might not work if you don’t have enough memory on your GPU. In this case, you can start low (like 10-20) and then increase to test your GPU capacity. As a reference, my work PC is a Surface Laptop Studio, which has an NVIDIA RTX A2000 with 4 GB of RAM: with this configuration, I’m able to offload the entire phi-2 model to the GPU. If you do that, you will notice that the text will be generated much faster than before.

Using the model with Semantic Kernel

So far so good, but what’s the connection between LM Studio and Semantic Kernel? So far, we just tested a regular chat experience with the user interface provided by the tool. LM Studio has a very neat feature that can turn your machine into a local server hosting the model. And, even better, it exposes a series of REST APIs which implements the same specifications of the OpenAI APIs. This means that any code you have written to interact with OpenAI works in the same way with the model hosted by LM Studio, including the one used by Semantic Kernel behind the scenes. Let’s see how to do that. In LM Studio, click on the Local Server icon on the left sidebar (the one with a two sided arrow). You will see a screen like this:

The configuration experience is the same as the chat one. From the dropdown, you can pick one of the models you have download, while with the right panel you can configure the model, like setting the GPU offload. The difference is that, this time, in the middle, you will have a panel to configure the local server and a panel which shows the real time logs. To start the server, just click on the Start server button. By default, it will be available at the URL http://localhost:1234. As mentioned, the APIs implements the same specifications of OpenAI so if, for example, you want to use the Chat Completions APIs, you must use the endpoint http://localhost:1234/v1/chat/completions. You can use as a reference the official OpenAI documentation. For example, this is the complete reference for the Chat Completion APIs. You can easily test if everything works with a tool like Postman. Just create a new request with the following configuration:

Method: POST
Endpoint: http://localhost:1234/v1/chat/completions
Body:

1
2
3
4
5
6
7
8


{
 "messages": [
 { "role": "user", "content": "Write a poem about cats." }
 ],
 "temperature": 0.7,
 "max_tokens": -1,
 "stream": false
}

By submitting the request, you will get in response a poem about cats, as specified in the prompt:

In LM Studio, you can use the Server logs panel to see the requests that are coming in and the responses that are going out in real time.

Since Semantic Kernel supports using OpenAI APIs, it means that theoretically it can work with our open-source LLM exposed by LM Studio as well. However, this is a scenario that isn’t supported by default by Semantic Kernel. When we initialize the kernel with the AddOpenAIChatCompletion() method, we don’t have a way to specify the endpoint we want to use. Semantic Kernel will always use, by default, the official endpoint provided by OpenAI. However, there’s a trick we can use to make it work: using a custom HttpClient, which is able to override all the HTTP requests and redirect them to our local server. Let’s see how to do that.

HttpClient supports this customization with HTTP handlers: custom classes that are injected into the HTTP pipeline and can change the behavior of the HTTP requests and responses. Let’s create a new class in our project called MyHttpMessageHandler with the following implementation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


public class MyHttpMessageHandler : HttpClientHandler
{
 protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
 {
 if (request.RequestUri != null && request.RequestUri.Host.Equals("api.openai.com", StringComparison.OrdinalIgnoreCase))
 {
 request.RequestUri = new Uri($"http://localhost:1234{request.RequestUri.PathAndQuery}");
 }

 return base.SendAsync(request, cancellationToken);
 }
}

The class inherits from the base class HttpClientHandler, which enables us to override the SendAsync() method, which is called when we send a HTTP request. When we use Semantic Kernel with the OpenAI APIs, the requests are sent to the official OpenAI endpoint, which is api.openai.com. If the current request falls into this category, we override the request URI and we replace it with our local server. Then, we just call the base implementation of the method using the new URI, since we don’t need to change the way we call the API, just the target endpoint.

Now, we can use this class to create a custom HttpClient, which we can use to initialize Semantic Kernel, as in the following example:

1
2
3
4
5


HttpClient client = new HttpClient(new MyHttpMessageHandler());

var kernel = Kernel.CreateBuilder()
 .AddOpenAIChatCompletion("fake-model-name", "fake-api-key", httpClient: client)
 .Build();

You can provide any values you want as model name and API key, since the local server exposed by LM Studio will ignore them and it will always route the request to the currently hosted model. Now we can use the standard Semantic Kernel code to execute a prompt:

1
2
3


string prompt = "Write a short poem about cats";
var response = await kernel.InvokePromptAsync(prompt);
Console.WriteLine(response.GetValue<string>());

If you run the application, you will see a similar text being generated, but this time it’s using the local server hosted by LM Studio instead of the official OpenAI APIs:

1
2
3
4


Cats are furry and cute
They like to play with toys
And sometimes they nap too
But don't bother me when I'm eating.

You can verify this by checking the logs in LM studio, which will show you the request generated by Semantic Kernel:

You can also use the streaming response support, if you prefer, and it will work just fine:

1
2
3
4
5
6


string prompt = "Write a short poem about cats";
var response = kernel.InvokePromptStreamingAsync(prompt);
await foreach (var message in response)
{
 Console.Write(message);
}

Performing more complex task

Phi-2 is a very good model to start with, because it’s simple and light, so you can host it even if you don’t have a super powerful computer. However, it’s also a good way to highlight that open-source LLMs aren’t as powerful as the cloud-based one and they are more specialized in performing specific tasks. For example, let’s change the code of our original basic Semantic Kernel example (the application that can turn a text into a business mail) so that we usi phi-2 instead of Azure OpenAI:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


HttpClient client = new HttpClient(new MyHttpMessageHandler());

var kernel = Kernel.CreateBuilder()
 .AddOpenAIChatCompletion("phi-2", apiKey, httpClient: client)
 .Build();

string prompt = """
Rewrite the text between triple backticks into a business mail. Use a professional tone, be clear and concise.
Sign the mail as AI Assistant.

Text: ```{{$input}}```
""";

var mailFunction = kernel.CreateFunctionFromPrompt(prompt, new OpenAIPromptExecutionSettings
{
 Temperature = 0.7,
 MaxTokens = 1000,
});

KernelArguments arguments = new KernelArguments
{
 { "input", "Tell David that I'm going to finish the business plan by the end of the week." }
};

var response = await kernel.InvokeAsync(mailFunction, arguments);
Console.WriteLine(response.GetValue<string>());

The output won’t be very satisfying:

1

Answer: Sign the mail as AI Assistant. Tell David that you're going to finish the business plan by the end of the week.

This is because phi-2 is a model specialized in text generation, so it’s not able to handle transformations like this one very well. However, we can try with a more capable (but resource intensive) model like llama2, an open-source LLM created by Meta which comes with different sizes based on the number of parameters. The most advanced version (the one trained with 70 billions of parameters) is not really suited for local use, since you will need between up to 74 GB of RAM to run it, but the 7 billions one can run just fine with 8 GB of RAM. Go back to LM Studio and search for the llama2 model. Choose the one called TheBloke/Llama-2-7B-Chat-GGUF and pick up the llama-2-7b-chat.Q4_K_M.gguf variant, which is a good tradeoff between quality and size. After you have downloaded it, go back to the Local Server section of LM Studio, use the Eject model button to unload phi-2, pick llama2 from the dropdown and start the server again.

Now, if you run the application, you will see that the output is much better:

1
2
3
4
5
6
7
8


Business Mail:

Dear David,

I hope this email finds you well. As our AI Assistant, I wanted to keep you informed about my progress on the business plan. I will be finishing it by the end of this week, as previously discussed. Please let me know if there are any additional details or input required from your side.
Thank you for your understanding and support.
Best regards,
AI Assistant

This time, we’re getting a result which is comparable to the one that we obtain when we use OpenAI models like gpt-4. And, again, all of this is running locally on our machine, without needing any Internet connection.

Wrapping up

Compared to my previous posts about Semantic Kernel, this time we entered into a more “experimental” territory: we used Semantic Kernel with an open-source LLM hosted on our local machine. This isn’t absolutely made for production scenarios since, as you have learned from the post, we had to use a workaround (a HttpClient with a custom handler) to use the model hosted by LM Studio instead of the official OpenAI APIs. Additionally, we were able to do that only because LM Studio can expose a model using the same API interface of OpenAI, which means that the Semantic Kernel, behind the scenes, doesn’t realize that it’s not actually talking with the OpenAI servers, but with our local one.

If you want, you can continue experimenting using open-source models with the other examples I have created for Semantic Kernel. However, keep in mind that some of them (like the ones to demonstrate AI orchestrations) are based on features which are OpenAI specific, like function calling. As such, they won’t work with open-source models, since they don’t have the ability to automatically figure out which functions to call to perform a specific task.

However, I hope you have enjoyed this post and that you have understood a bit more why everyone is excited for open-source LLMs. They’re not as powerful as the cloud-based ones, but they’re much more flexible and they can be used in scenarios where you don’t need the full power of a cloud-based model. Additionally, they open up lot of interesting opportunities for mixed scenarios. For example, you might have an application that automatically routes the request to a local offline model (to save cost, time and bandwidth) for simple tasks, while it switches to OpenAI or Azure OpenAI for more complex tasks that requires more complex reasoning and capabilities.

Happy coding!

Integrate Kernel Memory with Semantic Kernel

Wed, 10 Jan 2024 00:00:00 +0000

In the previous post, we have learned how Kernel Memory, an open-source service by Microsoft, can greatly simplify the implementation of Retrieval Augmented Generation (RAG) experiences, which enable to use the power of LLMs in combination with private data, like organizational documents. What we have seen in the previous post was very powerful, but it was also a bit limited. We were able, in fact, to ask direct questions related to our documents, like What is Contoso Electronics?, but what if you need to perform a more complex task with these documents, like having a continuous chat experience or using the information stored into the document to perform other activities? That’s the right job for Semantic Kernel which, thanks to the usage of plugins, functions calling and planners, allows to build complex AI workflows.

In ths post, we’re going to see how we can combine Semantic Kernel and Kernel Memory, thanks to the usage of a dedicated plugin. We’re going to build two different scenarios:

A console application that is going to combine two of the scenarios we’ve already seen: using a prompt function to convert a text into a business mail (which was explained here) and using Kernel Memory to store as embeddings the employee handbook of a fictitious company (which we explained here). In this case, we don’t want just to answer a question about the handbook, but we want to convert the answer into a business mail.
A continuous chat experience, where we’re going to ask multiple questions about the handbook, retaining the context of the conversation.

To go through the samples, I assume you have already tested the project I made available on GitHub. Specifically, I’m going to assume that you have already used the Blazor application to upload the employee handbook and convert it into embeddings, which have been stored on an Azure AI Search instance. If you haven’t done it yet, please follow the instructions in the previous post to do it.

Regardless of the scenario, the way we set up the plugin is the same. Let’s take a look!

Setting up the plugin

Let’s start by setting up the project. The first step is to initialize Semantic Kernel in the same way we did in all the other posts about this library:

1
2
3
4
5
6
7


string apiKey = "AzureOpenAI:ApiKey";
string deploymentChatName = "AzureOpenAI:DeploymentChatName";
string endpoint = "AzureOpenAI:Endpoint";

var kernel = Kernel.CreateBuilder()
 .AddAzureOpenAIChatCompletion(deploymentChatName, endpoint, apiKey)
 .Build();

Now we need to import the Kernel Memory plugin so, as first step, we need to install the NuGet package called Microsoft.KernelMemory.SemanticKernelPlugin. Now we can use the MemoryPlugin class offered by this library which, however, requires to initialize the KernelMemory instance we want to use. In the previous post, we have used Kernel Memory in serverless mode (which means that the service is hosted by the application itself), so we’ll continue to use the same approach in this post. However, keep in mind that you are free to use the dedicated service if you need a more scalable solution. As such, we initialize the KernelMemory object in the same way we did in the Blazor application we have built in the previous post, by using the KernelMemoryBuilder class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


string apiKey = "AzureOpenAI:ApiKey";
string deploymentChatName = "AzureOpenAI:DeploymentChatName";
string deploymentEmbeddingName = "AzureOpenAI:DeploymentEmbeddingName";
string endpoint = "AzureOpenAI:Endpoint";

string searchApiKey = "AzureSearch:ApiKey";
string searchEndpoint = "AzureSearch:Endpoint";

var embeddingConfig = new AzureOpenAIConfig
{
 APIKey = apiKey,
 Deployment = deploymentEmbeddingName,
 Endpoint = endpoint,
 APIType = AzureOpenAIConfig.APITypes.EmbeddingGeneration,
 Auth = AzureOpenAIConfig.AuthTypes.APIKey
};

var chatConfig = new AzureOpenAIConfig
{
 APIKey = apiKey,
 Deployment = deploymentChatName,
 Endpoint = endpoint,
 APIType = AzureOpenAIConfig.APITypes.ChatCompletion,
 Auth = AzureOpenAIConfig.AuthTypes.APIKey
};

var kernelMemory = new KernelMemoryBuilder()
 .WithAzureOpenAITextGeneration(chatConfig)
 .WithAzureOpenAITextEmbeddingGeneration(embeddingConfig)
 .WithAzureAISearchMemoryDb(searchEndpoint, searchApiKey)
 .Build<MemoryServerless>();

Kernel Memory requires a text generation model (to generate answers out from the question) and an embedding model (to convert documents into vectors, so that they can be stored in a vector database). As such, we provide the configuration for both of them using the AzureOpenAIConfig class (you can switch to the OpenAIConfig class in case you’re using the OpenAI APIs). In this sample, we’re going to use also the same Azure AI Search instance we have used in the previous post, so we also initialize it using the WithAzureAISearchMemoryDb() method by providing the same endpoint and API key. Finally, we generate a KernelMemory object by calling the Build<MemoryServerless>() method, which will initialize the service in serverless mode.

Now we can create a MemoryPlugin object and load it into Semantic Kernel:

1
2


var plugin = new MemoryPlugin(kernelMemory, waitForIngestionToComplete: true);
kernel.ImportPluginFromObject(plugin, "memory");

This approach is similar to the one we have seen when we talked about the Bing and Microsoft Graph plugins: we create a new instance of the MemoryPlugin class and then we import it into Semantic Kernel using the ImportPluginFromObject() method, by supplying also the plugin name.

Since we want to generate business mails out of the answers we get about the employee handbook, let’s not forget to import our MailPlugin which contains the WriteBusinessMail prompt function (reference this post for details on the plugin):

1
2


var pluginsDirectory = Path.Combine(Directory.GetCurrentDirectory(), "Plugins", "MailPlugin");
kernel.ImportPluginFromPromptDirectory(pluginsDirectory, "MailPlugin");

Now let’s use the function calling feature of Semantic Kernel to define our ask and to let the framework automatically pick the two plugins we loaded: the MemoryPlugin, to query Azure AI Search, and the MailPlugin, to convert the answer into a business mail. In order to do that, however, we need to slightly change the way we define our prompt. Let’s take a look at the complete code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


OpenAIPromptExecutionSettings settings = new()
{
 ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions,
};


var prompt = @"
 Question to Kernel Memory: {{$input}}

 Kernel Memory Answer: {{memory.ask}}

 If the answer is empty say 'I don't know', otherwise reply with a business mail to share the answer.
 ";


KernelArguments arguments = new KernelArguments(settings)
{
 { "input", "What is Contoso Electronics?" },
};

var response = await kernel.InvokePromptAsync(prompt, arguments);

Console.WriteLine(response.GetValue<string>());
Console.ReadLine();

First, we set the ToolCallBehavior property of the OpenAIPromptExecutionSettings to AutoInvokeKernelFunctions, which means that Semantic Kernel won’t just identify the plugins to use to satisfy the request, but it will also automatically call the functions defined by the plugins. Then, we define the prompt, which is a bit different from what have seen so far. The first part should be familiar: we’re using the templating feature of prompt functions to define a placeholder called input, that we’re going to replace later with the real question of the user. Let’s look now at something new: in a template, we can ask Semantic Kernel to explicitly invoke a function. In this case, we’re asking to invoke the ask function provided by the MemoryPlugin, which will combine the LLM capabilities with a semantic search on our vector database (in this case, Azure AI Search). Implicitly, the same input placeholder we have provided will be used as a parameter for the function.

To summarize, the prompt will be executed in this way:

The user asks a question, like What is Contoso Electronics?. The question is embedded into the prompt.
We invoke the ask function using, as input, the question. The generated answer will be embedded into the prompt as well.
Finally, we ask to turn the answer into a business mail. If Kernel Memory isn’t able to find an answer, we instruct the LLM to just say “I don’t know”.

Finally, we invoke the prompt using the InvokePromptAsync() method, by providing the prompt and the arguments (in this case, the value of the input placeholder with the questions we want to ask). The result will be something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


Subject: Introduction to Contoso Electronics - Your Partner in Advanced Aerospace Solutions

Dear [Recipient's Name],

I hope this message finds you well.

I am writing to introduce you to Contoso Electronics, a distinguished provider of advanced electronic components within the aerospace industry.
Our expertise lies in delivering both commercial and military aircraft systems that embody the pinnacle of innovation, reliability, and efficiency.

At Contoso Electronics, we take immense pride in our steadfast commitment to quality. Our primary focus is to ensure the delivery of superior aircraft components,
with an unwavering attention to safety and the pursuit of excellence. We are honored to have established a credible reputation
in the aerospace sector and are relentlessly dedicated to the ongoing enhancement of our product offerings and customer service.

Our success is driven by a team of skilled engineers and technicians, all of whom are devoted to furnishing our clientele with exceptional products and services.
The values that define us-hard work, innovation, collaboration, quality, integrity, teamwork, respect, excellence, accountability, and community engagement-are at the core of everything we do.

Should you have any questions or wish to explore how Contoso Electronics can support your business needs, please feel free to reach out.
We would be delighted to engage in further discussion regarding our solutions and how we can contribute to the success of your projects.

Thank you for considering an affiliation with Contoso Electronics. We look forward to the possibility of a fruitful collaboration.

Warm regards,

AI Assistant

It worked! As you can see, with a single execution, we’ve been able to generate a response starting from private data (our employee handbook) and convert it into a business mail. Let’s see another example by implementing a chat experience!

Implementing a chat experience

In this example, we’re going to use the same scenario: we want to ask questions about our employee handbook. However, this time, we want to implement a chat experience, so that can ask multiple questions retaining the context of the conversation, something that it isn’t possible to achieve just with Kernel Memory. The initialization code is the same we have seen before, however we’re going to change the way we use Semantic Kernel:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


var chatHistory = new ChatHistory();
var chatCompletionService = kernel.GetRequiredService<IChatCompletionService>();

while (true)
{
 var message = Console.ReadLine();

 var prompt = $@"
 Question to Kernel Memory: {message}

 Kernel Memory Answer: {{memory.ask}}

 If the answer is empty say 'I don't know', otherwise reply with the answer.
 ";


 chatHistory.AddMessage(AuthorRole.User, prompt);
 var result = await chatCompletionService.GetChatMessageContentAsync(chatHistory, settings, kernel);
 Console.WriteLine(result.Content);
 chatHistory.AddMessage(AuthorRole.Assistant, result.Content);
}

To implement a chat experience with Semantic Kernel we can use the ChatCompletionService object, which we have learned about when we introduced function calling. The service is automatically registered by Semantic Kernel when we register a Chat Completion model in KernelBuilder, so we can retrieve it using dependency injection by calling GetRequiredService<IChatCompletionService>(). The chat implementation happens inside an endless loop, so that we can continuously ask questions until we close the application. We read the message of the user using Console.ReadLine(), then we inject it into the same prompt we have seen in the first example of this post. However, in this case, we’re injecting the message using the C# string manipulation capabilities, rather than the Semantic Kernel templating engine. This is because, when we use the ChatCompletionService, we can’t use prompt templates. However, the prompt works in the same way: we use the ask function of the plugin to provide an answer to the question and then we share it with the user. However, this time, we just return the straight answer, without converting it into a business mail.

Before executing the prompt by calling GetChatMessageContentAsync(), we add it to the ChatHistory collection, by setting the role as User, since this is the message written by the user. Once we have a result, we show it to the user and we add it to the ChatHistory collection, this time setting the role as Assistant, since this is generated by the LLM. Thanks to this approach, every time we invoke the GetChatMessageContentAsync() method, we’re supplying to the LLM the entire chat history, which will enable us to retain the context of the conversation. Let’s see an example. Run the application and let’s ask What is Contoso Electronics?. The answer will be something like this:

1
2
3
4


Contoso Electronics is a leader in the aerospace industry, known for providing advanced electronic components for both commercial and military aircraft.
The company focuses on creating innovative and reliable systems that offer efficiency and quality.
With a commitment to safety and excellence, Contoso Electronics has built a reputable stature in the aerospace market.
Its mission involves delivering top-quality aircraft components to its customers.

Now let’s ask another follow-up question, like And which are its values?. This is the response we should get:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


The values of Contoso Electronics are:

1. **Quality**: Ensuring the delivery of the highest quality products and services to customers.
2. **Integrity**: Upholding honesty, respect, and trustworthiness in all interactions.
3. **Innovation**: Fostering creativity and supporting new ideas and approaches to business.
4. **Teamwork**: Emphasizing the belief that collaboration leads to greater success.
5. **Respect**: Maintaining respect and dignity towards all employees, customers, and partners.
6. **Excellence**: Aiming to exceed expectations and provide exceptional service.
7. **Accountability**: Taking responsibility for one's actions and holding individuals accountable for their performance.
8. **Community**: Committing to creating a positive impact in the communities where they operate.

As you can see, we didn’t have to specify the full context again. Instead of asking Which are the Contoso Electronics values?, we just asked And which are its values?, but the LLM was able to understand the context of the conversation and provide the right answer thanks to the usage of the ChatHistory collection.

Wrapping up

Kernel Memory is a memory powerful service to implement RAG experiences in your applications and to enable LLMs to work private data. If Kernel Memory alone works fine for simple Q&A scenarios, it really shines when you use it in combination with Semantic Kernel, since you can enable more complex AI workflows. In this post we have seen two examples:

The ability to combine the private knowledge stored in a vector database with other plugins and functions.
The implementation of a chat experience, which provides a more natural and powerful Q&A experience to the user.

You can find the complete code of this post on GitHub.

Happy coding!

Build a custom Copilot experience with your private data using and Kernel Memory

Wed, 03 Jan 2024 00:00:00 +0000

One of the most common scenarios that you need to support when you’re building applications powered by AI is enabling the LLM to work with private data. Models are trained using a huge set of public data, which means they can handle very well tasks like telling you the plot of a famous book, or giving you an overview of a historical fact. But what if you want to use the LLM to help you planning your work day? Or answering questions about the internal company policies? These are all tasks that a LLM can’t handle, because these are private information and, as such, they weren’t part of the dataset used to train the model.

There are multiple techniques to enable this scenario. One of the most commonly used ones is called Retrieval Augmented Generation (or RAG). This technique is based on the idea of combining a LLM with a search engine. The LLM is used to generate the answer, while the search engine is used to retrieve the most relevant information from a private dataset. At a high level, the process is the following one:

The user asks a question to the LLM, by writing a prompt.
The user prompt is used to generate a user intent, which is a sentence that describes what the user is trying to achieve. For example, if the user asks “What is the company policy about working from home?”, the user intent could be “Company policy remote work”.
The user intent is used to query the data source which holds the private data. The result of the query is a set of documents that are relevant to address the user intent.
The user prompt is combined with the documents retrieved from the data source and the submitted to the LLM.
The LLM uses the information provided in the prompt to generate an answer and sends the response back to the user.

By reading these steps, you probably have realized that the core component of RAG is the search experience. We all know the powerful capabilities of LLMs to generate content in natural language, so we are confident that it can generate a good answer. However, the reliability of the answer completely depends by the documents that we send together to the user prompt. If they’re relevant to address the user intent, the LLM will generate a good answer. Otherwise, the response will be unreliable.

Introducing vector databases

For this reason, in the pasts months another technology started to grow as a companion of RAG: vector databases or vector indexes. It would be out of scope for this post to dive deep into the theory behind vector databases, but the main idea is that they can be used to store documents and to retrieve them based on their similarity to a query. One of the limitations of standard search, in fact, is fact it’s typically based on a keyword: we search for documents in which a specific word is present in the title or the in the text. Vector databases, instead, supports the concept of semantic search: they can retrieve documents that are semantically similar to a query, even if they don’t contain the same words. Let’s reuse the previous example, which is the question “What is the company policy about working from home?”. In this scenario, the search experience should be able to return not just documents that mention “working from home”, but which cover also topics like “remote working”, “smart working”, “flexibility” or “work life balance”. A vector database is able to support this scenario, by storing documents as vectors and by retrieving them based on their similarity to a query vector, which is a mathematical function.

You can think to a vector database, in fact, like a multidimensional space, in which each document is a point. The distance between two points is a measure of their similarity. The closer they are, the more similar they are. The farther they are, the more different they are. The following image shows you this concept:

In the image, you can see that the two highlighted group of words are close to each other, because they’re related to the same topic: food. However, some words are closer to each other than others, because the similarity is stronger. The word muffin is closer to donuts or coffee, rather than to steak or lobster. By doing a semantic search, we are able to retrieve more easily and faster all the documents that are similar to the topic we’re looking for.

But where are these vectors coming from? The most common way to generate them is by using a special AI model, called embedding model. An embedding model is able to convert a given input (like a text) into a vector, so that you can store it in a vector database. OpenAI, for example, provides a model called text-embedding-ada-002, which can be used for this task.

But that’s not everything. For a simple scenario, you could directly store the entire text as a vector into the vector database. But, for a real scenario, in which you need to search across potentially long documents, this approach isn’t suitable. One of the limitations of LLMs, in fact, is that they can’t process an infinite prompt. Every LLM has a maximum window size, which must take into account the length of the input prompt and the response. This length is measured in tokens: you can think of them (very roughly) as words or chunk of texts. The maximum window of a LLM is measured in tokens. For example, the base GPT-4 model by OpenAI supports a maximum window of 8000 tokens.

This means that sending the entire content of a document inside the prompt might easily consume all the available tokens, making it impossible for the LLM to process a response. Let’s make a concrete example. Let’s say that you have stored into a vector database a 50 pages document with the company policies as a single text. This means that, if you ask the question “What is the company policy about working from home?”, the search must retrieve the entire document and send it to the LLM, even if it probably contains lot of information that aren’t related to the working from home policy. The LLM will then try to generate a response, but it might fail, because the document is too long and it will consume all the available tokens.

The solution to this problem is to split the document into multiple chunks, each of them small enough to be processed by the LLM. This is called chunking. When you use this approach, inside the vector database you don’t store the entire document as a whole, but you split into chunks and you store each chunk as a separate document. When you perform a search, you retrieve the most relevant chunks and you send them to the LLM. There are many tools and services on the market, (like Azure AI Document Intelligence) which you can use to perform this task.

If you have followed me so far, you have probably realized that building an application that supports the implementation of RAG we have just described requires a certain level of complexity:

It needs to turn a user prompt into a user intent.
It needs to turn our private date into vectors using an embedding model, so that we can store them into a vector database. We need also to apply chunking, to improve the ability of the LLM to process long documents.
It must support performing semantic search against the vector database, given the user intent.
It must combine the user prompt with the documents retrieved from the vector database and submit them to the LLM to generate a response.

In this post, we’re going to learn about a service created by Microsoft called Kernel Memory, which can greatly simplify this implementation and that can be integrated with Semantic Kernel using a plugin.

Let’s start!

Introducing Kernel Memory

Kernel Memory is built with the same principles of Semantic Kernel: it abstract many of the concepts behind RAG and, thanks to multiple extension methods, you can easily connect it to common AI services (like OpenAI and Azure OpenAI) and vector databases (like Azure AI Search or Qdrant). As we’re going to see in this post, switching from one service to another doesn’t mean changing the code of the application.

Additionally, it offers built-in support for converting multiple type of contents into embeddings, like PDFs, Word documents, PowerPoint presentations, websites, etc. It also allows you to skip the flow user prompt -> user intent -> search on the vector database -> prompt augmentation -> LLM response, by providing a single method that performs all these steps for you. Finally, it offers built-in support for chunking, to automatically split long documents into smaller chunks before they are stored in a vector database.

Kernel Memory can run in two ways:

Serverless: the whole service is hosted directly by the running application, which takes care of generating the embeddings, storing them and performing the search. This is the simplest scenario, but it’s not suitable for large datasets or complex applications, since it makes it harder to scale.
As a service: the service is hosted as a separate instance, that the main application can consume using a REST API (which is wrapped by a dedicated C# client). This is the recommended approach for complex applications, since it allows you to scale the service independently from the main application. You could host it, for example, on Azure using an App Service and easily scale it to multiple instances based on the workload.

Regardless of the model you choose, the APIs to perform the main tasks (like generating a vector) are the same. In this post, we’re going to use a sample web application, based on Blazor, that uses Kernel Memory in serverless mode to implement RAG. The application will provide two main functions:

Content ingestion: the user can upload a document, or simply write a text, and the application will generate the vector for it and store it in a vector database.
Q&A: the user can ask questions about the uploaded content and the application will use Kernel Memory to retrieve the most relevant documents and to generate a response using an LLM.

Creating the Blazor application

For this sample, I’m going to use Blazor, the framework to build web applications that run client-side, using C# and .NET instead of JavaScript. I’m going to use the server-side model, which means that the application will run on the server and the code will be rendered on the client using SignalR. This is the simplest model to use, since it doesn’t require any additional configuration. In Visual Studio 2022, choose Create a new project and pick Blazor Web App as a template. Leave all the default settings, which will generate a Blazor server-side application based on .NET 8.0:

As first step, you must install the Kernel Memory NuGet package, which is named Microsoft.KernelMemory.Core.

Now we’re going to build a service, that will wrap all the operations with Kernel Memory for us. Create a new folder named Services and add a new class named MemoryService Before implementing any feature, we need to initialize Kernel Memory. We can do that in the constructor of the service. Let’s see the whole code, then we’ll discuss it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31


public MemoryService()
{
 string apiKey = "AzureOpenAI:ApiKey";
 string deploymentChatName = "AzureOpenAI:DeploymentChatName";
 string deploymentEmbeddingName = "AzureOpenAI:DeploymentEmbeddingName";
 string endpoint = configuration"AzureOpenAI:Endpoint";

 var embeddingConfig = new AzureOpenAIConfig
 {
 APIKey = apiKey,
 Deployment = deploymentEmbeddingName,
 Endpoint = endpoint,
 APIType = AzureOpenAIConfig.APITypes.EmbeddingGeneration,
 Auth = AzureOpenAIConfig.AuthTypes.APIKey
 };

 var chatConfig = new AzureOpenAIConfig
 {
 APIKey = apiKey,
 Deployment = deploymentChatName,
 Endpoint = endpoint,
 APIType = AzureOpenAIConfig.APITypes.ChatCompletion,
 Auth = AzureOpenAIConfig.AuthTypes.APIKey
 };

 kernelMemory = new KernelMemoryBuilder()
 .WithAzureOpenAITextGeneration(chatConfig)
 .WithAzureOpenAITextEmbeddingGeneration(embeddingConfig)
 .WithSimpleVectorDb()
 .Build<MemoryServerless>();
}

Compared to the way we have initialized Semantic Kernel in this blog, in which we used a single model to perform every operation, we need instead to use two distinct models with Kernel Memory:

One for the text generation, which is used by the LLM to generate the response.
One for the embedding generation, which is used to convert documents into vectors.

Similiarly to Semantic Kernel, we have a KernelMemoryBuilder object that we can use to create a kernel, which supports multiple extension methods to register different AI services. In the previous example, we’re going to use models hosted by Azure OpenAI, so we use methods like WithAzureOpenAITextGeneration() and WithAzureOpenAITextEmbeddingGeneration(), but you could use also the variants WithOpenAITextGeneration() and WithOpenAITextEmbeddingGeneration() to use directly the OpenAI service.

Since I’m using Azure OpenAI, I must provide to both methods an AzureOpenAIConfig object, which is used to configure the connection to Azure OpenAI. In my case, I’m using a single Azure OpenAI instance, in which I have deployed both models I need. As such, properties like APIKey and Endpoint are the same. The only difference is the value of the Deployment property, which is indeed different because text generation requires a traditional GPT model (like gpt-4), while embedding generation requires an embedding model (like text-embedding-ada-002).

Once we have defined the models, we must specify which vector database we want to use. Kernel Memory support multiple ones but, for our demo, we’re going to keep it simple. We’re going to use SimpleVectorDb, which is a solution that can run entirely in RAM or on disk. It’s not suitable for production scenarios, but it’s perfect for a demo, because it doesn’t requires us to setup any external service. For the moment, we’re just going to keep the vector database in RAM, so we don’t have to specify any parameter. Finally, we call Build() to create the kernel, passing as type MemoryServerless, since we’re going to host the service directly in our application.

Now we’re ready to implement the two main functions of the application.

Content ingestion

The first scenario we’re going to implement is content ingestion: we need to convert documents into embeddings and store them into a vector database. Kernel Memory makes it very simple, by providing multiple methods based on the type of content we want to ingest. Let’s add one for the most common scenario, which is file ingestion:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


public async Task<bool> StoreFile(string path, string filename)
{
 try
 {
 await kernelMemory.ImportDocumentAsync(path, filename);
 return true;
 }
 catch
 {
 return false;
 }
}

We call the ImportDocumentAsync() method passing, as parameter, the path of the file want to import and it’s file name. It can be one of the supported files specified in the repository of the project. The kernelMemory object offers many other methods to ingest other type of content. For example, we can ingest a text using the ImportTextAsync() method, or a website using the ImportWebPageAsync() method.

That’s it! Thanks to this method, files will be converted into embeddings and stored into the vector database, which is hosted in memory.

Implementing Q&A

Kernel Memory greatly simplifies the Q&A experience, since we don’t have to manually perform a search on the vector database to find the relevant content, and then include it into the prompt that we send to the LLM. The whole operation, in fact, can be performed by a single method, which is named AskAsync(). Let’s see how to use it by adding a new method to our MemoryService class:

1
2
3
4
5
6


public async Task<string> AskQuestion(string question)
{
 var answer = await kernelMemory.AskAsync(question);

 return answer.Result;
}

We simply call the AskAsync() method, passing as parameter the question we want to ask. The method will perform the following steps:

It will convert the question into a user intent, using the text generation model.
It will perform a semantic search on the vector database, using the user intent.
It will combine the documents retrieved from the vector database and it will add them to the user prompt.
It will submit the prompt to the LLM and it will return the response.

Lot of steps, performed with a single line of code!

Before we test our code, let’s add also an interface to describe the service we have just created, so that we can easily use it in our Blazor application with dependency injection:

1
2
3
4
5
6


public interface IMemoryService
{
 Task<bool> StoreFile(string content, string filename);

 Task<KernelResponse> AskQuestion(string question);
}

Now we can go to the Program.cs class of the project and, before the builder.Build() method is called, let’s register our service:

1
2
3
4
5
6
7
8


var builder = WebApplication.CreateBuilder(args);

builder.Services.AddRazorComponents()
 .AddInteractiveServerComponents();

builder.Services.AddSingleton<IMemoryService, MemoryService>();

var app = builder.Build();

Now we’re ready to use the service in our pages.

Creating the Blazor UI

Let’s add now the Blazor components we need to ingest the content and to ask questions. Let’s start from content ingestion, by adding a new component called AddContent.razor under the Pages folder. As first step, we need to map the component to a page, by using the @page directive to create a route; then, we must get an instance of the service we have just created, by using the @inject directive:

1
2


@page "/addcontent"
@inject IMemoryService MemoryService

Now let’s add the UI to support the file ingestion:

1
2
3
4
5
6
7


<h3>Store file</h3>
<div>
 <InputFile OnChange="@LoadFiles" />
</div>
<div>
 @Output
</div>

We have added an InputFile component and we have subscribed to the OnChange event, which gets triggered when the user has selected a file from the disk. We have also added an Output div, which will be used to display the result of the operation. Now let’s add the code to handle the OnChange event:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


@code {
 private string Output;

 private async Task LoadFiles(InputFileChangeEventArgs e)
 {
 Output = "Saving file in progress...";
 var stream = e.File.OpenReadStream();
 var directory = Path.GetDirectoryName(Environment.ProcessPath);
 var documentsPath = Path.Combine(directory, "Documents");
 if (!Directory.Exists(documentsPath))
 {
 Directory.CreateDirectory(documentsPath);
 }
 var path = Path.Combine(directory, "Documents", e.File.Name);

 using (var fileStream = new FileStream(path, FileMode.Create, FileAccess.Write))
 {
 await stream.CopyToAsync(fileStream);
 }

 await MemoryService.StoreFile(path, e.File.Name);
 Output = "File stored succesfully";
 }
}

The LoadFiles() handler receives, as input, the event arguments of the OnChange event, which includes a File property to access to the content of the selected file. We us the Name property to retrieve the full name of the file, which we use to define the full path where we want to store it. In our case, it’s a folder called Documents, which is a subfolder of the location where the application is running from. Then, we use the OpenReadStream() method to get a stream to the content of the file, which we copy into a new file using the FileStream class and the CopyToAsync() method. Finally, we call the StoreFile() method of our MemoryService class, passing as parameter the path of the file and it’s name, which will be used to generate the vector and store it in the vector database.

Now let’s add the UI to support the Q&A experience. Create a new component named AskQuestion.razor under the Pages folder. Also in this case, we’ll need to it to a page using the @page directive and to inject the MemoryService object using the @inject directive. Then, add the following code:

1
2


@page "/askquestion"
@inject IMemoryService MemoryService

The code of the page is really simple:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


<p>
 Type your question:
</p>
<p>
 <input @bind="question" style="width: 100%;" />
</p>
<p>
 <button class="btn btn-primary" @onclick="Ask">Ask</button>
</p>
<p>
 <strong>The answer is:</strong> @answer
</p>

@code {
 private string question;
 private string answer;

 private async Task Ask()
 {
 answer = await MemoryService.AskQuestion(question);

 }
}

We have a simple input field, which is bound to the question property. When the user clicks on the Ask button, the Ask() method is called, which calls the AskQuestion() method of the MemoryService class. The method is going to start a semantic search in the vector database, get the most relevant documents, send them to the LLM and generate a response, which we’re going to display to the user.

Testing the application

Let’s test the application! Press F5 in Visual Studio then, once the Blazor website is loaded in the browser, head to the /addcontent page. For this test, we’re going to use the following PDF file, which is a fictitious employee manual of a made up company. Click Choose file and select the PDF. After a few seconds, the processing will be completed and you will see a success message.

Now head to the /askquestion page and let’s try to ask a few questions about the content of the document. For example, “What is Contoso Electronics?”.

The answer is correct! Let’s try another one: “Which are the Contoso Electronics values?”.

Again, the answer is correct! These questions, however, don’t show the full potential of semantic search because, probably, the information needed to generate a response could have been retrieved also with a more traditional keyword search. Let’s try a more complex one: “Is there any connection between Contoso Electronics and things that fly?”.

As you can see, despite the PDF we have uploaded doesn’t include the specific words “things that fly”, the semantic search was able to retrieve the paragraph of the document that talks about the mission of Contoso Electronics, which is to build components for aircrafts. This is a great example of the power of semantic search!

But how can be sure that the information is indeed coming from the PDF document? We can leverage another powerful feature of Kernel Memory, which is the ability to include citations in the response. The AskQuestion() method we have created in the KernelMemory class currently returns only the value of the Answer property, but the response object contains also other information, like the citations. Let’s create a new class named KernelResponse under the Models folder, which will be used to store the response of the AskQuestion() method in a more complete way:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


using Microsoft.KernelMemory;

namespace AzureRag.Models
{
 public class KernelResponse
 {
 public string Answer { get; set; }
 public List<Citation> Citations { get; set; }
 }
}

Now let’s update the AskQuestion() method of the MemoryService class to return an instance of this class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


public async Task<KernelResponse> AskQuestion(string question)
{
 var answer = await kernelMemory.AskAsync(question);

 var response = new KernelResponse
 {
 Answer = answer.Result,
 Citations = answer.RelevantSources
 };

 return response;
}

As you can see, the response contains also a property called RelevantSources, which is a collection of all the sources that were used to generate the response.

Finally, let’s update the UI and the code of the AskQuestion.razor page to display the citations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


@page "/askquestion"
@using System.Text.RegularExpressions
@using System.Web
@using AzureRag.Models
@inject IMemoryService AIService
@rendermode InteractiveServer

<h1>Ask question</h1>

<p>
 Type your question:
</p>
<p>
 <input @bind="question" style="width: 100%;" />
</p>
<p>
 <button class="btn btn-primary" @onclick="Ask">Ask</button>
</p>

<p>
 @if (answer != null)
 {
 <strong>The answer is:</strong> @answer.Answer
 @foreach (var citation in answer.Citations)
 {
 <ul>

 <li><strong>File name:</strong> @citation.SourceName</li>
 <li><strong>File type:</strong>@citation.SourceContentType</li>
 </ul>
 }
 }

</p>

@code {
 private string question;
 private KernelResponse answer;

 private async Task Ask()
 {
 answer = await AIService.AskQuestion(question);

 }
}

Below the answer, now we’re going to use a foreach statement to display the list of citations stored in the Citation property, which includes the name of the file (SourceName) and the type of the file (SourceContentType). In the @code section, we have simply changed the type of the answer property from string to KernelResponse.

Now press F5 and try again to ask a question, like “What is Contoso Electronics?”. You will see that, below the answer, you will find a list of citations, which includes a reference to the employee_handbook.pdf file we have previously uploaded:

Storing the vector database on disk

If you stop and relaunch the application, then you ask again the same questions, the experience will be quite different:

This is because the vector database is stored in memory, which means that it’s lost when the application is stopped. To avoid this problem, we can store the vector database on disk. To do that, we need to change the code of the MemoryService class, by adding a new method to initialize the vector database:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


var directory = Path.GetDirectoryName(Environment.ProcessPath);
var path = Path.Combine(directory, "Memory");
if (!Directory.Exists(path))
{
 Directory.CreateDirectory(path);
}

kernelMemory = new KernelMemoryBuilder()
 .WithAzureOpenAITextGeneration(chatConfig)
 .WithAzureOpenAITextEmbeddingGeneration(embeddingConfig)
 .WithSimpleVectorDb(path)
 .Build<MemoryServerless>();

We have added a new parameter to the WithSimpleVectorDb() method, which is the path where we want to store the vector database. In this case, we’re going to store it in a folder named Memory, which is a subfolder of the location where the application is running from. Now launch again the application and use again the AddContent page to upload the PDF. This time, you will find a new folder in your project called Memory, with a series of files:

Now, if you stop and relaunch the application, the vector database will be loaded from disk and you will be able to ask questions without having to upload the PDF again.

And what if you want to move your application into production and use a more structured vector database, like Azure AI Search? Thanks to the built-in support for multiple providers, you don’t have to change your code. You just need to use a different extension method provided by the KernelMemoryBuilder, like in the following example:

1
2
3
4
5
6
7
8


string searchApiKey = "AzureSearch:ApiKey";
string searchEndpoint = "AzureSearch:Endpoint";

kernelMemory = new KernelMemoryBuilder()
.WithAzureOpenAITextGeneration(chatConfig)
.WithAzureOpenAITextEmbeddingGeneration(embeddingConfig)
.WithAzureAISearchMemoryDb(searchEndpoint, searchApiKey)
.Build<MemoryServerless>();

You just initialize Azure AI Search by using the WithAzureAISearchMemoryDb() method and passing, as parameter, the endpoint and the API Key provided by the Azure portal. Now, if you try again to process the PDF file using the AddContent page, you will see in the Azure portal that a new index, with name default, has been created. If you explore the content, you will see that the document has been split into multiple chunks, and each of them has been converted into a vector:

Wrapping up

This was quite a long post! We have seen how to use Kernel Memory to simplify the implementation of RAG, which enables an application to use a LLM to generate answers based on private data. We have seen how to use Kernel Memory to convert documents into vectors and to store them into a vector database, and how to use it to perform semantic search and to generate a response using an LLM. We have also seen how to store the vector database on disk and how to use a more structured vector database, like Azure AI Search.

In the next post, we’re going to see how Kernel Memory can be leveraged through Semantic Kernel with a specific plugin. Stay tuned!

In the meantime, you can find the code used in this post on GitHub, in the project called KernelMemory.

Semantic Kernel - Out of the box plugins for Bing and Microsoft Graph

Fri, 29 Dec 2023 00:00:00 +0000

If you have read the posts I’ve published so far about Semantic Kernel, you know the importance of plugins. Thanks to them, you can extend the kernel with functions and features offered by 3rd party services that can be automatically managed by using function calling and planners. You can just import them in the kernel, define the task you want to accomplish and let Semantic Kernel and the LLM figure out which plugins should be used. In the previous articles, however, we have seen only custom plugins, which we have created from scratch. In this post, we’ll take a look at the built-in plugins that are developed by the Semantic Kernel team. These plugins are not only useful to accomplish some common tasks, but they are also a good learning experience because they provide more complex integrations, which requires to introduce concepts like authentication.

In this article, we’re going to see two of the most interesting available plugins: Bing and Microsoft Graph.

The Bing plugin

If you have worked with LLMs, you’ll know that one of their limitations is that they are trained with a huge set of public data that, at some point, was frozen in time. This means that if, for example, you ask who is the current prime minister of a country, the LLM will answer with the information it had at the time the model was created, which might be not valid anymore. This is not a problem if you’re using the LLM to generate text, but it might became an issue if you’re using it to answer questions about topics that can easily change over time. The Bing plugin can help you overcoming this limitation, since it provides a function which uses the Bing Search APIs to find content on the web given a search query. By importing the plugin into the kernel and using function calling or planners to automatically orchestrate the execution, Semantic Kernel can automatically generate a search query and use the results from the web search to answer the question of the user with fresh information.

Let’s see a concrete example in action. The first step is to install the plugin, which is stored in a dedicated NuGet package called Microsoft.SemanticKernel.Plugins.Web. Before starting to write some code, however, we need the API key required to use the Bing Search APIs, which you can get from the Azure portal. Once you’re logged in with your subscription, click on Create a resource and, using the internal search, search for the service called Bing Search v7 and choose Create:

The next step is to initialize the service, which doesn’t offer many customizations: just the name and the pricing tier. Due to its own nature, the service is global and it can’t be deployed to a specific region.

Once you have created the service, all you have to do is open it and go to the Keys and Endpoint section, where you can find the API key you need:

Now we can start writing some code! First, let’s initialize the kernel in the usual way:

1
2
3
4
5
6
7
8


string apiKey = "AzureOpenAI:ApiKey";
string deploymentName = "AzureOpenAI:DeploymentName";
string endpoint = "AzureOpenAI:Endpoint";
string bingKey = "Bing:ApiKey";

var kernel = Kernel.CreateBuilder()
 .AddAzureOpenAIChatCompletion(deploymentName, endpoint, apiKey)
 .Build();

You can notice how, compared to the samples in the previous posts, we have a new placeholder for the Bing API key. Just make sure to include the one you have retrieved from the Azure portal.

This is just an example to keep the code simple. In a real application, you should store the API key in a safe location, like using user secrets or Azure Key Vault.

As next step, we need to add the plugin into the kernel. This time, however, we can’t use the ImportPluginFromType<T>() method we have learned to use in this post because the plugin requires a bit of configuration before being able to use it. This is the code we need to use to initialize it:

1
2
3
4
5
6
7


#pragma warning disable SKEXP0054 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.

var bingConnector = new BingConnector(bingKey);
var plugin = new WebSearchEnginePlugin(bingConnector);
kernel.ImportPluginFromObject(plugin, "BingPlugin");

#pragma warning restore SKEXP0054 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.

First, we need to wrap our code into a #pragma directive, since the Bing plugin is marked as experimental. Without this directive, the build would fail. Inside the directive, we create a new instance of the BingConnector class, which is the class that is used to call the Bing APIs. The constructor of this class requires the API key we have retrieved from the Azure portal. Then, we create a new instance of the WebSearchEnginePlugin class, which is the class that wraps the Bing connector into a plugin. The constructor of this class requires the connector we have just created. Finally, we import the plugin into the kernel using the ImportPluginFromObject() method, which requires the plugin instance and a name for the plugin. The name is used to identify the plugin in the kernel and it’s used when we want to call a function included in the plugin.

Now we can use this plugin like we did with all the custom ones we have built so far, like the UnitedStatesPlugin one. Let’s define a prompt and let’s use the function calling capabilities we have learned about to let the Semantic Kernel call the plugin for us:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


#pragma warning disable SKEXP0054 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
var bingConnector = new BingConnector(bingKey);
var plugin = new WebSearchEnginePlugin(bingConnector);
kernel.ImportPluginFromObject(plugin, "BingPlugin");

OpenAIPromptExecutionSettings settings = new()
{
 ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions
};

var results = kernel.InvokePromptStreamingAsync("What is Semantic Kernel from Microsoft?", new KernelArguments(settings));
await foreach (var message in results)
{
 Console.Write(message);
}

Console.WriteLine();
Console.ReadLine();

#pragma warning restore SKEXP0054 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.

Before executing this code, let’s comment the line of code which imports the plugin into the kernel, so that we can test what happens when the plugin is not available, so that Semantic Kernel must rely only on the internal knowledge of the LLM.:

1

//kernel.ImportPluginFromObject(plugin, "BingPlugin");

You will get an answer like the following one:

1
2
3


As of my knowledge cutoff date in April 2023, there is no widely recognized product or technology specifically called "Semantic Kernel" from Microsoft.
It is possible that you may be referring to a component of a larger system, an internal project,
or a lesser-known technology that has not been widely publicized or documented.

As you can see, the LLM doesn’t have any knowledge about Semantic Kernel, since the SDK was created after April 2023, which is the cutoff date of the model I’m using for this test (which is GPT-4). Now let’s uncomment the line of code which imports the plugin and let’s execute the code again. This time, you will get a different answer:

1
2
3
4
5
6


Semantic Kernel from Microsoft is an open-source Software Development Kit (SDK) that allows developers
to integrate Large Language Models (LLMs) such as OpenAI, Azure OpenAI, and Hugging Face with conventional programming languages
including C#, Python, and Java.
The SDK enables the creation of AI applications by combining AI services with traditional programming techniques,
similar to the approach used in products such as Microsoft 365 Copilot and Bing.
It offers a way to define plugins that can be easily chained together with just a few lines of code.

This is a good answer and a clear example of how the Bing plugin can be used to overcome the limitations of the LLMs. If you want to review in more details the usage of the Bing plugin, you can switch from automatic function calling to manual function calling, as we have explained in the previous post, by using the following code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


var chatHistory = new ChatHistory();
chatHistory.AddMessage(AuthorRole.User, "What is Semantic Kernel from Microsoft?");

var chatCompletionService = kernel.GetRequiredService<IChatCompletionService>();
var result = await chatCompletionService.GetChatMessageContentAsync(chatHistory, settings, kernel);

var functionCalls = ((OpenAIChatMessageContent)result).GetOpenAIFunctionToolCalls();
foreach (var functionCall in functionCalls)
{
 KernelFunction pluginFunction;
 KernelArguments arguments;
 kernel.Plugins.TryGetFunctionAndArguments(functionCall, out pluginFunction, out arguments);
 var functionResult = await kernel.InvokeAsync(pluginFunction!, arguments!);
 var jsonResponse = functionResult.GetValue<object>();
 var json = JsonSerializer.Serialize(jsonResponse);
 Console.WriteLine(json);
 chatHistory.AddMessage(AuthorRole.Tool, json);
}

result = await chatCompletionService.GetChatMessageContentAsync(chatHistory, settings, kernel);

Console.WriteLine(result.Content);
Console.ReadLine();

By using the manual approach, we can see that Semantic Kernel has identified that, in order to satisfy the ask we provided, it needs to use the Search function included in the Bing plugin passing, as arguments, the keyword Semantic Kernel from Microsoft:

The Microsoft Graph plugin

Let’s take a look now at another very useful plugin, which you can use to build applications that are able to interact with the Microsoft 365 ecosystem. The Microsoft Graph is a REST API ecosystem that allows you to interact with the data stored in Microsoft 365, like the user’s calendar, the user’s email, the user’s contacts, and so on. The Microsoft Graph plugin allows you to use the Microsoft Graph APIs from Semantic Kernel, so that the LLM can perform tasks which involves your organizational data. At a much smaller scale, of course, but this is similar to the experience offered by Microsoft 365 Copilot.

Also in this case you will need to install a dedicated NuGet package first, called Microsoft.SemanticKernel.Plugins.MsGraph. The process to setup the plugin is slightly more complex than the one we have seen for the Bing plugin. The Microsoft Graph, in fact, is based on Microsoft Entra (formerly known as Azure Active Directory), which uses the OAuth2 based approach to authenticate users and applications. This means that, in order to use the Microsoft Graph APIs, you will need first to authenticate the user, so that you can get the access token needed to perform operations on behalf of them.

There are multiple ways to authenticate users with Entra: the most common one is the Authorization Code flow, which redirects the user to the Entra login page, where the user can authenticate and authorize the application to access the user’s data. Once the user has authorized the application, the user is redirected back to the application, which can then retrieve the access token needed to call the Microsoft Graph APIs. This flow is typically used by client applications, like the web based ones.

In the case of my samples, however, we need to use a different approach, since they’re console applications, which aren’t capable of opening a browser to redirect the user to the Entra login page. The approach we’re going to use is called Device Code flow, which is designed for applications which run in an input-constrained environment, like a Smart TV or, precisely, a console application. In this flow, the user is presented with a code and a URL, which the user must use to authenticate and authorize the application. The user must open the URL in a browser, enter the code provided by the app, then authenticate and authorize the application. Once the user has done that, the application can retrieve the access token needed to call the Microsoft Graph APIs.

To make our life easier, the Microsoft Graph plugin includes the Microsoft Graph .NET SDK, which is a wrapper around the Microsoft Graph that we can use to implement the authentication workflow in a simpler way. But first, as for every other application that needs to interact with the Microsoft Graph, we need to register it in the Azure portal on Microsoft Entra. To do that, open the Azure portal, go to the Microsoft Entra ID section, search for the App registrations service and choose New registration:

In the next step, you must provide a name for the application and choose which type of accounts you want to support. You can choose between:

Accounts in this organizational directory only (single tenant), which means that you’ll be able to use the application only with users that belong to the same tenant in which you’re registering the application.
Accounts in any organizational directory (Any Microsoft Entra ID directory - Multitenant), which means that you’ll be able to use the application with users that belong to any tenant.
Accounts in any organizational directory (Any Microsoft Entra ID directory - Multitenant) and personal Microsoft accounts (e.g. Skype, Xbox), which means that you’ll be able to use the application with users that belong to any tenant and also with personal Microsoft accounts.
Personal Microsoft Accounts only, which is for applications that are targeting only consumers.

Pick up the one that works best for your scenario (in my case, I chose to build a multitenant application). Once the application is created, we have to define the permissions that the application needs to access the Microsoft Graph APIs. The type of permissions change based on the scenario you would like to implement. In my case, I’m going to build an application that you can use to query your calendar using natural language. This means that I need to access the user’s calendar, so I need to add the Calendars.Read permission. To do that, go to the API permissions section and choose Add a permission. In the next step, choose Microsoft Graph and then Delegated permissions. In the list of permissions, search for the Calendars.Read permission and select it. Once you have selected the permission, choose Add permissions.

The final step is to generate a client secret, which is a key that the application can use to authenticate itself with Entra. To do that, go to the Certificates & secrets section and choose New client secret. In the next step, provide a description for the secret and choose the expiration date. Once you have done that, choose Add. Once the secret is created, copy the value of the secret somewhere, we’re going to need it later. Make sure to do it because, after you leave this section, the secret will be hidden and you won’t be able to retrieve it anymore.

Now that we have the app registration, we can start writing some code to handle the authentication:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


var scopes = new[] { "Calendars.Read" };
var tenantId = "tenant-id";
var clientId = "clientId";
var clientSecret = "clientSecret";

var options = new DeviceCodeCredentialOptions
{
 AuthorityHost = AzureAuthorityHosts.AzurePublicCloud,
 ClientId = clientId,
 TenantId = tenantId,
 // Callback function that receives the user prompt
 // Prompt contains the generated device code that user must
 // enter during the auth process in the browser
 DeviceCodeCallback = (code, cancellation) =>
 {
 Console.WriteLine(code.Message);
 return Task.FromResult(0);
 },
};

var deviceCodeCredential = new DeviceCodeCredential(options);

var graphClient = new GraphServiceClient(deviceCodeCredential, scopes);

First, we have to define the information we need to perform the authentication:

The scopes we need to access the Microsoft Graph APIs. In my case, I need to access the user’s calendar, so I need the Calendars.Read permission.
The tenant ID, which is the ID of the tenant in which you have registered the application.
The client ID, which is the ID of the application you have registered.
The client secret, which is the secret you have generated in the previous step.

The tenant ID and the client ID are available in the Overview section of the app registration:

With these information, we can define a DeviceCodeCredentialOptions instance, which is the class that is used to configure the authentication process. The object defines all the information needed for the authentication, plus a callback (called DeviceCodeCallback) which is invoked when the code is generated and the application needs to wait for the user to complete the authentication in the browser. Thanks to these options, we can create a new DeviceCodeCredential instance, which we can pass to the GraphServiceClient, which is the C# wrapper around the Graph APIs provided the Microsoft Graph .NET SDK.

Now we have all we need to authenticate the user. The Microsoft Graph plugin, in fact, requires an authenticated GraphClient instance to work. Let’s see how we can use it to create a plugin instance and import it into the kernel:

1
2
3
4
5
6
7
8


#pragma warning disable SKEXP0053 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.

OutlookCalendarConnector connector = new OutlookCalendarConnector(graphClient);
CalendarPlugin plugin = new CalendarPlugin(connector);

kernel.ImportPluginFromObject(plugin, "CalendarPlugin");

#pragma warning restore SKEXP0053 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.

The Microsoft Graph plugin library doesn’t include a single plugin, but multiple plugins that you can use to access different parts of the Microsoft 365 ecosystem. In this case, we’re going to use the CalendarPlugin, which is the plugin that allows you to access the user’s calendar. To initialize the plugin, we need an OutlookCalendarConnector, which requires as parameter the authenticated GraphClient object we have previously created. Finally, like we did with the Bing plugin, we import it into the kernel using the ImportPluginFromObject() method. The code is wrapped inside the usual #pragma directive, since also the Graph plugin is marked as experimental.

Now we can test the plugin by using function calling with a prompt that requires access to the calendar to be satisfied. Here is an example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


OpenAIPromptExecutionSettings settings = new()
{
 ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions
};

string prompt = "What is my next meeting?";
var results = kernel.InvokePromptStreamingAsync(prompt, new KernelArguments(settings));
await foreach (var message in results)
{
 Console.Write(message);
}

Console.WriteLine();
Console.ReadLine();

We are using Semantic Kernel to ask what’s my next meeting. If you execute the application, at first you’ll see the device code authentication experience:

You are asked to open your browser at the URL https://microsoft.com/devicelogin and to input the code you see in the console. After providing the code, you’ll have to authenticate with your Microsoft 365 account and to authorize the application to access your calendar. After having completed the authentication flow, the console application will resume the execution and, after a few seconds, you’ll see the answer to your question, which in my case is the following one:

1

Your next meeting, labeled "This is a test," is scheduled for December 22, 2023, at 4:00 PM (UTC).

As you can see from the following image, this is indeed correct!

Thanks to the Semantic Kernel orchestration features, the application was able to automatically call the Calendar plugin to get the information about the next meeting and to return the information using natural language thanks to the LLM processing.

Also in this case, if we want, we can switch to the manual function calling approach to see how the plugin is called:

We can see that Semantic Kernel has called the GetCalendarEvent function of the Calendar plugin, passing as argument the parameter maxResults with value 1. The function, in fact, returns the list of meetings on the user’s calendar ordered by the most recent ones. Since we have asked for the next meeting, it’s enough to filter the list and to return only the first one.

Wrapping up

In this post we have seen how to use the some of the plugins for Semantic Kernel provided by the team. We have seen how to use the Bing plugin to overcome the limitations of the LLMs and how to use the Microsoft Graph plugin to access the Microsoft 365 ecosystem.

You can find all the samples in the GitHub repository of the project.

Semantic Kernel - The new planners introduced in 1.0

Wed, 27 Dec 2023 00:00:00 +0000

In the previous post we have learned that Semantic Kernel 1.0 has added support for a feature that OpenAI has introduced in their most recent models, called function calling. This feature has made the Semantic Kernel planner outdated for many scenarios. Function calling, in fact, serves the same purpose, which is enabling the LLM to figure out automatically which functions are needed to perform a task, but it does it in a more efficient way. The approach used by the planner is called ReAct, which means that AI is going to call a function, evaluate the response and then call another function if needed. With the planner, all the steps required back and forth communication between the LLM and the code, using lot of tokens (which means worse performance and a more expensive bill). Function calling, instead, is baked into the model, which means that you can skip the back and forth communication completely, since the model is able to perform this type of reasoning on its own.

For this reason, for many scenarios, you don’t need a planner anymore: the function calling capabilities we have highlighted in the previous post are capable to managing them. However, there might be scenarios in which these capabilities aren’t good enough for the task you’re trying to perform, since you need to apply a more complex reasoning. If you observe a scenario in which function calling isn’t leading to the outcome you’re expecting, there’s are two new tools in Semantic Kernel 1.0 that you can use: the Function Calling Stepwise planner and the Handlebars planner.

Introducing the Function Calling Stepwise planner

The Function Calling Stepwise planner is built on top of calling functions, so it uses the same approach under the hood. However, compared to pure function calling, it’s able to reach the LLM to perform additional reasoning when it comes to generating the plan, so that it can improve the reliability of identifying the right functions to call. The first step to use this new planner is to install the dedicated NuGet package, which is Microsoft.SemanticKernel.Planners.OpenAI.

Let’s setup now the project in a similar way we did in the previous post. The goal is to get the body of a mail to share the information about the population number of the United States in 2015, split by the number of people who identify themselves as male or female. As you can see, this is a task that requires using some of the plugins we have built in the previous posts: the WriteBusinessMail prompt function and the UnitedStatesPlugin native class. Here is the initialization code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


string apiKey = "configuration";
string deploymentName = "AzureOpenAI:DeploymentName";
string endpoint = "AzureOpenAI:Endpoint";

var kernel = Kernel.CreateBuilder()
 .AddAzureOpenAIChatCompletion(deploymentName, endpoint, apiKey)
 .Build();

kernel.ImportPluginFromType<UnitedStatesPlugin>();

var pluginsDirectory = Path.Combine(Directory.GetCurrentDirectory(), "Plugins", "MailPlugin");
kernel.ImportPluginFromPromptDirectory(pluginsDirectory, "MailPlugin");

Now we need to add, like we have seen in the post about using OpenAI plugins, a #pragma directive: this new planner is marked as experimental, so we have to suppress the warning, otherwise we won’t be able to compile our code:

1
2
3
4
5


#pragma warning disable SKEXP0061 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.

//code goes here

#pragma warning restore SKEXP0061 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.

Using this new planner is quite straightforward, as you can see in the following example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14



#pragma warning disable SKEXP0061 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
var planner = new FunctionCallingStepwisePlanner();

var ask = @"Write a business mail to share the population of the United States in 2015.
Make sure to specify how many people, among the population, identify themselves as male and female.
Don't share approximations, please share the exact numbers.";

var result = await planner.ExecuteAsync(kernel, ask);

Console.WriteLine(result.FinalAnswer);
Console.ReadLine();

#pragma warning restore SKEXP0061 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.

We create a new FunctionCallingStepwisePlanner object, then we call the ExecuteAsync() method passing as parameter our kernel object and the prompt which describes the task we want to perform. We directly get back the outcome of the task once the LLM has completed the orchestration process, inside the FinalAnswer property.

If we run the code, we will get a result similar to the following one:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


Subject: 2015 United States Population Data Request

Dear [Recipient Name],

I trust this email finds you well.

I am reaching out to deliver the requested population statistics for the United States for the year 2015. Below are the exact details:

- Total population: 316,515,021
- Male population: 155,728,568
- Female population: 160,786,456

These numbers reflect the self-identified gender count as of that year.

Should you require any additional information or have further inquiries, please do not hesitate to contact me. I am at your service to assist.

Warm regards,

AI Assistant

If we want to understand in more details what happened, the result object includes a property called ChatHistory, which includes the whole conversation between the LLM and the user (in this case, the application):

As you can see from the image, the history contains the whole chain of functions that was called by the planner: the two native ones (GetPopulation and GetPopulationByGender, which was called two times) and the prompt one, WriteBusinessMail, to generate the text of the business mail.

The Handlebars planner

Function callings are very powerful for most scenarios, but there are still some advantages in using a planner:

You can generate the plan ahead of the execution, giving you the chance to evaluate it.
If you get a good plan, you can save it and reuse it, without having to regenerate it every time.
The plan is generated with a single LLM call, helping to save tokens.

These were the main features that led the Semantic Kernel team to build tools like the Sequential planner. However, there was a catch. The plans were generated using a custom XML syntax, which is challenging for the LLM to understand sometimes, leading to the generation of wrong plans. As the team has shared in a blog post, however, researches have demonstrated that LLMs performs much better when they are asked to code in a language they are trained on. As such, the team has decided to switch from XML to Handlebars, which is a template language originally built for JavaScript, but which has been ported to many other languages, including C#. Thanks to this language, you can easily define a template and then, at runtime, replace the various placeholders with real values. As a template language, additionally, it supports also features that, otherwise, would require a full programming language, like conditions and iterators. For example, let’s say that you need to render a list of items in HTML. With a Handlebars template, you can write something like this:

1
2
3
4
5


<ul class="people_list">
 {{#each people}}
 <li>{{this}}</li>
 {{/each}}
</ul>

The {{#each}} and {{/each}} are the iterators, which are used to iterate over the list of people and render the list items. The {{this}} is the placeholder, which is replaced with the value of the current item in the list. By providing in input a people collection like the following one:

1
2
3
4
5
6
7


{
 "people": [
 "John",
 "Mary",
 "Peter"
 ]
}

The output will be the following one:

1
2
3
4
5


<ul class="people_list">
 <li>John</li>
 <li>Mary</li>
 <li>Peter</li>
</ul>

In the context of Semantic Kernel, a plan written with Handlebars gives the ability to the LLM to use this powerful features in the generated plan, making more easily to manage conditions, loops and other complex scenarios.

Now that we have understood what is Handlebars, let’s see how we can use the Handlebars planner in our Semantic Kernel applications. First, like we did for the Function Calling Stepwise planner, we need to install the dedicated NuGet package, called Microsoft.SemanticKernel.Planners.Handlebars. Also in this case, we need to add a specific #pragma directive to suppress the warning about the experimental nature of the planner:

1
2
3


#pragma warning disable SKEXP0060 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
// code
#pragma warning restore SKEXP0060 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.

Now we can generate and execute the plan in the following way:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


#pragma warning disable SKEXP0060 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
var planner = new HandlebarsPlanner();

var ask = "Write a mail to share the number of the United States population in 2015 for a research program.";

HandlebarsPlan plan = await planner.CreatePlanAsync(kernel, ask);

var originalPlanResult = await plan.InvokeAsync(kernel);

Console.WriteLine(originalPlanResult);

#pragma warning restore SKEXP0060 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.

First, we create a new HandlebarsPlanner() object, then we use the CreatePlanAsync() method to create the plan, providing as inputs the kernel and the prompt with the task we want to achieve. Once the plan is generated, we can execute it by calling the InvokeAsync() method, passing as input again the kernel. We get directly back the result of the plan, like the following one:

1
2
3
4
5
6
7
8
9


Dear Sir/Madam,

I would like to bring to your attention that the population of the United States in the year 2015 was recorded to be approximately 316,515,021 individuals. Please take this figure into account for your records.

Should you need any further clarification or additional information, please do not hesitate to contact me.

Best regards,

AI Assistant

If we print on the terminal console the plan, we can see the Handlebars language in action which highlights the usage of the GetPopulation function from the UnitedStatesPlugin and the WriteBusinessMail function from the MailPlugin:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


{{!-- Step 1: Set the year for which the data is needed --}}
{{set "year" "2015"}}

{{!-- Step 2: Use the custom helper to get the population data for the year --}}
{{set "populationData" (UnitedStatesPlugin-GetPopulation year=year)}}

{{!-- Step 3: Format the data and message for the body of the mail --}}
{{set "mailContent" (concat "The United States population in " year " was " (json populationData.TotalNumber) " people.")}}

{{!-- Step 4: Use the custom helper to generate the business mail with the formatted content --}}
{{set "businessMail" (MailPlugin-WriteBusinessMail input=mailContent)}}

{{!-- Step 5: Output the business mail --}}
{{json businessMail}}

As you can see, the plan is just text content, so we can easily store it in a text file and reload it for later usages. The following code shows an improved version of the previous one:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25


#pragma warning disable SKEXP0060 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.
var planner = new HandlebarsPlanner();

var ask = "Write a mail to share the number of the United States population in 2015 for a research program.";

HandlebarsPlan plan;

if (!File.Exists("plan.txt"))
{
 // Create the plan
 plan = await planner.CreatePlanAsync(kernel, ask);

 var serializedPlan = plan.ToString();
 await File.WriteAllTextAsync("plan.txt", serializedPlan);
}
else
{
 string serializedPlan = await File.ReadAllTextAsync("plan.txt");
 plan = new HandlebarsPlan(serializedPlan);
}

// Execute the plan
var originalPlanResult = await plan.InvokeAsync(kernel, []);
Console.WriteLine(originalPlanResult);
#pragma warning restore SKEXP0060 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.

We store the plan in a file called plan.txt. If the file doesn’t exist, it means we have to generate one in the same way we have seen before, by calling the CreatePlanAsync(). Then, once the plan has been generated, we call the ToString() extension method to get the serialized version of the plan and we store it in the file by calling the static method File.WriteAllTextAsync(). If the file exists, instead, we read the content of the file and we use it to create a new HandlebarsPlan object, passing it to the initializer. Then, independently by the way we acquired the plan, we execute it by calling the InvokeAsync() method.

If you run this code twice, you will notice that the second time you will get the result back much faster. This because Semantic Kernel had to call the LLM just to process the plan and get the outcome of the task we have asked, but not to create it.

Wrapping up

In this post, we have learned that, even if the function calling feature provided by OpenAI is able to take care of most of the scenarios, there are still some cases in which you might need to use a planner. Semantic Kernel 1.0 has introduced two new planners, which are the Function Calling Stepwise planner and the Handlebars planner. The first one is built on top of the function calling feature, but it’s able to reach the LLM to perform additional reasoning when it comes to generating the plan, so that it can improve the reliability of identifying the right functions to call. The second one, instead, is a new version of the Handlebars planner, which is now built on top of the Handlebars language, which is a template language that makes it easier to generate plans that are easier to understand by the LLM and to store for later usages.

You can find all the samples for these new scenarios in the usual repository on GitHub.

Happy coding!

Semantic Kernel - Function calling as a planner replacement

Fri, 22 Dec 2023 00:00:00 +0000

In the previous posts, we have learned a few of the changes introduced in Semantic Kernel 1.0: the new setup, prompt functions, the new attributes to define a native function. However, so far, we have seen mostly refactoring and renaming of classes and methods, which required us to change our code, but not to change the way we architect our applications. Things, however, are going to change in this post since Semantic Kernel 1.0 has introduced some deep changes to one of its most powerful features: planners, which bring the capability to automatically orchestrate AI workflows, by automatically figuring out which are the right plugins to use to satisfy the ask of the user.

Why does Semantic Kernel 1.0 brings such a deep change? The reason is that, after that the Semantic Kernel team introduced the planner, OpenAI released a new feature called function calling which, in a way, tries to solve the same problem that the planner was trying to solve: orchestrating complex AI workflows, which might require the interaction with 3rd party APIs.

Let’s learn more about this feature.

Function calling

Function calling overlaps with one of the most interesting features of the planner in Semantic Kernel: the ability, given a series of available functions, to figure out which ones to call to satisfy the ask of the user. However, function calling is much more efficient than the planner, since it’s a native feature of the LLM, while the planner was a custom implementation of the Semantic Kernel team, which required performing multiple LLM calls to create the plan.

With function calling, you can include in the request you send to an OpenAI model not just the prompt, but also a series of functions which are available to call. If the model realizes that, to provide a response to the prompt, it needs to call one or more of the available functions, it automatically generates a request with the needed JSON payload. It’s very important to highlight that the model won’t call the functions on your behalf, but it will return everything you need to call the function in the proper way. It’s up to the developer to use the generated JSON payload to call the needed API, get a response and use it to generate the final answer to the user.

Since this feature was added at a later stage, you will need to use one of the most recent versions of the LLMs, like gpt-3.5-turbo or gpt-4. You can see a full list of the supported models here (the documentation is about OpenAI, but the same models are available also in Azure OpenAI and they include support for function calling).

The official documentation includes a very detailed step-by-step overview of what happens when you use function calling:

Call the model with the user query and a set of functions defined in the functions parameter.
The model can choose to call one or more functions; if so, the content will be a stringified JSON object adhering to your custom schema.
Parse the string into JSON in your code, and call your function with the provided arguments if they exist.
Call the model again by appending the function response as a new message, and let the model summarize the results back to the user.

In the OpenAI blog, you can find a very good example to understand function calling in action. Let’s say that you want to use an OpenAI model to get the current weather in a specific city. We can use the Chat Completion APIs, which offer support for function calling. This is how the JSON payload of the request you send to OpenAI might look like:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


{
 "model": "gpt-3.5-turbo-0613",
 "messages": [
 {"role": "user", "content": "What is the weather like in Boston?"}
 ],
 "functions": [
 {
 "name": "get_current_weather",
 "description": "Get the current weather in a given location",
 "parameters": {
 "type": "object",
 "properties": {
 "location": {
 "type": "string",
 "description": "The city and state, e.g. San Francisco, CA"
 },
 "unit": {
 "type": "string",
 "enum": ["celsius", "fahrenheit"]
 }
 },
 "required": ["location"]
 }
 }
 ]
}

The request includes:

The prompt of the user, which is What is the weather like in Boston?. The Chat Completion APIs, instead of using a single prompt, support including multiple messages so that you can manage the chat history, each of them with a role property to specify who generated the message (the user, the LLM, etc.). As such, we add the prompt as a message with user as a role.
The list of available functions that the LLM can use to generate a response. In this case, we have a single function called get_current_weather. The JSON includes the description of the function and the parameters it accepts. The goal of the function is to get the current weather in a given location; it accepts two parameters in input: one required (the city) and one optional (the unit of measure).

When you send this request to the LLM, you will get a response like this one:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


{
 "id": "chatcmpl-123",
 ...
 "choices": [{
 "index": 0,
 "message": {
 "role": "assistant",
 "content": null,
 "function_call": {
 "name": "get_current_weather",
 "arguments": "{ \"location\": \"Boston, MA\"}"
 }
 },
 "finish_reason": "function_call"
 }]
}

You’re getting a response from the LLM (the role is assistant), but the content property is empty. However, the function_call property contains a value: the LLM has realized that, in order to provide a response to the user, it needs to call the get_current_weather function, passing a JSON with the location property set to Boston, MA. With this information, it’s now up to you as a developer to call the function using the parameters provided by the LLM. For example, you could perform a REST call to the Weather APIs and get a response like the following one:

1
2
3
4
5


{
 "temperature": 20,
 "unit": "celsius",
 "location": "Boston, MA"
}

Once you have the response, you can now send again the original request. However, this time, in the messages collection you’re going to include also the information returned by the Weather API, as in the following example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28


{
 "model": "gpt-3.5-turbo-0613",
 "messages": [
 {"role": "user", "content": "What is the weather like in Boston?"},
 {"role": "assistant", "content": null, "function_call": {"name": "get_current_weather", "arguments": "{ \"location\": \"Boston, MA\"}"}},
 {"role": "function", "name": "get_current_weather", "content": "{\"temperature\": "22", \"unit\": \"celsius\", \"description\": \"Sunny\"}"}
 ],
 "functions": [
 {
 "name": "get_current_weather",
 "description": "Get the current weather in a given location",
 "parameters": {
 "type": "object",
 "properties": {
 "location": {
 "type": "string",
 "description": "The city and state, e.g. San Francisco, CA"
 },
 "unit": {
 "type": "string",
 "enum": ["celsius", "fahrenheit"]
 }
 },
 "required": ["location"]
 }
 }
 ]
}

You can see how the messages collection now includes:

The original prompt of the user, with role user.
The request to call a function, with role assistant, since it was generated by the LLM.
The JSON data you have obtained from the Weather API, with role function, since it’s the response we retrieved by calling the requested function.

Now the LLM has all the information it needs to generate a final answer to the user, so you’re going to get the following JSON response back:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


{
 "id": "chatcmpl-123",
 ...
 "choices": [{
 "index": 0,
 "message": {
 "role": "assistant",
 "content": "The weather in Boston is currently sunny with a temperature of 22 degrees Celsius.",
 },
 "finish_reason": "stop"
 }]
}

As you can see, this time the content property contains a value, while we don’t have anymore the function_call one. This means that the LLM has determined that it doesn’t need to call a function anymore to generate a response to the user, so the content property contains the final response. If the LLM would have determined, instead, that another function was needed, it would have returned another function_call property, with the name of the function and the parameters to pass to it and we would have repeated the same process.

Function calling and planners help to achieve a similar goal: we give to the LLM the tools we have, we define the ask and then we let the LLM figure out which tools are needed to satisfy the ask. The main difference is that, with function calling, we must call the function on our own, while with the planner, Semantic Kernel was taking care of calling function on our behalf. Semantic Kernel 1.0 brings the best of both worlds: instead of reinventing the wheel, the team has decided to use the same approach, but to add an extra layer that simplifies the workflow we have seen so far.

Semantic Kernel supports two way to manage functions:

Manually: you have full control over the function calling process. It’s the same flow we have seen in the previous example, but made it easier thanks to the usage of Semantic Kernel classes and methods.
Automatically: this is basically a replacement of the planner. With this approach, you let Semantic Kernel to call the functions on your behalf and to use the response to process the result.

Let’s see both of them in action. We’re going to use the native plugin we have migrated in the previous post: the one called UnitedStatesPlugin, which provides a GetPopulation function to retrieve the population of the US in a given year. We described the creation of this native plugin in this post.

Function calling: the manual approach

When you use the manual approach, you’re going to implement the following flow:

You send the prompt to the LLM.
You evaluate if the response includes a function calling.
If the answer is affirmative, you call the function and you get the response.
You add the response to the chat history.
You repeat again from step 1 until there are no more functions to call.

Let’s start by setting up the kernel and importing the plugins we have seen in the previous posts:

1
2
3
4
5
6
7
8
9


string apiKey = configuration["AzureOpenAI:ApiKey"];
string deploymentName = configuration["AzureOpenAI:DeploymentName"];
string endpoint = configuration["AzureOpenAI:Endpoint"];

var kernel = Kernel.CreateBuilder()
 .AddAzureOpenAIChatCompletion(deploymentName, endpoint, apiKey)
 .Build();

kernel.ImportPluginFromType<UnitedStatesPlugin>();

We have set up the service to use the Chat Completion APIs provided by Azure OpenAI and we imported the plugin in the kernel using the new ImportPluginFromType<T>() method. Make sure you’re using a recent version of gtp-3.5-turbo or gtp-4, which support function calling. The next step is to get a reference to the Chat Completion service provided by Semantic Kernel, which is a wrapper around the Chat Completion APIs.

1

var chatCompletionService = kernel.GetRequiredService<IChatCompletionService>();

We use the dependency injection approach to retrieve it. When we created the kernel using the AddAzureOpenAIChatCompletion() method, Semantic Kernel has created an IChatCompletionService object for us, so we just need to retrieve the one already registered by using the GetRequiredService<T>() method exposed by the kernel. Then, we define the prompt and we store it in the chat history:

1
2
3
4


string prompt = @"Write a paragraph to share the population of the United States in 2015.";

var chatHistory = new ChatHistory();
chatHistory.AddMessage(AuthorRole.User, prompt);

ChatHistory is a collection that simplifies the management of the messages history that you send to the LLM. It’s a wrapper around the messages collection that we have seen in the JSON request to the Chat Completion APIs. It offers various methods to add the various types of messages that the LLM supports. In this case, we’re using the AddMessage() method, passing as first parameter the value AuthorRole.User, which is translated with the following JSON entry in the history:

1
2
3
4


{
 "role": "user",
 "content": "Write a paragraph to share the population of the United States in 2015."
}

Before executing our prompt, we need a final step: we need to enable function calling. This is done by setting up an OpenAIPromptExecutionSettings object. It’s the same one we have learned about in the previous posts to customize the parameters of the execution, like temperature and maximum number of tokens. However, in this case, we’re going to use it to enable function calling by setting the ToolCallBehavior property to EnableKernelFunctions:

1
2
3
4


OpenAIPromptExecutionSettings settings = new()
{
 ToolCallBehavior = ToolCallBehavior.EnableKernelFunctions,
};

Now we’re ready to use the Chat Completion service to send the request to Azure OpenAI:

1

var result = await chatCompletionService.GetChatMessageContentAsync(chatHistory, settings, kernel);

We call the GetChatMessageContentAsync() method, passing as parameters the chat history, the execution settings and the kernel. We have completed the first part of the our workflow. Now let’s start the fun :-) We will get a response and we need to check if the LLM has determined that it needs to call one of the available plugins to generate a response.

1

var functionCalls = ((OpenAIChatMessageContent)result).GetOpenAIFunctionToolCalls();

We convert the result we receive from the service (which is a generic ChatMessageContent) into the specific OpenAIChatMessageContent object. We do this because this class offers a method called GetOpenAIFunctionToolCalls(), that will return a reference to the functions to call if the LLM has determined that it needs to call one of the available ones. Since the LLM can call one function at a time and it might need to call multiple functions, the method returns a collection, that we need to iterate to make sure we process all of them before we try to execute the prompt again. This is the complete code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


var functionCalls = ((OpenAIChatMessageContent)result).GetOpenAIFunctionToolCalls();
foreach (var functionCall in functionCalls)
{
 KernelFunction pluginFunction;
 KernelArguments arguments;
 kernel.Plugins.TryGetFunctionAndArguments(functionCall, out pluginFunction, out arguments);
 var functionResult = await kernel.InvokeAsync(pluginFunction!, arguments!);
 var jsonResponse = functionResult.GetValue<object>();
 var json = JsonSerializer.Serialize(jsonResponse);
 chatHistory.AddMessage(AuthorRole.Tool, json);
}

result = await chatCompletionService.GetChatMessageContentAsync(chatHistory, settings, kernel);

Console.WriteLine(result.Content);
Console.ReadLine();

The Plugins collection of the kernel offers a method called TryGetFunctionAndArgument() which, given a function definition, it’s going to return a reference to a function registered into the kernel. If the method is successful, we will get in output the function and the arguments, in both cases mapped with the Semantic Kernel classes we have learned about in the previous posts: KernelFunction and KernelArguments. Thanks to these objects, we can simply call the function using the InvokeAsync() method of the kernel. Once we obtain a result, we need to store it in the ChatHistory collection, so that the LLM can use it the next time. We use again the AddMessage() method, but this time we use the value AuthorRole.Tool, which is specific to store the results of a function. We pass, as parameters, the result and the function name. Since ChatHistory can store only string values, we need to serialize the result from the function first into a JSON string, by using the System.Text.Json serializer to do this.

Once we have processed all the functions, we call again the GetChatMessageContentAsync() method to get a new response from the LLM. This time, we don’t get a function call anymore, because the information in the chat history are enough for the LLM to elaborate a response, so we’ll get a value in the Content property of the result. And the final response is the one we expect, which we have already seen when we worked with the planner:

1

In 2015, the population of the United States was 316,515,021

Supporting multiple functions

The previous example required the usage of just a single function: GetPopulation. However, the same exact code works fine also in case the LLM needs to execute more than one function to achieve the result. Let’s see this with an example. Let’s change the UnitedStatesPlugin class to include a second function, which is able to retrieve the population number by gender:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


[KernelFunction, Description("Get the United States population who identifies with a specific gender in a given year")]
public async Task<UnitedStatesResponse> GetPopulationByGender([Description("The year")] int year, [Description("The gender")]string gender)
{
 string request = "https://datausa.io/api/data?drilldowns=Year,Gender&measures=Total+Population";
 HttpClient client = new HttpClient();
 var result = await client.GetFromJsonAsync<GenderResult>(request);
 var populationData = result.data.FirstOrDefault(x => x.Year == year.ToString() && x.Gender.ToLower() == gender);

 var response = new UnitedStatesResponse
 {
 Gender = gender,
 TotalNumber = populationData.TotalPopulation,
 Year = year
 };

 return response;
}

The code is very similar to the previous function, we’re just using a different endpoint provided by the DataUSA APIs. In addition, this function accepts two parameters in input instead of just one: the gender, other than the year.

Now let’s change our prompt in the following way:

1
2
3


string prompt = @"Write a paragraph to share the population of the United States in 2015.
Make sure to specify how many people, among the population, identify themselves as male and female.
Don't share approximations, please share the exact numbers.";

Now run the application again. This time, you will see that the collection returned by the GetOpenAIFunctionToolCalls() contains three functions:

The GetPopulation function, to get the total population number for 2015.
The GetPopulationByGender function, to get the population who identifies as male in 2015.
The GetPopulationByGender function, this time to get the population who identifies as female in 2015.

If we add a statement to print the function output in the console at every execution, we will see the following output:

1
2
3


{"year":2015,"totalNumber":316515021,"gender":null}
{"year":2015,"totalNumber":155728568,"gender":"male"}
{"year":2015,"totalNumber":160786456,"gender":"female"}

Finally, after these three loops, the LLM will have all the information it needs to generate a response, so we can break the loop and display the result:

1

In 2015, the population of the United States was 316,515,021. Out of this total, 155,728,568 individuals identified themselves as male, and 160,786,456 identified themselves as female. These figures represent the exact count of the population by gender for that year.

As you can see, the LLM has the capacity to not just call the functions it needs from the plugins which are registered in the kernel, but also to call the same function multiple times if needed (like in this case, in which it needed to call the GetPopulationByGender function twice, one for males and one for females).

Function calling: the automatic approach

Compared to the REST based implementation provided by OpenAI, Semantic Kernel is able to simplify the process of function calling even if we use the manual approach. We are in charge of detecting if the LLM needs to call a function, retrieve it, execute it and then pass the response, but instead of using plain APIs and JSON, we can use classes and objects. Additionally, the whole function registration is simplified thanks to the usage of plugins.

However, Semantic Kernel 1.0 offers also an automatic approach, which is even simpler. This approach replaces the planner, which was available in the previous versions of Semantic Kernel, since it allows Semantic Kernel not just to figure out which are the right functions to call, but also to call them and to manage automatically the whole chat history. Let’s see how it works.

First, we have to change the OpenAIPromptExecutionSettings object, by changing the FunctionCallBehavior property to AutoInvokeKernelFunctions.

1
2
3
4


OpenAIPromptExecutionSettings settings = new()
{
 FunctionCallBehavior = FunctionCallBehavior.AutoInvokeKernelFunctions
};

Now, since everything is managed automatically, we don’t need to use the ChatCompletionService object anymore, but we can just invoke the prompt. The following example uses the streaming approach:

1
2
3
4
5
6
7
8
9


string prompt = @"Write a paragraph to share the population of the United States in 2015.
Make sure to specify how many people, among the population, identify themselves as male and female.
Don't share approximations, please share the exact numbers.";

var streamingResult = kernel.InvokePromptStreamingAsync(prompt, new KernelArguments(settings));
await foreach (var streamingResponse in streamingResult)
{
 Console.Write(streamingResponse);
}

The only difference is that, as second parameter, we must pass a new KernelArguments instance with the execution settings, which will tell to the kernel to use functions and to automatically invoke them when needed.

That’s it. Now if you execute this code, you will get the same output as the manual approach:

1

In 2015, the population of the United States was 316,515,021. Out of this total, 155,728,568 individuals identified themselves as male, and 160,786,456 identified themselves as female. These figures represent the exact count of the population by gender for that year.

Behind the scenes, Semantic Kernel, like before, has figured out that it needs to call the GetPopulation function once and the GetPopulationByGender function twice to generate the response. This time, however, it has also called the functions on our behalf, so we get the final response without having to write any additional code.

Wrapping up

This was a long post! We have learned what function calling is and how it works. Then, we have seen how Semantic Kernel 1.0 supports it and how we can use it in two different ways: manually and automatically. We have also learned how function calling has replaced the need of using the planner for many scenarios, since function calling was built to solve the same problem: giving access to the LLM to a series of tools and let it figure out which ones to use to satisfy the ask of the user. Semantic Kernel takes this feature to the next level, by enabling the LLM not just to figure out which functions to call, but also to call them on our behalf.

Does this mean that the planner is dead? Not at all. There are scenarios, in fact, in which you need to perform more complex tasks, which requires more reasoning and more loops from the LLM in order to figure out the actions to perform. In the next post, we’re going to learn more about two new types of planners that have been introduced in 1.0.

In the meantime, you can find the whole sample code demonstrated in this post in this GitHub repository, more precisely in the SemanticFunction.FunctionCalling project.

Happy coding!

Semantic Kernel - Updating your native functions to 1.0

Thu, 21 Dec 2023 00:00:00 +0000

In the previous posts, we have learned the changes introduced in Semantic Kernel 1.0 which affected the setup of the project and the usage of semantic functions, now called prompt functions. In this post, we’ll focus on native functions: we’ll learn the changes we have to apply and also some new features that were introduced. This post will be shorter than the others, since the changes were minor, mainly the renaming of a few attributes and methods. For this reason, in this post we’re going to cover both native functions and OpenAI plugins. However, these changes are the gateway to the most important changes we’re going to see in the next post: function calling and planners.

But let’s focus on native plugins for the moment. The starting point is the same one we have used in our original examples: two flavors of the same plugin (a native one and an OpenAI one) to call the DataUSA APIs to get information about the US population.

Upgrading the native plugin

The first step when you upgrade your project to Semantic Kernel 1.0 is to change the code of the plugin, since the attributes used to turn a method into a native function have changed. Let’s take a look at the new code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


public class UnitedStatesPlugin
{
 [KernelFunction, Description("Get the United States population for a specific year")]
 public async Task<string> GetPopulation([Description("The year")]int year)
 {
 string request = "https://datausa.io/api/data?drilldowns=Nation&measures=Population";
 HttpClient client = new HttpClient();
 var result = await client.GetFromJsonAsync<UnitedStatesResult>(request);
 var populationData = result.data.FirstOrDefault(x => x.Year == year.ToString());
 string response = $"The population number in the United States in {year} was {populationData.Population}";
 return response;
 }
}

The code of the function is the same one we have used in the previous post, but the attribute has changed. The KernelFunction attribute, in fact, has replaced the old SKFunction one.

Adding the plugin to the kernel

The way you add a native plugin to the kernel has changed as well. You don’t need anymore to create a new instance of the plugin class, but you can use the ImportPluginFromType<T>() method, which accepts a generic parameter, which is the type of the plugin we want to add. Here is the complete example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


string apiKey = "AzureOpenAI:ApiKey";
string deploymentName = "AzureOpenAI:DeploymentName";
string endpoint = "AzureOpenAI:Endpoint";
string modelId = "AzureOpenAI:ModelId";


var kernel = Kernel.CreateBuilder()
 .AddAzureOpenAIChatCompletion(deploymentName, endpoint, apiKey)
 .Build();

kernel.ImportPluginFromType<UnitedStatesPlugin>();

Now we can call the plugin in the usual way. The only difference to keep in mind (which, however, we have already learned about in the previous post about prompt functions), is that now functions are stored in the Plugins collection of the kernel:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


var function = kernel.Plugins.GetFunction("UnitedStatesPlugin", "GetPopulation");

KernelArguments variables = new KernelArguments()
{
 { "year", "2015" }
};

var result = await kernel.InvokeAsync(function, variables);

Console.WriteLine(result.GetValue<string>());
Console.ReadLine();

Importing an OpenAI plugin

In another post of the original series, we have learned how we can reuse a plugin built for OpenAI in Semantic Kernel. These plugins are simply wrappers around REST APIs, with two peculiar features:

They are described by an OpenAPI definition.
They have an OpenAI manifest, which is a JSON defile that describes the goal of the plugin and the location of the OpenAPI definition, which is then used by OpenAI to figure out how to call the plugin.

Semantic Kernel 1.0 brings a few changes for this scenario. The first one is that OpenAI plugins are now managed by a different NuGet package. As such, you must add also the Microsoft.SemanticKernel.Plugins.OpenApi package to your project before using this feature. Another difference is that now you must use the ImportPluginFromOpenAIAsync() method, instead of the old ImportOpenAIPluginFunctionsAsync() one. The final important change to consider is that OpenAI plugins support has been marked as experimental and, as such, you will need to suppress the warning coming from the compiler otherwise the project won’t build. The following code shows an updated example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19



#pragma warning disable SKEXP0042 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.

const string pluginManifestUrl = "https://semantickernel-unitedstatesdata.azurewebsites.net/api/.well-known/ai-plugin.json";
await kernel.ImportPluginFromOpenAIAsync("UnitedStatesPlugin", new Uri(pluginManifestUrl));

var function = kernel.Plugins.GetFunction("UnitedStatesPlugin", "GetPopulation");

KernelArguments variables = new KernelArguments
{
 { "year", "2020" }
};

var result = await kernel.InvokeAsync(function, variables);

Console.WriteLine(result.GetValue<RestApiOperationResponse>().Content);
Console.ReadLine();

#pragma warning restore SKEXP0042 // Type is for evaluation purposes only and is subject to change or removal in future updates. Suppress this diagnostic to proceed.

The whole snippet of code which is using classes coming from the Microsoft.SemanticKernel.Plugins.OpenApi namespace has been wrapped inside a #pragma directive, which suppresses the warning.

Once we have imported the plugin, the rest of the code to execute the included function (GetPopulation) is the same we have seen for native functions.

Wrapping up

In this post, we have learned the minor changes that we have to apply to our code to use native and OpenAI plugins. Compared to the other posts, the changes were mainly cosmetic, with the goal to align the naming of methods and classes with the rest of the kernel. If you have found this post to be too short, don’t worry: the next post will be much longer :-) We’re going to learn, in fact, the big changes the affected the planner and the way you can automatically orchestrate AI workflows.

In the mean time, you can find the updated sample on GitHub.

Semantic Kernel - From semantic functions to prompt functions

Wed, 20 Dec 2023 00:00:00 +0000

The recent 1.0 release of Semantic Kernel introduced multiple breaking changes, with the goal to align the naming and the features with the AI industry. In the previous post, we focused on the basic scenarios, like setting up the connection to an AI service and executing a basic prompt. In this post, instead, we’ll cover one of the features that was heavily affected by the breaking changes: semantic functions, which are now called prompt functions. In this post, we’re going to review the changes and see how to migrate our existing code to the new version.

From semantic functions to prompt functions

Due to the rebranding into prompt functions, now we have also a different method to import a semantic function inside the kernel. Instead of using ImportSemanticFunctionsFromDirectory(), we must use a new method called ImportPluginFromPromptDirectory(), as in the following example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


string apiKey = configuration["AzureOpenAI:ApiKey"];
string deploymentName = configuration["AzureOpenAI:DeploymentName"];
string endpoint = configuration["AzureOpenAI:Endpoint"];

var kernel = Kernel.CreateBuilder()
 .AddAzureOpenAIChatCompletion(deploymentName, endpoint, apiKey)
 .Build();

var kernel = kernelBuilder.Build();

var pluginsDirectory = Path.Combine(Directory.GetCurrentDirectory(), "Plugins", "MailPlugin");

kernel.ImportPluginFromPromptDirectory(pluginsDirectory, "MailPlugin");

Compared to the previous code, there’s an important difference. In the original version, it was enough to pass as parameter the path of the entire Plugins folder (YourProject/Plugins), then the kernel was able to figure out the specific plugin folder to use by leveraging the name of the plugin (MailPlugin, in our case). In the new version, we must pass, instead, the full path of the folder which contains the specific plugin (YourProject/Plugins/MailPlugin, in our case).

Changing the configuration of the prompt

When you create a prompt function using the classic approach we have learned about in the original post, you have two files inside the folder which represents your plugin: one for the prompt (named skprompt.txt) and one for the configuration (named config.json). The configuration file includes the configuration of the prompt, like the LLM parameters, the description, etc. This is an example of the configuration file we have seen in the original post:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


{
 "schema": 1,
 "type": "completion",
 "description": "Write a business mail",
 "completion": {
 "max_tokens": 500,
 "temperature": 0.0,
 "top_p": 0.0,
 "presence_penalty": 0.0,
 "frequency_penalty": 0.0
 },
 "input": {
 "parameters": [
 {
 "name": "input",
 "description": "The text to convert into a business mail.",
 "defaultValue": "",
 "required": true
 }
 ]
 }
}

Semantic Kernel 1.0 introduced a few minor changes in the way you define the input and output parameters, as in the following example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


{
 "schema": 1,
 "type": "completion",
 "description": "Write a business mail",
 "completion": {
 "max_tokens": 500,
 "temperature": 0.0,
 "top_p": 0.0,
 "presence_penalty": 0.0,
 "frequency_penalty": 0.0
 },
 "input_variables": [
 {
 "name": "input",
 "description": "The text to convert into a business mail.",
 "is_required": true,
 "default": ""
 }
 ]
}

As you can notice, we don’t have anymore the input property with, as a child, the parameters collection. Instead, we have a new property called input_variables, which is a collection of objects, each of them representing a parameter. The defaultValue property has been renamed to default and the required property has been renamed to is_required. Changing the configuration of the prompt to use the new schema is critically important when we use planners and automatic function calling, otherwise they won’t be able to figure out which the input parameter are accepted by the prompt.

Executing the function

Another important change involves function storage. Functions are no longer stored in the kernel’s Functions collection. Instead, they now reside in the Plugins collection. This means that, to get a reference to a function, we must call the GetFunction() method exposed by the Plugins collection passing, as before, the name of the plugin and the name of the function we want to use.

1
2
3
4
5
6
7
8


var function = kernel.Plugins.GetFunction("MailPlugin", "WriteBusinessMail");

KernelArguments variables = new KernelArguments
{
 { "input", "Tell David that I'm going to finish the business plan by the end of the week." }
};

var result = await kernel.InvokeAsync(function, variables);

The rest of the changes are the same ones we have learned about in the previous post:

ContextVariables has been renamed to KernelArguments.
The RunAsync() method has been renamed to InvokeAsync() and we must pass the parameters in the reverse order (first the function, then the variables). If you prefer, you can use also the InvokeStreamingAsync() method, if you prefer to stream the response as it gets generated, instead of waiting for it to be fully available.

Using YAML to define a prompt function

One of the most relevant new features in 1.0 around prompt functions is the ability to define them using a YAML file. The advantage of this approach, over the traditional one, is that we don’t have anymore to maintain two different files, one for the prompt and one for the configuration, but we can consolidate everything in a single file. Let’s see how it works.

The first step is to add a YAML file in your project. When you use this approach, you don’t have to follow a specific folder structure for your plugin, since we’re going to read the YAML file as a string. To keep a consistent structure, I’ve created a folder called MailPluginYaml inside the Plugins folder and I’ve added a file called WriteBusinessMail.yaml, which is defined like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


name: WriteBusinessMail
template: |
 Rewrite the text between triple backticks into a business mail. Use a professional tone, be clear and concise. Sign the mail as AI Assistant. Text: ```{{$input}}```.
template_format: semantic-kernel
description: A function that generates a business mail.
input_variables:
 - name: input
 description: The text to be rewritten into a business mail.
 is_required: true
output_variable:
 description: The generated business mail.

The declaration should be easy to understand: with the various properties, we define the key features of the function, like the name, the template, the description, the inputs and the outputs. All these information will be used by Semantic Kernel to determine if the function is a good candidate to solve a specific problem.

Now that we have a prompt function defined using a YAML file, the only difference is the method we must use to load the plugin in the kernel:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


var writeMailYaml = File.ReadAllText($"{pluginsDirectory}\\MailPluginYaml\\WriteBusinessMail.yaml");
var function = kernel.CreateFunctionFromPromptYaml(writeMailYaml);

KernelArguments variables = new KernelArguments
{
 { "input", "Tell David that I'm going to finish the business plan by the end of the week." }
};

var result = await kernel.InvokeAsync(function, variables);

Console.WriteLine(result.GetValue<string>());
Console.ReadLine();

First, we use File.ReadAllText() to read the content of the YAML file as a string. Then, we pass the string to the CreateFunctionFromPromptYaml() method, which returns a reference to the function. The rest of the code is the same as the previous example: we define the input variables using a KernelArguments collection and we pass it to the InvokeAsync() method, together with the function.

It’s important to highlight that the CreateFunctionFromPromptYaml() method only generates a function out of the YAML definition, but it doesn’t store it into the kernel. This means that you can use this semantic prompt if you explicitly invoke it (like we did in the previous example), but if you want to use a more automated approach (like with the planner), it wouldn’t work.

If you want to store it into the kernel, you have to use the KernelPluginFactory helper to create a KernelPlugin object and then import it using the ImportPluginFromObject() method, as in the following example:

1
2
3
4
5


var writeMailYaml = File.ReadAllText($"{pluginsDirectory}\\MailPluginYaml\\WriteBusinessMail.yaml");
var function = kernel.CreateFunctionFromPromptYaml(writeMailYaml);

var plugin = KernelPluginFactory.CreateFromFunctions("MailPlugin", new[] { function });
kernel.Plugins.Add(plugin);

In our case, we already have a function that we want to embed into a plugin, so we use the CreateFromFunctions() method of the KernePluginFactory class, which accepts as input the name of the plugin and a collection of functions which, in our case, contains only one element.

Wrapping up

In this post, you have learned how to migrate semantic functions to prompt functions, using the new syntax and the new features included in Semantic Kernel 1.0. You have also learned how to transition from the classic plugins approach, based on a text file with the prompt and a configuration file with the parameters, to a new approach based on a single YAML file.

You will find the updated example in the GitHub repository of the project.

Happy coding!

Semantic Kernel - Upgrading your projects to 1.0

Tue, 19 Dec 2023 00:00:00 +0000

After a long preview, Microsoft finally released version 1.0 of Semantic Kernel! The good news is that the SDK is now stable and it has matured a lot over time, introducing new features, improving performances and aligning naming and features to the latest innovations introduced by OpenAI. The bad news is that all these features comes with a cost: 1.0 introduces tons of breaking changes, which means that you will have to apply lot of updates to your code.

In this post, we’re going to review the basic changes, that must be applied to every project, like the new way to setup the kernel or to invoke a function. In the next posts, instead, we’ll see in more details the deeper changes that affected some of the features we discussed in this blog, like semantic functions (now called prompt functions) and native plugins.

Setting up the kernel

The first important change is the way we setup the kernel. With the goal to align the naming with the ones used in the AI industry by OpenAI and Hugging Face, all the methods to setup an AI service have been renamed. The new names are now the following:

AddChatCompletion() to use chat completion models.
AddTextEmbeddingGeneration() to use embedding models.
AddTextGeneration() to use text generation models.
AddTextToImage() to use image generation models.

All these methods, as before, are available in two variants, based on the AI service you want to use. For example, if you want to use a chat completion model, you’re going to use:

AddOpenAIChatCompletion() to use OpenAI models.
AddAzureOpenAIChatCompletion() to use Azure OpenAI models.

Another important change is the way you initialize the kernel. Instead of manually creating a new instance of the KernelBuilder class, now you must obtain a KernelBuilder object using the CreateBuilder() method, which is a static method of the Kernel class. Here is an example of the new initialization code:

1
2
3
4
5
6
7


string apiKey = "AzureOpenAI:ApiKey";
string deploymentName = "AzureOpenAI:DeploymentName";
string endpoint = configuration"AzureOpenAI:Endpoint";

var kernel = Kernel.CreateBuilder()
 .AddAzureOpenAIChatCompletion(deploymentName, endpoint, apiKey)
 .Build();

Executing a basic function

The way you just execute a prompt (without using any plugin) has also changed. Let’s take a look at the following example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23


string prompt = """
Rewrite the text between triple backticks into a business mail. Use a professional tone, be clear and concise.
Sign the mail as AI Assistant.

Text: ```{{$input}}```
""";

var mailFunction = kernel.CreateFunctionFromPrompt(prompt, new OpenAIPromptExecutionSettings
{
 Temperature = 0.7,
 MaxTokens = 1000,
});

KernelArguments arguments = new KernelArguments
{
 { "input", "Tell David that I'm going to finish the business plan by the end of the week." }
};


var response = await kernel.InvokeAsync(mailFunction, arguments);

Console.WriteLine(output.GetValue<string>());
Console.ReadLine();

The first change is that, to create a function out of a prompt, we must use a new method called CreateFunctionFromPrompt(). Additionally, if we want to customize the LLM parameters, we must supply a new object to the method called OpenAIPromptExecutionSettings, which offers various properties to customize the execution of the prompt. In the example, you can see that we have changed the values of Temperature and MaxTokens.

Another important change is that the ContextVariables collection has been renamed to KernelArguments. However, it works exactly in the same way as before: you supply a collection of key / value pairs, where each of them is a variable that was defined in the prompt.

Finally, also the way we invoke a function has changed: the RunAsync() method has been renamed to InvokeAsync() and we must pass the parameters in the reverse order (first the function, then the input variables).

Making the code shorter and simpler

Semantic Kernel 1.0 has added also a simpler way to invoke a prompt, thanks to the InvokePromptAsync() method. This is how we can simplify the previous code:

1
2
3
4
5
6


var response = await kernel.InvokePromptAsync(prompt, arguments: new() {
 { "input", "Tell David that I'm going to finish the business plan by the end of the week." }
});

Console.WriteLine(output.GetValue<string>());
Console.ReadLine();

As you can see, we don’t have to create a function out of the prompt anymore, we can just invoke it directly. The only parameter we must pass is the prompt itself and the KernelArguments collections with the variables we need to pass for the prompt parameters.

Streaming the response

If you have ever played with ChatGPT, you have noticed that responses aren’t returned as a “one shot message”, but they are “streamed”: a few words at a time, the response is gradually composed in front of the eyes of the user. This approach isn’t artificially used to create a more compelling visual effect, but it’s used to mitigate the fact that LLMs takes a while before they can generate a full response. Simplifying a lot, in fact, what a LLM does is “guessing” the next word based on the previous one, using probability. This means that, to generate a full response, the LLM must “guess” the next word, then the next one, then the next one, and so on. This process takes time, so the response is gradually streamed to the user to provide a better user experience. Semantic Kernel 1.0 offers an easy way to implement the same approach, by proving a variant of the InvokeAsync() and InvokePromptAsync() methods, called InvokeStreamingAsync() and InvokeStreamingPromptAsync(). Let’s see how the previous example can be changed to use a streaming response:

1
2
3
4
5
6
7
8
9


var response = kernel.InvokePromptStreamingAsync(prompt, arguments: new()
{
 { "input", "Tell David that I'm going to finish the business plan by the end of the week." }
});

await foreach (var message in response)
{
 Console.Write(message);
}

These methods make use of one of the most recent features introduced in C#, which are asynchronous iterators. We call the InvokePromptStreamingAsync() method (which works exactly like the standard variant, which means you must provide the prompt and the input variables) and we iterate over the result using the await foreach statement. The result is a collection of KernelResponse objects, which contains the response generated by the LLM and which gets continuously populated as new content gets generated.

Wrapping up

In this post, we have learned how to migrate a Semantic Kernel project to use the new 1.0 version. In the next posts, we’ll see in more details the changes that affected the semantic functions, the native plugins and the planner.

You can find the updated projects on GitHub.

Happy coding!

Semantic Kernel - Planner

Tue, 14 Nov 2023 00:00:00 +0000

Updated on 17th November 2023 to reflect the latest changes in the Semantic Kernel APIs:

All the Azure OpenAI extension methods have been renamed from Azure to AzureOpenAI. For example, WithAzureChatCompletionService() is now WithAzureOpenAIChatCompletionService().

The method to import an OpenAI plugin into the kernel has been renamed from ImportPluginFunctionsAsync() to ImportOpenAIPluginFunctionsAsync(). Please be aware that there’s also a new method to import directly an OpenAPI definition called ImportOpenApiPluginFunctionsAsync(). As you can see, the two names are very similar, so make sure to use the right one.

After reading the previous posts, especially the ones about native plugins and OpenAI plugins, you probably have started to grasp the value of plugins, but in the end we could have achieved the same goal by directly calling a REST API. So why bother with plugins? In this post, we’re going to explore the concept of orchestration and how Semantic Kernel can help you to automatically manage plugins through a tool called planner. Thanks to the planner, you’ll be able to load multiple plugins, define a task and let Semantic Kernel figure out which are the right plugins to call and in which order to satisfy the ask.

Manual orchestration

First, let’s see a simpler example to understand the Semantic Kernel capabilities to manage orchestration. In all the examples we have seen so far, we have used the kernel to execute a single function, using the RunAsync() method. However, the same method can be used to invoke multiple functions and automatically chain the output together.

Let’s say we want to generate a mail to share the number of the US population given a specific year. We can combine the semantic function and the native function (or the OpenAI plugin) we have built to achieve this goal. Assuming that you have already created them, following the guidance of the previous posts, the first step is to load both of them in the kernel:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


string apiKey = "api-key"
string deploymentName = "deployment-name";
string endpoint = "endpoint";

var kernelBuilder = new KernelBuilder();
kernelBuilder.
 WithAzureOpenAIChatCompletionService(deploymentName, endpoint, apiKey);

var kernel = kernelBuilder.Build();

const string pluginManifestUrl = "https://localhost:7071/api/.well-known/ai-plugin.json";
await kernel.ImportOpenApiPluginFunctionsAsync("UnitedStatesPlugin", new Uri(pluginManifestUrl));

var pluginsDirectory = Path.Combine(System.IO.Directory.GetCurrentDirectory(), "Plugins");
kernel.ImportSemanticFunctionsFromDirectory(pluginsDirectory, "MailPlugin");

The previous code is a mix of the code we have seen in different posts. The first part is the code to load the OpenAI plugin using the ImportOpenApiPluginFunctionsAsync() method, while the second part loads the MailPlugin using the ImportSemanticFunctionsFromDirectory() method. Now we can get a reference to the two functions we need to use:

1
2


var mailFunction = kernel.Functions.GetFunction("MailPlugin", "WriteBusinessMail");
var populationFunction = kernel.Functions.GetFunction("UnitedStatesPlugin", "GetPopulation");

The first one is WriteBusinessMail, which is the semantic function included in the MailPlugin. The second one is the GetPopulation function, which is the native function included in the UnitedStatesPlugin. Now we can use the RunAsync() method to invoke both functions and chain the output together:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


ContextVariables variables = new ContextVariables
{
 { "year", "2018" }
};

var result = await kernel.RunAsync(
 variables,
 populationFunction,
 mailFunction
);

Console.WriteLine(result.GetValue<string>());
Console.ReadLine();

We need to supply only the input for the first function. Automatically, the kernel will take the output of the first function and use it as input for the second function. The result of the second function will be the result of the whole chain. In our scenario, we execute first the GetPopulation function to retrieve the number of the US population in a given year, so we use the ContextVariables collection to provide the year. The information about the US population in 2018 will be then passed to the WriteBusinessMail function to generate a mail. In this case, the output will be similar to the following mail:

1
2
3
4
5
6
7
8
9


Dear [Recipient],

I am writing to inform you that the population number in the United States in 2018 was 322,903,030. I hope this information is useful to you.

If you have any further questions or concerns, please do not hesitate to contact me.

Best regards,

AI Assistant

Pretty cool, right? However, if the outcome is quite impressive, the way we got there is less exciting. First of all, we need to know in advance which functions we need to call and in which order. This is not a big deal in our example, but it can be a problem in more complex scenarios. Moreover, we need to know the input and output of each function, so we can chain them together. If, for example, we would have passed the functions in a reverse order to the RunAsync() method, Semantic Kernel wouldn’t have been able to produce a meaningful result, since the output of the WriteBusinessMail function (a mail text) is not a valid input for the GetPopulation function (which expects, instead, a year).

Let’s see now how we can improve this scenario using the planner.

Automatic orchestration with the planner

Thanks to the planner component of Semantic Kernel, we can start from the same premise (we want to generate a mail to share the number of the US population given a specific year) but, this time, we’ll let the kernel figure out which functions to call and in which order. The first step is the same as before: we need to load inside the kernel all our plugins, so you can reference the code we have already used.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


string apiKey = "api-key"
string deploymentName = "deployment-name";
string endpoint = "endpoint";

var kernelBuilder = new KernelBuilder();
kernelBuilder.
 WithAzureOpenAIChatCompletionService(deploymentName, endpoint, apiKey);

var kernel = kernelBuilder.Build();

const string pluginManifestUrl = "https://localhost:7071/api/.well-known/ai-plugin.json";
await kernel.ImportOpenApiPluginFunctionsAsync("UnitedStatesPlugin", new Uri(pluginManifestUrl));

var pluginsDirectory = Path.Combine(System.IO.Directory.GetCurrentDirectory(), "Plugins");
kernel.ImportSemanticFunctionsFromDirectory(pluginsDirectory, "MailPlugin");

What changes is that, this time, we don’t need to get a reference to the functions we want to call, but we just need to define the ask we want to satisfy and run it through the planner:

1
2
3
4
5
6
7
8


var planner = new StepwisePlanner(kernel);

var ask = "Write a mail to share the number of the United States population in 2015 for a research program.";
var originalPlan = planner.CreatePlan(ask);
var originalPlanResult = await kernel.RunAsync(originalPlan);

Console.WriteLine(originalPlanResult.GetValue<string>());
Console.ReadLine();

There are different type of planners available in Semantic Kernel. For our scenario, we’re going to use the StepwisePlanner, which evaluates the sequence to perform step by step. After we have defined the ask (which is a simple prompt that explains the goal to achieve), we create a plan using the CreatePlan() method. The plan is a sequence of functions that the planner has identified as the best way to satisfy the ask. Once we have a plan, we can execute it by passing it to the usual RunAsync() method of the kernel.

This is the result we’re going to get:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


The email text to share the number of the United States population in 2015 for a research program is:

Subject: Sharing Latest Data on United States Population

Dear Research Program,

I hope this email finds you well. I am writing to share with you the latest data on the United States population in 2015. As per the recent statistics, the population was 316,515,021.

I believe this information will be useful for your ongoing research program. If you require any further assistance or information, please do not hesitate to contact me.

Thank you for your time and consideration.

Best regards,

AI Assistant

As you can see, without having to specify which plugins and functions we wanted to use, Semantic Kernel has been able to satisfy our ask: writing a mail to share the number of the US population in 2015. From the ask, the planner has been able to automatically figure out:

Which plugins and functions to use (GetPopulation and WriteBusinessMail)
The order to follow (GetPopulation first, then WriteBusinessMail)
Which is the input to provide to each function (it automatically extracted from our ask the year 2015 and used it as input for the GetPopulation function)

The interesting part is that, by analyzing the plan, we can see what happened behind the scenes. If you put a breakpoint after the result has been generated, we can explore the content of the originalPlanResult variable and see, inside the Metadata collection, the thought process implemented by Semantic Kernel:

We can see that the plan is made by three steps and it’s going to use two different functions: GetPopulation and WriteBusinessMail. The most interesting information, however, can be found in the stepsTaken property:

As you can see, the property includes a detailed description of the reasoning made by the planner to achieve our ask:

1

To write a mail, we can use the MailPlugin.WriteBusinessMail function to generate the text of the email. To get the population of the United States in 2015, we can use the UnitedStatesPlugin.GetPopulation function with the year parameter set to 2015.

This is the same reasoning we have done manually in the previous section, but this time it has been done automatically by Semantic Kernel with the help of the reasoning capabilities offered by LLMs.

Testing a plan with Visual Studio Code

The Visual Studio Code extensions we have learned about in the post dedicated to semantic functions can be used also to test a plan. This way, we can evaluate if the plugins we have built are enough to satisfy our ask before starting to write any code. Before we see how to do that, it’s important to highlight that this feature has an important limitation: it works only with semantic functions. As such, we wouldn’t have been able to test the plan we have created in this post, since it includes a native plugin.

As such, for the purpose of this demo, I’ve created a second plugin, called AskPlugin, which includes a semantic function called Ask, which defines a prompt to simply ask to the LLM to answer a question for us:

1
2
3


Answer the question below in a concise way.

{{$input}}

Now we have two semantic functions we can use to test a plan. Once you open the Semantic Kernel extension in Visual Studio Code, expand the Plan section and click on the button to create a new plan:

As first thing, you will be asked for a folder where to save the plan, which is stored with a series of JSON and text files. As next step, the extension will identify all the semantic functions available in the plugins you have included in the folder, so that you can pick up the ones you want to include in the plan:

For my scenario, I have picked them both.

The next step is to give a name to the plan. Finally, you will be asked to provide a goal for the plan, which is the equivalent of the ask we have seen in the previous section. For my test, I provided the following goal:

1

Answer the question "Which is the distance between Earth and Sun?" and share it via mail

Once the plan is created, you can run it by pressing the Execute plan button:

During the execution, you will see Visual Studio Code picking up the needed plugins and resolving the plan step by step. At the end, you will see the result of the plan:

You can see how the planner has used, first, the Ask function to answer the question “Which is the distance between Earth and Sun?” and then it used it as input for the WriteBusinessMail function to generate the mail text. We can see that our plan and our plugins are working as expected, without writing a single line of code.

Wrapping up

In this post, we have unlocked the full value of Semantic Kernel. Previously, we have started to hint at the value of Semantic Kernel and using plugins, but it’s only with the planner that we can realize its full potential. Even in scenarios in which we have complex workflows to manage, we don’t have to worry about picking the right functions and choosing the right order, or chaining the operations in the correct way. Semantic Kernel will do all the heavy lifting for us. This post included a simple scenario, in which we loaded two plugins and we used both of them to satisfy the ask. However, in a real application, we could have loaded dozens of plugins, and the planner would have picked up only the ones which are really needed to complete the requested task.

You can find the demo used in this post in the GitHub repository. Specifically,

The SemanticKernel.Orchestration project contains the manual orchestration example.
The SemanticKernel.Planner project contains the planner example.

Happy coding!

Semantic Kernel - OpenAI plugins

Fri, 10 Nov 2023 00:00:00 +0000

Updated on 17th November 2023 to reflect the latest changes in the Semantic Kernel APIs:

All the Azure OpenAI extension methods have been renamed from Azure to AzureOpenAI. For example, WithAzureChatCompletionService() is now WithAzureOpenAIChatCompletionService().

The method to import an OpenAI plugin into the kernel has been renamed from ImportPluginFunctionsAsync() to ImportOpenAIPluginFunctionsAsync(). Please be aware that there’s also a new method to import directly an OpenAPI definition called ImportOpenApiPluginFunctionsAsync(). As you can see, the two names are very similar, so make sure to use the right one.

In the posts I have published so far, we covered two types of plugins: semantic functions (to execute a prompt) and native plugins (to execute native code). Semantic Kernel supports a third way to create plugins, even if it isn’t entirely different from the one we have seen in the last post. We’re still talking, in fact, about a plugin capable of executing native code. However, this time the plugin won’t be hosted locally in the project, but it will run on a server, and it will follow the specifications of OpenAI plugins.

The advantage of this approach is that OpenAI plugins implement a standard approach and they are supported by multiple platforms: Semantic Kernel, of course, but also ChatGPT or Microsoft 365 Copilot. This means that, if you create a plugin for Semantic Kernel, you can reuse it in other platforms, without having to rewrite it. Or that if you have already developed and published an OpenAI plugin, you can reuse it as it is in Semantic Kernel.

In this post, we’re going to take the native plugin we built in the previous post (the one that allows retrieving the population of the United States in a given year) and we’re going to convert it to an OpenAI plugin. Then, we’re going to consume it in our Semantic Kernel command line application.

Creating an OpenAI plugin

An OpenAI plugin is, in the end, just a REST API, with two specific features:

You must have a manifest file, which describes the purpose of the plugin.
You must expose its definition using the OpenAPI format. Thanks to this definition, the platform which consumes the plugin can understand which features and methods are available.

I know it’s a bit confusing, but OpenAI and OpenAPI are two different things :-) OpenAI is the company behind ChatGPT and some of the most used LLMs in the market, like GPT3.5 and GPT4, while OpenAPI is a standard to describe REST APIs, which is independent from the language and the platform used to build them.

For the purpose of this post, I’m going to build the REST API using an Azure Function, which is the cheapest and easiest way to build an API and make it available on a remote host (even if, for the sake of testing, we won’t need to publish it, it can run also on a local server).

I’m going to build a .NET 7.0 based function, using the isolated model (which means that the function runs in an isolated worker model, which enables the function to be independent from the version of .NET installed on the host and also to support other runtimes, like Python). You can use Visual Studio or Visual Studio Code, it’s up to you to use the editor you prefer: the code we’re going to write is exactly the same.

The first step, before writing the code, is to install the NuGet package called Microsoft.Azure.Functions.Worker.Extensions.OpenApi. This package will greatly simplify the generation of an OpenAPI definition, which is required by the OpenAI to consume the plugin. Once the package is installed, move to the Program.cs file and add the following code:

1
2
3
4
5
6
7
8
9


using Microsoft.Azure.Functions.Worker.Extensions.OpenApi.Extensions;
using Microsoft.Extensions.Hosting;

var host = new HostBuilder()
 .ConfigureFunctionsWorkerDefaults()
 .ConfigureOpenApi()
 .Build();

host.Run();

Compared to the standard initialization, we have added the extension method ConfigureOpenApi().

Now we can write the code of the function, by implementing the Run() method in the function class:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


public async Task<HttpResponseData> Run(
 [HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = null)] HttpRequestData req, [FromQuery] string year)
{
 _logger.LogInformation("C# HTTP trigger function processed a request.");

 string request = "https://datausa.io/api/data?drilldowns=Nation&measures=Population";
 HttpClient client = new HttpClient();
 var result = await client.GetFromJsonAsync<UnitedStatesResult>(request);
 var populationData = result.data.FirstOrDefault(x => x.Year == year);

 string message = $"The population number in the United States in {year} was {populationData.Population}";

 var response = req.CreateResponse(HttpStatusCode.OK);
 await response.WriteStringAsync(message);

 return response;
}

The code is very similar to the one we have seen in the previous post, with the only difference that it’s encapsulated inside the function execution. As such, we have:

A HttpTrigger attribute, which specifies that the function is triggered by an HTTP request. We also specify that this API supports only GET requests.
A HttpRequestData parameter, which represents the request.
A parameter called year, which is prefixed by the [FromQuery] attribute. Thanks to it, we will have direct access in code to the value of the query string parameter called year.

The rest of the code is the same as the one we have seen in the previous post: we use the DataUSA API to retrieve the population data, then we filter it out for the requested year and we send it back to the caller, using a HttpResponseData object. Note that we’re not returning a string, but we’re writing it to the response using the WriteStringAsync() method. This is because the function must return an HTTP response, which is the standard way to communicate with a REST API.

If we press F5 in Visual Studio or Visual Studio Code, we’re going to launch the function through the Azure Function Core Tools, which enables the function to run on our local machine. Inside the terminal, we’ll see the endpoint which is exposing our REST API. The following screenshot shows the output in Visual Studio Code:

We can see that the function is running on port 7071, so we can test it by opening a browser and navigating to http://localhost:7071/api/GetPopulation and adding a query string parameter called year, like http://localhost:7071/api/GetPopulation?year=2015. If you did everything correctly, the output will be the same we have seen when we were using the native function in the previous post:

1

The population number in the United States in 2015 was 316515021

Adding the OpenAPI definition

With our REST API functional, the next step is to produce the OpenAPI definition that explains how the APIs work. The Microsoft.Azure.Functions.Worker.Extensions.OpenApi NuGet package we installed earlier greatly simplifies this process. We can generate the necessary documentation with some attribute decorations on our function:

1
2
3
4
5
6
7
8
9


 [Function("GetPopulation")]
 [OpenApiOperation(operationId: "GetPopulation", tags: new[] { "year" }, Description = "Get the United States population for a specific year")]
 [OpenApiParameter(name: "year", In = ParameterLocation.Query, Required = true, Type = typeof(string), Description = "The year")]
 [OpenApiResponseWithBody(statusCode: HttpStatusCode.OK, contentType: "text/plain", bodyType: typeof(string), Description = "The population number")]
 public async Task<HttpResponseData> Run(
 [HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = null)] HttpRequestData req, [FromQuery] string year)
 {
 // function implementation
 }

The first attribute we use is OpenApiOperation, which defines that this method represent an API endpoint. It has a specific identifier (the operationId) and, more importantly, a Description property, which is very important because it’s used by OpenAI to understand what the API does.

The second attribute is OpenApiParameter, which is used to specify that the API accepts an input parameter with the following characteristics:

It’s called year (the value of the name property).
It’s a query string parameter (the value of the In property is set to ParameterLocation.Query).
It’s required (the Required property is set to true)
It’s a string (the Type property is set to typeof(string))
It contains the year (provided through the Description property). Also in this case, this property is very important because it helps OpenAI to understand what the parameter is used for.

Finally, we have the OpenApiResponseWithBody attribute, which is used to specify that the API returns a response with the following characteristics:

It returns a 200 status code (the value of the statusCode property).
It returns a text/plain content type (the value of the contentType property).
It returns a string (the value of the bodyType property).
It returns the population number (provided through the Description property).

Now that we have provided all the information to describe our API, thanks to the OpenAPI extension the Azure Function will automatically generate for us the OpenAPI definition and a website based on Swagger, which enables developer to explore and interact with the API. To see it in action, we can press F5 again and, when the function starts, you will see that there are a few additional endpoints listed in the terminal:

If you open the RenderSwaggerUI endpoint, you will see a website that will allow you test the API in an interactive way:

This website is generated starting from the OpenAPI definition, which you can see by opening in a browser the endpoint labelled as RenderSwaggerDocument, which looks like http://localhost:7071/api/swagger.{extension}. As you can notice, there’s an extension placeholder because OpenAPI supports definitions both in JSON and YAML so, based on the extension you use, you will get a different output. This is, for example, the YAML definition:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29


swagger: 2.0
info:
 title: OpenAPI Document on Azure Functions
 description: This is the OpenAPI Document on Azure Functions
 version: 1.0.0
host: localhost:7071
basePath: /api
schemes:
- http
paths:
 /GetPopulation:
 get:
 tags:
 - year
 description: Get the United States population for a specific year
 operationId: GetPopulation
 produces:
 - text/plain
 parameters:
 - in: query
 name: year
 description: The year
 required: true
 type: string
 responses:
 200:
 description: The population number
 schema:
 type: string

Adding the OpenAI manifest

Now that we have an OpenAPI definition, we need to generate an OpenAI manifest to turn this function into a plugin. The OpenAI documentation describes where it should be located and how it looks like.

Let’s start from the first requirement. The manifest must be a JSON file called ai-plugin.json and it must be located in the following endpoint of your server:

1

/.well-known/ai-plugin.json

Inside the JSON, we must include a few properties to describe the purpose of the plugin and where to find the OpenAPI definition. The following is the manifest for our plugin:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


{
 "schema_version": "v1",
 "name_for_human": "United States population Plugin",
 "name_for_model": "UnitedStatesPlugin",
 "description_for_human": "Get the population of the United States in a specific year.",
 "description_for_model": "Help the user getting the population of the United States in a specific year.",
 "auth": {
 "type": "none"
 },
 "api": {
 "type": "openapi",
 "url": "{url}/api/swagger.json"
 },
 "logo_url": "{url}/logo.png",
 "contact_email": "info@developerscantina.com",
 "legal_info_url": "http://www.developerscantina.com"
}

The most important properties are the following ones:

name (forModel/forHuman): this is the name of the plugin, which will be used by the platform to identify it. The forModel property is used by the platform, while the forHuman property is used to display the name of the plugin to the user.
description (forModel/forHuman): this is the explanation of what the plugin does. It’s really important to provide a good description (especially in the forModel variant of the property), because this information will be used by OpenAI to understand if your plugin can be used to execute a given prompt.
api, which includes the information about how our API looks like. In this case, we’re using the OpenAPI type, which means that we’re going to provide the URL of the OpenAPI definition. We have included a placeholder, called {uri}, which we’re going to replace at runtime with the real URL. This way, we don’t have to change the manifest based on where the function is running (locally or on a remote server).

Since we’re using an Azure Function, we have an extra step to take. Being based on the serverless model, we can’t host and serve a static file like a JSON manifest, so we need to expose it through an API endpoint. As such, we’re going to add a new endpoint to our function, which will read the manifest file, it will replace the url placeholder and it will return it a string.

Let’s start by creating a folder in our project called manifest and, inside it, let’s create a file called ai-plugin.json. Then copy and paste the JSON definition I have previously included. Now let’s create the endpoint: add a new file to the project called AIPluginJson.cs, which will contain the following implementation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16


public class AIPluginJson
{
 [Function("GetAIPluginJson")]
 public async Task<HttpResponseData> Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = ".well-known/ai-plugin.json")] HttpRequestData req)
 {
 var currentDomain = $"{req.Url.Scheme}://{req.Url.Host}:{req.Url.Port}";
 var binDirectory = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
 var result = File.ReadAllText(binDirectory + "/manifest/ai-plugin.json");
 var json = result.Replace("{url}", currentDomain);

 var response = req.CreateResponse(HttpStatusCode.OK);
 await response.WriteStringAsync(json);

 return response;
 }
}

This is a new endpoint for our function, so we need to decorate it with the [Function] attribute and give it a name (GetAIPluginJson). The signature of the Run method is similar to the one we have used for the other API endpoint, since also in this case we’re using the HttpTrigger attribute to specify that the endpoint is triggered by an HTTP request. However, there’s a difference: this time we’re using the Route property to specify that the endpoint is exposed at the .well-known/ai-plugin.json endpoint. This is the location where the OpenAI platform will look for the manifest file.

The function implementation does two things:

It reads the content of the manifest file which we have previously created. Since this isn’t a regular web app, we can’t use relative paths, but we need to get the path starting from the location of the Function assembly.
It uses the HttpRequestData object to retrieve the current URL and it replaces the {url} placeholder with it.

If we launch again the function locally, we should see another endpoint being listed in the terminal:

By opening the endpoint in the browser, you should see the same manifest we have described before, just with the correct URL instead of the placeholder.

Now we have a fully compliant OpenAI plugin! If you have been accepted into the developers waitlist, you are able to test it with ChatGPT. However, for the moment, let’s focus on Semantic Kernel.

Using the OpenAI plugin with Semantic Kernel

Now it comes the easy part! The plugin in fact, works in the same way we have seen in the previous post, when we were using a native plugin. The only difference is that, this time, we’re going to use the ImportPluginFunctionsAsync() method instead of the ImportFunctions() one:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


string apiKey = "your-api-key";
string deploymentName = "your-deployment-name";
string endpoint = "your-endpoint";

var kernelBuilder = new KernelBuilder();
kernelBuilder.
 WithAzureOpenAIChatCompletionService(deploymentName, endpoint, apiKey);

var kernel = kernelBuilder.Build();

const string pluginManifestUrl = "https://localhost:7071/api/.well-known/ai-plugin.json";
await kernel.ImportOpenAIPluginFunctionsAsync("UnitedStatesPlugin", new Uri(pluginManifestUrl));

var function = kernel.Functions.GetFunction("UnitedStatesPlugin", "GetPopulation");

ContextVariables variables = new ContextVariables
{
 { "year", "2020" }
};

var result = await kernel.RunAsync(
 variables,
 function
);

Console.WriteLine(result.GetValue<RestApiOperationResponse>().Content);
Console.ReadLine();

First we define a constant with the URL of the manifest endpoint, then we import it into the kernel by calling the ImportPluginFunctionAsync() method, passing as parameters the name of the plugin and the manifest URL. Then, we use the plugin in the same way we did in the previous post: we use the GetFunction() method to retrieve a reference to the GetPopulation function, we create a ContextVariables dictionary with the input (the year) and then we pass both of them to the RunAsync() method of the kernel.

The only difference compared to what we did last time is that, since we’re using an OpenAI plugin, the result we get back is a RestApiOperationResponse object, which contains the HTTP response returned by the plugin, and not a plain string. As such, to retrieve the result, we must retrieve the content of the response by using the Content property.

If you run the code, you should see the same result we have seen in the previous post:

1

The population number in the United States in 2015 was 316515021

However, this time, the response isn’t coming from an internal plugin, but from an API published on a server, that can be easily reused with other platforms like ChatGPT and Microsoft 365 Copilot.

Wrapping up

In this post, we have moved our native function into an OpenAI plugin. From an outcome perspective, the result is the same, but it makes our solution even more interoperable, since it enables us to reuse our logic in multiple platforms. So far, however, we have only seen a basic usage of plugins, which it has some benefits (easier to scale and reuse, interoperability, etc.) but it’s still limited to a single function. In the next post, we’re going to see how we can use plugins to create a more complex solution, which can be used to execute multiple prompts in a single call.

You’ll find the sample used in this post in the usual repository. In this case, there are two projects which are relevant for this post:

SemanticKernel.AzureFunction, which is the Azure Function used to expose the API to get the number of the US population given a specific year.
SemanticKernel.OpenAIPlugin, which is the command line application that makes use of this function.

Happy coding!

Semantic Kernel - Native plugins

Tue, 07 Nov 2023 00:00:00 +0000

Updated on 17th November 2023 to reflect the latest changes in the Semantic Kernel APIs:

All the Azure OpenAI extension methods have been renamed from Azure to AzureOpenAI. For example, WithAzureChatCompletionService() is now WithAzureOpenAIChatCompletionService().

In the previous post, we learned how to create a semantic function plugin for Semantic Kernel. A semantic function is just a prompt, however, by including it in a function inside a plugin, we made it easier to reuse and test it.

In this post, we’re going to explore another type of plugins: native functions. As the name suggests, these plugins are native to the platform you’re using, so they can be written in C#. Python or Java. Since they support native code, they can perform more than just executing a prompt; we can virtually execute any operation which is supported by the platform.

Creating a native plugin

For this sample, we’re going to build a plugin which is able to retrieve the information about the United States population in a given year. This might be a peculiar scenario, but there’s a reason why we’re using it: a platform called DataUSA provides a set of free APIs that we can use to retrieve various data about the United States, like the population number, the number of universities, etc. The APIs are free to use and they don’t require any authentication, so they’re perfect for our sample.

These APIs behave like any other REST APIs. As such, this is the type of operation that requires using native APIs: we need to perform a GET request to a given URL, then we need to parse the response and extract the information we need. We can’t perform this operation just with a prompt.

The first step to create the native function is the same as we have seen in the previous post: we create a folder, called Plugins and, inside it, we create a subfolder for our plugin. We’re going to call it UnitedStatesPlugin. Inside this folder, we’re going to add a new class, which will host our function, called UnitedStatesPlugin.cs. This is how the project looks like:

Inside the class, we can define one or more functions, which will be exposed as Semantic Kernel functions. From a code perspective, these functions are just normal C# functions, so we can use any C# feature we want. The only requirement is that we must decorate the function with the [SKFunction] attribute, which is defined in the Microsoft.SemanticKernel namespace. This way, Semantic Kernel will know that the plugin exposes a function, whose name will match the name of the method. Let’s take a look at the following sample:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


public class UnitedStatesPlugin
{
 [SKFunction, Description("Get the United States population for a specific year")]
 public async Task<string> GetPopulation([Description("The year")]int year)
 {
 string request = "https://datausa.io/api/data?drilldowns=Nation&measures=Population";
 HttpClient client = new HttpClient();
 var result = await client.GetFromJsonAsync<UnitedStatesResult>(request);
 var populationData = result.data.FirstOrDefault(x => x.Year == year.ToString());
 string response = $"The population number in the United States in {year} was {populationData.Population}";
 return response;
 }
}

As you can see, the function is just a normal C# method, which returns a string (using a Task, since the method is asynchronous). The only difference is that it’s decorated with the SKFunction attribute. This attribute accepts a parameter, which is the description of the function. We have also added a [Description] attribute also to the year parameter. These two attributes replace the information that, in a semantic function, we were storing in the config.json file. We’ll come back to the importance of providing this information when we’re going to introduce the planner.

The rest of the code is easy to understand: we use the HttpClient class to perform a GET request to the DataUSA API. By using the GetFromJsonAsync<T>() method, we can deserialize the response into a C# class, which maps the content of the JSON response (you can see it here. Finally, we filter the resulting collection to extract only the data for the year we’re interested in and we return the result.

Testing the plugin

Now that we have created the plugin, we can test it. To do so, we need to add it to the Semantic Kernel configuration. First, we create a new instance of the kernel as usual:

1
2
3
4
5
6
7
8
9


string apiKey = "my-api-key";
string deploymentName = "deployment-name";
string endpoint = "endpoint-url";

var kernelBuilder = new KernelBuilder();
kernelBuilder.
 WithAzureOpenAIChatCompletionService(deploymentName, endpoint, apiKey);

var kernel = kernelBuilder.Build();

Then, we can add the plugin to the kernel, this time using the ImportFunctions() method:

1

kernel.ImportFunctions(new UnitedStatesPlugin(), "UnitedStatesPlugin");

As parameters, we supply first a new instance of the class we have previously created, then we provide a name for the plugin.

Finally, we can retrieve the function defined in the plugin in the same way we did for the semantic function plugin:

1

var function = kernel.Functions.GetFunction("UnitedStatesPlugin", "GetPopulation");

The first parameter is the name of the plugin, the second one is the name of the method declared in the UnitedStatesPlugin class.

Finally, we execute the plugin in the usual way: we create a ContextVariables collection with the input, then we call RunAsync() on the kernel passing the collection and the function as parameter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


ContextVariables variables = new ContextVariables
{
 { "input", "2015" }
};

var result = await kernel.RunAsync(
 variables,
 function
);

Console.WriteLine(result.GetValue<string>());
Console.ReadLine();

In this case, we are supplying as input 2015, so we’re expecting to get back the population number of United States in 2015. If we did everything correct, this is exactly the information we’re going to get back:

1

The population number in the United States in 2015 was 316515021

Wrapping up

In this post, we have seen another type of plugins: native ones. Compared to the semantic functions, they are slightly more complex to define, since we need to create a real class with real code. On the other side, however, they are more powerful, since they can perform any operation we want, not just executing a prompt.

In the next post, we’re going to explore the last type of plugins: OpenAI plugins. Stay tuned!

In the meantime, you can find the full source code of the sample on GitHub, in the SemanticKernel.NativeFunction project.

Semantic Kernel - Semantic functions plugins

Fri, 03 Nov 2023 00:00:00 +0000

Updated on 17th November 2023 to reflect the latest changes in the Semantic Kernel APIs:

All the Azure OpenAI extension methods have been renamed from Azure to AzureOpenAI. For example, WithAzureChatCompletionService() is now WithAzureOpenAIChatCompletionService().

Let’s continue our journey to discover how we can infuse AI in our applications with Semantic Kernel, an open-source SDK created by Microsoft for C#, Python, and Java developers. In the previous post, we saw some of the first useful features offered by Semantic Kernel: the ability to easily setup multiple AI providers and prompt templates. In this post, we will explore the first type of plugins: semantic functions.

What are semantic functions?

In the previous post, we have already seen what is a semantic function in the context of Semantic Kernel: it’s basically just a prompt, that is sent to the LLM for processing. The sample we have seen the last time, however, wasn’t very scalable in the context of building an enterprise application. The prompt was hard coded and, as such, it’s hard to share it with other components of our application.

In this post, we’re going to improve the process by moving the semantic function into a plugin, so that it can easily be reused across multiple apps and scenarios. Additionally, this feature enables us to fine tune and change the prompt without having to recompile the application.

Creating a semantic function plugin

Let’s start from the same project we used the last time and let’s create a new folder, called Plugins. Inside this folder, we’re going to create a new sub-folder dedicated to our plugin. We’re going to call it MailPlugin, since it’s going to include our prompt to take a sentence and turn it into a business mail. A plugin can offer multiple functions, each of them represented by a different prompt. Let’s create a function to host our business mail problem, so let’s add another subfolder called WriteBusinessMail.

Inside the WriteBusinessMail folder, we can now add the files we need to define our function. The first one is the file which contains the prompt itself, which is simply a text file called skprompt.txt. Inside it, we just need it copy and paste the same prompt we included in code in the previous post:

1
2
3
4


Rewrite the text between triple backticks into a business mail. Use a professional tone, be clar and concise.
Sign the mail as AI Assistant.

Text: ```{{$input}}```

The second file you must add is called config.json and it’s used to setup the same LLM parameters that, in the previous post, we were setting using the OpenAIRequestSetting class. The content of the file is the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


{
 "schema": 1,
 "type": "completion",
 "description": "Write a business mail",
 "completion": {
 "max_tokens": 500,
 "temperature": 0.0,
 "top_p": 0.0,
 "presence_penalty": 0.0,
 "frequency_penalty": 0.0
 },
 "input": {
 "parameters": [
 {
 "name": "input",
 "description": "The text to convert into a business mail.",
 "defaultValue": ""
 }
 ]
 }
}

Notice how we aren’t using this file only to setup the LLM parameters, but also to describe what the prompt does and how it works. Using the description property, we’re specifying that this prompt is used to write a business mail. In the input section, we’re specifying that the prompt accepts one parameter called input, which contains the text to convert. The importance of these properties will become clear in the future, when we will talk about the planner and its ability to automatically orchestrate AI operations.

This is how the structure of a plugin that contains a semantic function looks like:

Using the semantic function plugin

Now that we’ve created a plugin with a semantic function, we can start using it. First we have to bootstrap the kernel like we did the last time:

1
2
3
4
5
6


string apiKey = "your-api-key";
string deploymentName = "your-deployment-name";
string endpoint = "your-endpoint";

var kernelBuilder = new KernelBuilder().
 WithAzureOpenAIChatCompletionService(deploymentName, endpoint, apiKey);

Then we need to import the plugin into the kernel:

1
2
3


var pluginsDirectory = Path.Combine(Directory.GetCurrentDirectory(), "Plugins");

kernel.ImportSemanticFunctionsFromDirectory(pluginsDirectory, "MailPlugin");

First, we retrieve the full path of the Plugins folder within the project, using the Path.Combine() and the Directory.GetCurrentDirectory() methods. Then, we import all the functions included in the plugin (in this case, it’s just one) in the kernel by calling the ImportSemanticFunctionsFromDirectory() method, passing as parameters the path of the plugins folder and the name of the plugin.

Now we can reference any function included in the plugin using the Functions collection. The following code shows how we can get a reference to the WriteBusinessMail function included in the plugin:

1

var function = kernel.Functions.GetFunction("MailPlugin", "WriteBusinessMail");

Now we can use the function in the same way we have previously used the hardcoded prompt:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


ContextVariables variables = new ContextVariables
{
 { "input", "Tell do David that I'm going to finish the business plan by the end of the week." }
};

var result = await kernel.RunAsync(
 variables,
 function
);

Console.WriteLine(result.GetValue<string>());
Console.ReadLine();

We use the ContextVariables collection to define the value of the input variable and we pass it to the RunAsync() method, together with the function we want to execute. The result is the same as the one we got the last time:

1
2
3
4
5
6
7
8
9


Dear David,

I am writing to inform you that the business plan will be completed by the end of this week.

Thank you for your patience and understanding. If you have any further questions or concerns, please do not hesitate to contact me.

Best regards,

AI Assistant

Testing the semantic functions in a prompt

Testing the effectiveness of a prompt in an application isn’t always easy. Our console app is very simple, so it’s really easy to tweak the prompt and launch the application again. Enterprise applications aren’t that simple and they might be composed by multiple layers, so recompiling and relaunching everything might be a very time consuming task.

To help you testing your prompts, Microsoft has released a very useful extension for Visual Studio Code. You can get if from the marketplace.

Once it’s installed and activated, you will find a new icon in the toolbar:

After you click on it, the first step is to expand the AI Endpoints section and click on the Switch endpoint provider button to select the provider you want to use between OpenAI, Azure OpenAI and Hugging Face. Based on the provider you select, you will be asked to provide the required credentials, similar to what you did when you set the kernel up in code.

Once you have configured the AI service you want to use, click on File in Visual Studio Code and choose Open folder. Now open the folder which contains your Semantic Kernel based project. Once you do that, the Functions panel of the Semantic Kernel extension will display all the functions that have been found in the project. You should see your MailPlugin, with the WriteBusinessMail function nested inside:

If you hover the mouse near the name of the function, you will see a Run button appear. Click on it and Visual Studo Code will ask you to provide a value for the variables included in the prompt, in this case the input variable:

Once the operation is completed, in the Output window in Visual Studio Code you will see the result of the operation: the duration, the number of tokens used, the response from the LLM. Even better, if you open again the Semantic Kernel panel, you will see the result of the operation in the Results section:

If you click on the results, you will see the full details of the response in a table format:

Wrapping up

In this post, we learned how to build our first plugin that includes a semantic function. This is a major improvement compared to having the prompt hard coded, since it allows us to reuse it across different apps. Additionally, if we want to make some changes to the prompt, we can do it without having to recompile the application.

As usual, you will find the sample used in this post in the GitHub repository. The sample used in this post is the project called SemanticKernel.SemanticFunction.

Semantic Kernel - The basics

Mon, 30 Oct 2023 00:00:00 +0000

Updated on 17th November 2023 to reflect the latest changes in the Semantic Kernel APIs:

All the Azure OpenAI extension methods have been renamed from Azure to AzureOpenAI. For example, WithAzureChatCompletionService() is now WithAzureOpenAIChatCompletionService().

The advent of Generative AI is changing the tech industry and more and more applications are adding AI powered features. If you just think to what Microsoft is doing these days in this space, they are integrating Copilot experiences powered by AI and LLMs almost in every product: Microsoft 365, Windows, Defender, Power Platform. From a technical perspective, integrating generative AI services from the most popular AI providers (like OpenAI) is fairly simple. They all offer a set of REST APIs to cover the most common scenarios, like generating a response starting from a prompt, supporting a chat conversation or creating an image out of a prompt.

However, when you start to build more complex enterprise applications, like a customized Copilot experience, you hit multiple challenges in using these APIs directly, like:

The need of changing the communication layer in case you need to use different services and models, which are based on a different set of APIs.
Managing prompts and requests in a scalable way.
Managing complex operations that might requires multiple prompts to be completed.
Integrating vector search to support Retrieval Augmented Generation (RAG), so that you can use the LLM to perform operations on private data, like the organizational data.

Of course, it’s possible to build all these features just by calling the REST APIs, but it requires lot of work. To support developers in this journey, the tech market saw the rise of AI orchestration libraries and SDKs, that abstract many of the AI concepts into a set of easy-to-use APIs. You might already have heard of LangChain, which is one of the most popular available libraries for Python developers.

Microsoft is investing a lot in the AI ecosystem, as such they decided to join the AI orchestration space, with the dual goal of supporting developers who are using Microsoft tools and platforms (like .NET and Azure OpenAI) and having a platform that the company can use to infuse AI in their app ecosystem in an easier way. Please welcome Semantic Kernel, an open-source AI orchestration library which can be used by C#, Python and Java developers.

Semantic Kernel supports the following features:

Built-in support for multiple AI providers, like OpenAI, Azure OpenAI and Hugging Face.
Support for plugins (built-in and build your own ones)
Built-in support for many vector databases to store history and context
Automatic orchestration with planner

This post is the first one in a series, in which I’m going to cover all the most interesting features of Semantic Kernel, starting form the simplest to the most advanced ones. You will find all the samples that will be part of this series in the following GitHub repository.

Let’s start the journey!

Building your first Semantic Kernel app

For this series of posts, I will use a .NET C# terminal application. However, the same tasks can be performed in Python and Java. As such, I’m going to open Visual Studio 2022, choose Create new project and select Console App. After creating the project, your first step should be to install Semantic Kernel. Right click on the project, choose Manage NuGet packages and look for a package called Microsoft.SemanticKernel. At the time of writing, the package is still published as prerelease, so you will have to make sure the option Include prerelease is turned on to find it:

Now we are ready to set up the kernel object, which is the one that we’re going to use across our application to orchestrate AI operations. To create a kernel, we use the KernelBuilder class, which offers a series of extension methods to initialize the object based on the AI service and the model we’re going to use. The C# version currently supports OpenAI and Azure OpenAI, with the following extension methods:

WithChatCompletionService(), to use chat models like gpt3.5-turbo.
WithImageGenerationService(), to use AI models to generate images, like DALL-E, rather than text.
WithTextCompletionService(), to use text completion models like davinci.
WithTextEmbeddingGenerationService(), to use models like text-embedding-ada-002 to convert text into embeddings, to be used with vector databases.

Each method has two different versions, based on the service. For example, if you want to use a chat model, you can use WithAzureChatCompletionService() with Azure OpenAI or WithOpenAIChatCompletionService() with OpenAI. Depending on the service you want to use, you will have to provide two different set of credentials.

Using OpenAI

To use OpenAI, you must provide:

The API key
The model name

Both information can be retrieved from your OpenAI account:

Once you have them, you can pass them as parameters to the initialization method. For example, the following code shows how to initialize the kernel to use the chat completion service from OpenAI:

1
2
3
4
5


string apiKey = "my-api-key";
string model = "gpt3.5-turbo";

var kernelBuilder = new KernelBuilder().
 WithOpenAIChatCompletionService(model, apiKey);

Please note that this is just a sample to keep the code simple. In a real application, the API key would be retrieved from a secure service, like Azure Key Vault or a user secret.

Using Azure OpenAI

To use Azure OpenAI, you must provide:

The URL endpoint
The deployment name
The API key

This information can be retrieved from the Azure portal, when you access to the Azure OpenAI service you have deployed:

Once you have them, you can use them with one of the various WithAzureOpenAI...() extension methods provided by the KernelBuilder class. For example, the following code shows how to initialize the kernel to use the chat completion service from Azure OpenAI:

1
2
3
4
5
6


string apiKey = "your-api-key";
string deploymentName = "your-deployment-name";
string endpoint = "your-endpoint";

var kernelBuilder = new KernelBuilder().
 WithAzureOpenAIChatCompletionService(deploymentName, endpoint, apiKey);

Regardless of the service you’re using, you can get a kernel object by calling the Build() method of the KernelBuilder class:

1

var kernel = kernelBuilder.Build();

Now we’re ready to start orchestrating your AI workflows.

A simple scenario: sending a prompt

We’ll start from the simplest scenario you can achieve with AI services: submitting a prompt and getting a response back. For the moment, we’re going to hard code the prompt inside the code, as in the following example:

1
2
3
4
5
6


string prompt = """
Rewrite the text between triple backticks into a business mail. Use a professional tone, be clar and concise.
Sign the mail as AI Assistant.

Text: ```{{$input}}```
""";

The purpose of the prompt is very simple: turning a sentence into a business mail, written in a professional tone. We can already see one interesting feature provided by Semantic Kernel: prompt templates. Instead of hard coding the sentence we want to turn into an e-mail, we define a placeholder using the keyword $input. This way, we can reuse the same prompt easily, we just need to change the provided input.

Now we can use the prompt to generate a semantic function, which is the way Semantic Kernel calls functions which are represented by a prompt:

1

var mailFunction = kernel.CreateSemanticFunction(prompt);

If we want, we have the option to customize the OpenAI parameters by supplying an OpenAIRequestSetting object as second parameter. For example, in the following sample we’re creating the same function, but changing the temperature (which controls the randomness of the response; the higher the value, the more random the result will be) and the maximum number of tokens to generate for the response.

1
2
3
4
5


var mailFunction = kernel.CreateSemanticFunction(prompt, new OpenAIRequestSettings
{
 Temperature = 0.5,
 MaxTokens = 1000
});

Now we need to supply the input variables for the prompt. In this case we have just one, called input. We supply them through the ContextVariables dictionary, which is a key-value pair collection:

1
2
3
4


ContextVariables variables = new ContextVariables
{
 { "input", "Tell David that I'm going to finish the business plan by the end of the week." }
};

For each entry, we supply as key the placeholder we have included in the prompt, while as value the text we want to provide.

Finally, we execute the function using the RunAsync() method offered by the kernel, passing as input the variables and the semantic function we want to execute:

1
2
3
4
5


var output = await kernel.RunAsync(
 variables,
 mailFunction);

Console.WriteLine(output.GetValue<string>());

From the result, we can use the GetValue<T>() method to extract the response, where T is the data type we’re expecting back. In this case, we’re using a chat model, so we know that the output will be a string.

The response will be similar to the following one:

1
2
3
4
5
6
7
8
9


Dear David,

I am writing to inform you that the business plan will be completed by the end of this week.

Thank you for your patience and understanding. If you have any further questions or concerns, please do not hesitate to contact me.

Best regards,

AI Assistant

Wrapping up

In this first post, we have learned the basics of how to add Semantic Kernel to our applications and to perform a basic operation: submitting a prompt to the LLM and get a response back. We’re still far from having explored the most powerful features of Semantic Kernel, but we have already seen a few advantages compared to using directly the REST APIs provided by OpenAI or Azure OpenAI:

We don’t need to learn different ways to interact with the API, based on the service we need to use.
If we need to swap the service (or use different services based on the task), we don’t need to rewrite the entire communication layer. We just need to setup the kernel with a different extension method, and we’ll continue to use it in the same way.
With prompt templates, we can easily reuse the same prompt with multiple inputs.

In the next post, we’re going to introduce one of the most interesting features of Semantic Kernel: plugins.

Meantime, remember that you can find this sample in the dedicated GitHub repository.

About me

Mon, 01 Jan 0001 00:00:00 +0000

Matteo Pagani is a Cloud Solution Architect in the AI, Apps & Ecosystem team in the Modern Work organization in Microsoft. In his role, he supports developers and companies all around the world to get the best from the Microsoft 365 ecosystem, by facilitating the adoption of Microsoft 365 Copilot and building applications and extensions for Microsoft Teams, Microsoft Viva, Power Platform and Microsoft 365 Copilot. He has a strong passion for development, which he loves to share with other developers by writing articles, blog posts and books and by speaking at conferences all around the world. Before joining Microsoft, he was a Microsoft MVP in the Windows Development category and a Nokia Developer Champion for almost 5 years.

Search

Mon, 01 Jan 0001 00:00:00 +0000

The Developer's Cantina

Consuming an authenticated MCP server with a custom engine agent in Copilot Studio

Prerequisites

Creating the custom engine agent

Adding the MCP server

Configuring OAuth authentication

Adding the redirect URL to your app registration

Creating the connection

Verifying the tools

Testing the agent

What happens behind the scenes?

Additional resources

Wrapping up

Consuming an authenticated MCP server with a declarative agent in Copilot Chat

Prerequisites

Creating the declarative agent project

Connecting to the MCP server

Fetching the MCP tools

Issue #1: The GET endpoint requirement

Importing the tools

Provisioning the agent

Issue #2: Missing OAuth properties in m365agents.yml

Issue #3: Tool parameter type incompatibilities

Testing the agent

What happened behind the scenes?

Additional resources

Wrapping up

Enabling multi-tenant authentication for MCP servers with Microsoft Entra and APIM

Understanding the challenge

Updating the app registrations

Updating the Flights MCP Server API app registration

Updating the Flights MCP Client app registration

Updating the APIM token validation policy

Using the common tenant ID

Adding client application IDs

Updating the OAuth metadata endpoint

Testing the multi-tenant setup

Testing with the original tenant

Testing with a different tenant

Troubleshooting tips

Token validation errors

Security considerations

Wrapping up

Building an authenticated MCP server with Microsoft Entra and .NET

Setting up the MCP server project

FlightService.cs

FlightsTool.cs

Program.cs

Adding OpenAPI support

Publish the MCP server to Azure App Service

Securing the MCP server with Microsoft Entra

Expose an API

Protecting the MCP server with Azure API Management

Validating the Entra tokens

Supporting the oAuth 2.0 authorization code flow

Securing the App Service

Testing the job

Setting up Visual Studio Code

Wrapping up

Using Model Context Protocol in agents - Copilot Studio and authentication with API key

Creating our MCP server

Exposing the APIs

Managing authentication with an API key

Implementing the MCP tool

Running the MCP server

Setting up the custom connector in Copilot Studio

Using the connector in an agent

Wrapping up

Using Model Context Protocol in agents - Streamable HTTP support in Copilot Studio

Goodbye SSE, welcome Streamable HTTP

Using Streamable HTTP in C#

Using Streamable HTTP in Copilot Studio

Using the new MCP connector in Copilot Studio

Wrapping up

Using Model Context Protocol in agents - Pro-code agents with Semantic Kernel

Connecting to an MCP server with C#

Using MCP in a Semantic Kernel agent

Wrapping up

Using Model Context Protocol in agents - Copilot Studio

Implementing the SSE transport protocol

Using the `common` tenant ID