Lift-off with Microsoft Copilot

Our Microsoft Copilot 365 trial

Since our inception in 2016, Cielo Costa’s has provided Microsoft solutions and consulting services. So, it’s natural that we have a deep curiosity about emerging Microsoft services. Indeed, we were at Microsoft’s Envision event in London Docklands in 2023 where it unveiled its latest AI capabilities.

Whilst the General Availability date of November 2023 came and went, we have now been able to purchase the licences through our admin portal and trial Copilot for Microsoft 365…

Use cases and anticipated benefits

Being an organisation that comprises a 100% remote workforce, we favour Teams for our collaboration. We had trialled Teams Premium before and had been eagerly waiting to see how Copilot stacked-up against this for summarising meetings in terms of key topics, decisions and action items.

With a large part of formal business communication still occurring via email, we were also keen to test Copilot in Outlook in terms of its ability to summarise email threads and draft email responses.

Like many businesses, we use the standard office apps including Word, Excel and PowerPoint. We were already using the free generative AI capabilities of Bing Chat to create prompts, ideas and draft text for use in everything from internal policy documents, job descriptions for hiring, marketing copy and text for inclusion in proposals. However, with the paid-for licences we would now be able to feed Copilot with files to summarise and/or convert into other types of output.

Approach

With a not insignificant price tag of nearly £300+VAT per user to be paid up-front for a commitment of one year, we decided not to roll Copilot for Microsoft 365 to everyone in Cielo Costa without first testing its potential value to our business.

We took a cross-section of users, including:

• • Directors/Heads of

• • Project Managers

• • Business Analysts

• • Developers

• • Infrastructure Consultants

We wanted to see which personas received the most value. If that was everyone, that’s fine! If some people in the trial don’t benefit, that is OK too, as the licences are transferable.

Regarding the process, we set up a Teams channel to discuss, share tips, and ask questions on how to use the technology. We met at the end of the first week to a) promote the use of the licences, and b) garner early feedback. We met again after approximately one month to capture feedback more fully.

Key findings

Below are the key findings after just a couple of weeks of piloting Microsoft Copilot in Cielo Costa:

• • There are similar/related Microsoft capabilities available at a lower cost

• • Copilot is an emerging technology

• • Not everyone on your call/meeting will benefit

• • Don’t underestimate the people impact!

• • Copilot may surface issues where underlying weak access controls exist

• • You shouldn’t take everything you read as gospel!

There are similar/related Microsoft capabilities available at a lower cost

The upfront costs are significant. Therefore, piloting the capability with different types of users within the business makes a lot of sense to understand its potential value.

We need to do further analysis of the return on the investment. Initial thoughts have been positive, however, there are cheaper or free alternatives for some of the capabilities, e.g.:

• • Teams meeting summaries and actions: One of the most valuable features we have found with Microsoft Copilot is the ability to summarise Teams meetings and list the action items. Teams Premium has similar functionality, is significantly cheaper at £5.80/user/month, and does not require an annual commitment. The critical difference is that Copilot is an anytime and interactive experience (you can ask anything you like during or after a meeting), whereas Teams Premium only provides AI-generated notes after the meeting.

• • Generative AI: Copilot for Microsoft 365 uses the GPT4 model, which is also available free (albeit somewhat limited) in the Edge browser and via Bing, the Microsoft search engine.

• • Summarising files: We found this capability promising; however, we did not have the opportunity to utilise it extensively during the pilot.

• • Other bells and whistles: Email summaries and asking Copilot to “catch me up” are useful; however, most of our pilot users said they could live without them.

Copilot is an emerging technology

There are a few little niggly things which we think are likely to improve as the Copilot experience matures. For example, the generative AI capabilities will default to American English. Even having asked Copilot to always respond in British English seemed to have no effect. It’s not a deal-breaker, but it’s a little annoying all the same.

You might ask Copilot for information you know you have access to. In my example, I was meeting a client the next day and offered to collect my colleague (let’s call them “Eric”) from their house as it was on the way. I’d done this before, and I knew they’d furnished me with their address. Great, I thought I would ask Copilot what their address is.

Grrrr, that’s annoying. I’ll look it up myself then!

Not everyone on your call/meeting will benefit

The disparities between Copilot and Teams Premium users caught us out once or twice. You might think, “Copilot can create meeting notes and actions, and Teams Premium generates the same”. Well, yes, they do. However, Teams Premium users need a meeting to be recorded (as well as transcribed), whereas clicking the Copilot button during a Teams meeting starts only transcription:

For the “AI-powered meeting notes” to be available for all meeting attendees, including those without a Copilot licence (but with a Teams Premium licence), a video recording of the meeting is also required.

Furthermore, if the person you’re meeting with is from a different organisation and they scheduled the meeting, you will not be able to use Copilot features.

Copilot can’t access external transcripts, even if you are set up in a multi-tenant organisation (MTO) with the other party. We understand the need for data security, but this does lead to frustration when you are paying a lot of money for a licence for something you can only use for half your meetings (unless you set them all up)!  This is an issue compounded in a business like ours where we work with many different organisations.

Don’t underestimate the people impact!

During our trial, staff commented that they felt like they had been given the tools they needed for their job and were more valued. Conversely, staff not provided with a licence felt left out of the hype – at least in the early days.

Copilot may surface issues where underlying weak access controls exist

Microsoft Copilot has access to your organisation’s tenant, but it presents only data that you can access already.  It uses the same underlying access controls for data in other Microsoft 365 services. I.e. Copilot doesn’t expose information to users that they do not already have access to.

Because Copilot has access to all the data you can access within Microsoft 365, including sensitive, private, or confidential information, this can lead to unexpected consequences. Unfortunately, employees often have more permissions than they strictly need. Copilot might search and compile data from various sources (documents, presentations, emails, calendars, etc) and surface information to a user that they weren’t aware they had access.

The mitigating actions to this are just reasonable security hygiene measures:

• • Regular reviews of user access.

• • Purging outdated content.

It would seem that Copilot does have some privacy rules built in, though it is difficult to understand what exactly it considers privileged information. For example, you may have access to salary information at your organisation. However, Copilot responds with stock answers if you ask a question about it. The same is true for other personal information like addresses etc.

You shouldn’t take everything you read as gospel!

In Microsoft’s own words:

Teams Generative AI features strive to provide accurate and informative responses, based on the data available. However, answers may not always be accurate as they are generated based on patterns and probabilities in language data. Use your own judgment and double check the facts before making decisions or taking action based on the responses.

You need to remain mindful that whilst meeting recaps are helpful, you are hearing about the meeting content second-hand. The AI algorithm may but the wrong emphasis on topics or decisions that were discussed. Language is remarkably complex, and it does a great job most of the time. However, the AI focuses on what was said, and not how it was said or other non-verbal cues you would pick up on in a meeting.

Worse, you may rely on information that is misleading or just outright untrue.

I asked Copilot in Word to:

And Copilot very kindly added the following:

The thing is, I know that Liverpool didn’t play Arsenal last night! And a quick search suggests that the Manchester United vs Brighton result was from April 2021!