{"id":10388,"date":"2023-09-04T22:25:45","date_gmt":"2023-09-04T22:25:45","guid":{"rendered":"https:\/\/andrewdoran.uk\/blog\/?p=10388"},"modified":"2023-09-04T22:25:50","modified_gmt":"2023-09-04T22:25:50","slug":"when-good-is-the-enemy-of-the-perfect","status":"publish","type":"post","link":"https:\/\/andrewdoran.uk\/blog\/2023\/09\/04\/when-good-is-the-enemy-of-the-perfect\/","title":{"rendered":"When good is the enemy of the perfect"},"content":{"rendered":"
I\u2019ve been working with Microsoft Teams Premium<\/a> for a couple of weeks and it has got me worried. Part of the Teams Premium bundle is the ability to summarise a meeting recording, giving you an AI-generated set of collapsible bullet points of the key things that were discussed. It\u2019s neat, clever, and…a little underwhelming, particularly when compared to the promise of the Microsoft marketing videos<\/a>.<\/p>\n

From the meetings that I have recorded, the \u2018AI notes\u2019 seem to lose a certain essence of some parts of the discussion, and the suggested follow-up tasks often don\u2019t make sense. Aside from the often hilarious Teams chat response suggestions that have been in the product for a while now1<\/a><\/sup>, Teams Premium is my first encounter with the new large language model-based AI products from Microsoft. What concerns me is not how poor the product is today, but how close to perfect it is going to get over time.<\/p>\n

I think the most worrisome aspect of AI systems in the short term is that we will give them too much autonomy without being fully aware of their limitations and vulnerabilities.
\n(Melanie Mitchell, Artificial Intelligence: A Guide for Thinking Humans<\/em>)<\/p><\/blockquote>\n

I see the output of these AI large language models on a spectrum. At one end, the tools may spit out complete and utter garbage, perhaps not even words. Their uselessness would be obvious to everyone who uses them. At the other end, the AI could output a perfect response (or summary of a meeting, in the case of Teams Premium) every single time. The problem lies in the middle, and gets worse the closer the system is to being consistently perfect:<\/p>\n

<\/p>\n

Right now, I think that the quality of the Teams Premium \u2018AI notes\u2019 feature sits somewhere in the green area. It\u2019s good and useful a lot of the time. For example, I can scan the notes and check whether a topic was mentioned. If that topic isn\u2019t in the notes, it doesn\u2019t mean that it wasn\u2019t discussed; I\u2019d have to watch the video back to check that the AI didn\u2019t miss it. If the meeting was very<\/em> important and needed to be formally minuted, I would still rely on the video.<\/p>\n

As the product improves over time, we\u2019ll move out of the green zone and into the yellow. At this point, I may consciously or subconsciously decide to stop routinely verifying the AI-generated output. It\u2019s good enough, most of the time. Again, if a meeting is really<\/em> important, I may watch the video.<\/p>\n

The real danger comes in the red zone. Here, the AI output is superb<\/em> most of the time, so much so that I never<\/em> check it. I rely on the summary even for my important meeting minutes. But it\u2019s not quite<\/em> at the \u2018completely perfect\u2019 end of the spectrum. Occasionally it will trip up. Something<\/em> will get missed \u2014 maybe one meeting in a hundred \u2014 and perhaps that something is critical to the conversation we\u2019ve had. Perhaps it will attribute a comment to the wrong person, or miss the nuance of a discussion which was important to get exactly right. We may only find out that the AI produced flawed output for this meeting when an incident arises down the line.<\/p>\n

This isn\u2019t a concern about AI getting \u2018too good\u2019 and becoming \u2018sentient\u2019 in a general sense.2<\/a><\/sup> It\u2019s more that we have decided to stop thinking, that we have handed control of some part of our workflow over to the AI and no longer verify its output. For me personally, one bad output every 100 recorded meetings might be tolerable. But if we scale this across a large organisation where hundreds or thousands of meetings take place every day, we\u2019re going to have problems.<\/p>\n

Baldur Bjarnason explores this in his book The Intelligence Illusion<\/a><\/em>:<\/p>\n

I mentioned two of [the flaws] before, automation<\/em> and anchoring<\/em> biases. We, as human beings, have a strong tendency to trust machines over our own judgement. This kills people, as it\u2019s been a major problem in aviation. Anchoring bias comes from our tendency to let the initial perceptions, thoughts, and ideas set the context for everything that follows. AI adds a third issue: anthropomorphism<\/em>. Even the smartest people you know will fall for this effect as large language models are incredibly<\/em> convincing. These biases combined lead people to feel even more<\/em> confident in the AI\u2019s work and believe that it\u2019s done a better job than it has.<\/p>\n

We\u2019re using the AI tools for cognitive assistance<\/em>. This means that we are specifically using them to think less. In every other industry this dynamic inevitably triggers our automation bias<\/em> and compromises our judgement of the work done by the tools. We use the assistant to think less, so we do.<\/p>\n

These models are incredibly fluent<\/em> and\u2014as we saw at the start of this book\u2014are consistently presented by their vendors as near-AGI. This triggers our instinct towards anthropomorphism<\/em>, making us feel like we have a fully human-level intelligence assisting us, creating an intelligence illusion<\/em> that again hinders are ability to properly assess the work it\u2019s doing for us.<\/p><\/blockquote>\n

AI-generated meeting summaries in Teams Premium is a useful starting point for thinking about this technology. There\u2019s no user input beyond hitting the \u2018record\u2019 button during a meeting, and everyone with a Teams Premium licence gets access to exactly the same summary. The possibility for getting something wrong is limited to how good or bad the summary of the meeting is. So far, so harmless. But Microsoft 365 Copilot will be arriving soon, vastly expanding the problem space with its interactive, prompt driven approach<\/a>. Where on the \u2018useless to perfect\u2019 spectrum will it land? What if just being \u2018very good\u2019 isn\u2019t good enough?<\/p>\n

\n
\n
    \n
  1. \n
  2. The more I learn, the less I\u2019m worried about General AI being a problem any time soon.\u00a0\u21a9<\/a><\/li>\n<\/ol>\n<\/div>\n<\/div>\n

    I\u2019ve been working with Microsoft Teams Premium for a couple of weeks and it has got me worried. Part of the Teams Premium bundle is the ability to summarise a meeting recording, giving you an AI-generated set of collapsible bullet points of the key things that were discussed. It\u2019s neat, clever, and…a little underwhelming, particularly […]<\/div>\n<\/a>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"webmentions_disabled_pings":false,"webmentions_disabled":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"activitypub_content_warning":"","activitypub_content_visibility":"","activitypub_max_image_attachments":3,"activitypub_interaction_policy_quote":"anyone","activitypub_status":"","footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[16,20],"tags":[],"class_list":{"0":"post-10388","1":"post","2":"type-post","3":"status-publish","4":"format-standard","6":"category-technology","7":"category-work","8":"h-entry","10":"hentry"},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":12001,"url":"https:\/\/andrewdoran.uk\/blog\/2025\/05\/06\/weeknotes-323-the-lilac-time\/","url_meta":{"origin":10388,"position":0},"title":"Weeknotes #323 \u2014 The lilac time","author":"Andrew Doran","date":"6 May 2025","format":false,"excerpt":"Every year we get about two weeks to enjoy our lilac, which looks fabulous and smells even better. My third four-day working week in a row, with one more to go. Work was busy, keeping my mind off of the big bike ride planned for the weekend. Many of my\u2026","rel":"","context":"In "Weeknotes"","block_context":{"text":"Weeknotes","link":"https:\/\/andrewdoran.uk\/blog\/category\/weeknotes\/"},"img":{"alt_text":"A very large blossoming pink and purple lilac tree.","src":"https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2025\/05\/084C7E14-FA94-4FAF-B54C-6BF68B79E19D_1_102_a.jpeg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2025\/05\/084C7E14-FA94-4FAF-B54C-6BF68B79E19D_1_102_a.jpeg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2025\/05\/084C7E14-FA94-4FAF-B54C-6BF68B79E19D_1_102_a.jpeg?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2025\/05\/084C7E14-FA94-4FAF-B54C-6BF68B79E19D_1_102_a.jpeg?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2025\/05\/084C7E14-FA94-4FAF-B54C-6BF68B79E19D_1_102_a.jpeg?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2025\/05\/084C7E14-FA94-4FAF-B54C-6BF68B79E19D_1_102_a.jpeg?resize=1400%2C800&ssl=1 4x"},"classes":[]},{"id":10363,"url":"https:\/\/andrewdoran.uk\/blog\/2023\/08\/28\/weeknotes-235-happy-place\/","url_meta":{"origin":10388,"position":1},"title":"Weeknotes #235 \u2014 Happy place","author":"Andrew Doran","date":"28 August 2023","format":false,"excerpt":"Back to work. My main thought when I landed at my desk on Monday morning was how lucky I am that being in front of my keyboard is my happy place. Well, at least one of my happy places, anyway. It was good to be back. It was one of\u2026","rel":"","context":"In "Weeknotes"","block_context":{"text":"Weeknotes","link":"https:\/\/andrewdoran.uk\/blog\/category\/weeknotes\/"},"img":{"alt_text":"The early cycling group, grateful for a rest having just climbed up Whiteleaf.","src":"https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2023\/08\/DraggedImage-28-scaled.jpeg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2023\/08\/DraggedImage-28-scaled.jpeg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2023\/08\/DraggedImage-28-scaled.jpeg?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2023\/08\/DraggedImage-28-scaled.jpeg?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2023\/08\/DraggedImage-28-scaled.jpeg?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2023\/08\/DraggedImage-28-scaled.jpeg?resize=1400%2C800&ssl=1 4x"},"classes":[]},{"id":11444,"url":"https:\/\/andrewdoran.uk\/blog\/2024\/12\/17\/weeknotes-303-aerodynamic-pudding\/","url_meta":{"origin":10388,"position":2},"title":"Weeknotes #303 \u2014 Aerodynamic pudding","author":"Andrew Doran","date":"17 December 2024","format":false,"excerpt":"My final week of work of 2024. It\u2019s our company policy that all staff need to take two weeks of consecutive leave each year. I\u2019ve been so busy that I\u2019m only able to do this now. Next week my wife is still working and my kids are still at school,\u2026","rel":"","context":"In "Weeknotes"","block_context":{"text":"Weeknotes","link":"https:\/\/andrewdoran.uk\/blog\/category\/weeknotes\/"},"img":{"alt_text":"A montage of photos of me dressed as a Christmas pudding, posing with my friend Ian and his well-decorated self (and bike) before the ride, some action shots, and me at the end of the ride, spattered in mud.","src":"https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2024\/12\/DraggedImage-4-scaled.jpeg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2024\/12\/DraggedImage-4-scaled.jpeg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2024\/12\/DraggedImage-4-scaled.jpeg?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2024\/12\/DraggedImage-4-scaled.jpeg?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2024\/12\/DraggedImage-4-scaled.jpeg?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2024\/12\/DraggedImage-4-scaled.jpeg?resize=1400%2C800&ssl=1 4x"},"classes":[]},{"id":10661,"url":"https:\/\/andrewdoran.uk\/blog\/2024\/01\/15\/weeknotes-255-meetings-are-not-enough\/","url_meta":{"origin":10388,"position":3},"title":"Weeknotes #255 \u2014 Meetings are not enough","author":"Andrew Doran","date":"15 January 2024","format":false,"excerpt":"Mentally this was an extremely busy week. From a work perspective, there\u2019s a mountain to climb this year. But, I have a map, a compass, provisions, fair weather and an excellent team; it\u2019s going to be tough but I think we can do it. This week felt as though I\u2026","rel":"","context":"In "Weeknotes"","block_context":{"text":"Weeknotes","link":"https:\/\/andrewdoran.uk\/blog\/category\/weeknotes\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2024\/01\/Image-15-01-2024-07-19.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2024\/01\/Image-15-01-2024-07-19.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2024\/01\/Image-15-01-2024-07-19.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2024\/01\/Image-15-01-2024-07-19.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2024\/01\/Image-15-01-2024-07-19.png?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2024\/01\/Image-15-01-2024-07-19.png?resize=1400%2C800&ssl=1 4x"},"classes":[]},{"id":12902,"url":"https:\/\/andrewdoran.uk\/blog\/2025\/12\/08\/weeknotes-354-lighting-up\/","url_meta":{"origin":10388,"position":4},"title":"Weeknotes #354 \u2014 Lighting up","author":"Andrew Doran","date":"8 December 2025","format":false,"excerpt":"Christmas bells outside the Holy Sepulchre Church in the City of London. The past couple of weeks have given us properly grotty weather. Wind and rain with a few days of freezing temperatures thrown in. This was another busy week where I\u2019ve been feeling that the year has caught up\u2026","rel":"","context":"In "Weeknotes"","block_context":{"text":"Weeknotes","link":"https:\/\/andrewdoran.uk\/blog\/category\/weeknotes\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2025\/12\/DraggedImage-scaled.jpeg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2025\/12\/DraggedImage-scaled.jpeg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2025\/12\/DraggedImage-scaled.jpeg?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2025\/12\/DraggedImage-scaled.jpeg?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2025\/12\/DraggedImage-scaled.jpeg?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/andrewdoran.uk\/blog\/wp-content\/uploads\/2025\/12\/DraggedImage-scaled.jpeg?resize=1400%2C800&ssl=1 4x"},"classes":[]},{"id":11646,"url":"https:\/\/andrewdoran.uk\/blog\/2025\/02\/10\/weeknotes-311-lurgy\/","url_meta":{"origin":10388,"position":5},"title":"Weeknotes #311 \u2014 Lurgy","author":"Andrew Doran","date":"10 February 2025","format":false,"excerpt":"A week dominated by illness. Early on Monday morning my wife collected our eldest son at the airport, back from his running trip to Boston University. He was quite ill with a horrible cough, sore throat and a high temperature, and ended up spending the whole week at home. A\u2026","rel":"","context":"In "Weeknotes"","block_context":{"text":"Weeknotes","link":"https:\/\/andrewdoran.uk\/blog\/category\/weeknotes\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/img.youtube.com\/vi\/vmPn-GyI7Wk\/0.jpg?resize=350%2C200","width":350,"height":200},"classes":[]}],"jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/andrewdoran.uk\/blog\/wp-json\/wp\/v2\/posts\/10388","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/andrewdoran.uk\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/andrewdoran.uk\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/andrewdoran.uk\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/andrewdoran.uk\/blog\/wp-json\/wp\/v2\/comments?post=10388"}],"version-history":[{"count":4,"href":"https:\/\/andrewdoran.uk\/blog\/wp-json\/wp\/v2\/posts\/10388\/revisions"}],"predecessor-version":[{"id":10392,"href":"https:\/\/andrewdoran.uk\/blog\/wp-json\/wp\/v2\/posts\/10388\/revisions\/10392"}],"wp:attachment":[{"href":"https:\/\/andrewdoran.uk\/blog\/wp-json\/wp\/v2\/media?parent=10388"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/andrewdoran.uk\/blog\/wp-json\/wp\/v2\/categories?post=10388"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/andrewdoran.uk\/blog\/wp-json\/wp\/v2\/tags?post=10388"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}