# The Battle of the Green Dot

# Why Teams thinks I'm away

Even if you've never used Microsoft Teams yourself, you've probably come across people complaining about it. There are plenty of complaints, but one seems to come up over and over again:

1. You're working as usual. Your Teams status is Available. All good.
2. You look away for ten seconds.
3. The status is Away. Your manager is writing you to ask why you are not working.

Okay, I'm exaggerating. But the underlying problem is real. The fact you are not moving mouse or typing doesn't mean you are away. You could be:

1. reading an article, documentation or something else work-related
2. drawing, sketching or taking notes on paper
3. waiting for an AI agent to finish
4. dictating feedback or instructions to an AI agent

But Microsoft Teams does not know that. Neither do your colleagues. Some teams even interpret Away status as "this person is not working". Not only is that assumption wrong, but it also puts unnecessary pressure on people to appear Available all the time.

You'd be "away" in the office as well. You'd grab a coffee, walk over to a colleague to ask a question, or go to the bathroom. Nobody questions that in the office...

Luckily, we see things differently at [BeZoned](https://bezoned.com?utm_source=malja&utm_source=blog&utm_campaign=presence_detector). We don't mind if you go Away during the day. We all work from home, and we trust each other. But trust doesn't make presence information useless. We still need to know when someone's around, especially when we're trying to have a quick conversation. When Teams switches to the Away status, but I'm still sitting at my desk, I'm sending the wrong signal. Others can still reach me.

And it works the other way around, too. When I have to step away for a moment - to answer the door for a mailman, grab a food delivery because I'm too lazy to cook, or rescue my dog's toy from wherever he's managed to get it stuck - I tend to forget to set my status accordingly.

That's when I thought: what if Teams could detect whether I'm sitting at my computer and automatically update my status?

# Can an old webcam fix it?

Sometimes you have the solution in front of you for years. In my case, it was an old Philips monitor with a built-in webcam. The webcam was terrible even when I bought the monitor, so I never used it for video calls. Maybe it was finally time to put it to use.

Could a little bit of AI (or rather ML) help me turn this unused webcam into a **presence detector**? When I'm sitting at the desk it would keep me Available on Teams. The second I move away, it would switch my status to Away.

## Step 1 - Getting the webcam to work

I could probably jump directly to the code, but I decided to start small. Let's take a picture with the webcam to see if it even works.

First, I needed to list available video devices:

```shell
ffmpeg -f avfoundation -list_devices true -i ""
```

Which outputs:

```plaintext
[AVFoundation indev @ 0x7513014140] AVFoundation video devices:
[AVFoundation indev @ 0x7513014140] [0] USB2.0 FHD UVC WebCam
```

(The output is simplified, so it only contains the WebCam I'm interested in.)

Now let's take a picture:

```shell
ffmpeg -f avfoundation -i "0:none" -frames:v 1 webcam.jpg
```

And I'm greeted with an error:

```plaintext
[avfoundation @ 0x7676c20000] Selected framerate (29.970030) is not supported by the device.
[avfoundation @ 0x7676c20000] Supported modes:
[avfoundation @ 0x7676c20000]   176x144@[30.000030 30.000030]fps
    Last message repeated 1 times
[avfoundation @ 0x7676c20000]   320x240@[30.000030 30.000030]fps
    Last message repeated 1 times
[avfoundation @ 0x7676c20000]   352x288@[30.000030 30.000030]fps
    Last message repeated 1 times
[avfoundation @ 0x7676c20000]   640x360@[30.000030 30.000030]fps
    Last message repeated 1 times
[avfoundation @ 0x7676c20000]   640x480@[30.000030 30.000030]fps
    Last message repeated 1 times
[avfoundation @ 0x7676c20000]   848x480@[30.000030 30.000030]fps
[avfoundation @ 0x7676c20000]   1280x720@[30.000030 30.000030]fps
[avfoundation @ 0x7676c20000]   1920x1080@[30.000030 30.000030]fps
[in#0 @ 0x7676c1c000] Error opening input: Input/output error
Error opening input file 0:none.
Error opening input files: Input/output error
```

<div data-node-type="callout">
<div data-node-type="callout-emoji">💡</div>
<div data-node-type="callout-text"><p>The issue was caused by ffmpeg defaulting to 29.97 frames per second. That's an old <a target="_blank" rel="noopener noreferrer nofollow" class="cursor-pointer" href="https://en.wikipedia.org/wiki/NTSC" style="pointer-events: none;">NTSC</a> standard used for television broadcast.<br>The other strange value, 30.00003 fps, is likely the result of a rounding error in <a target="_blank" rel="noopener noreferrer nofollow" class="cursor-pointer" href="https://developer.apple.com/documentation/coremedia/cmtime" style="pointer-events: none;">CMTime</a> structure which represents the framerate as rational numbers.</p></div>
</div>

After some fiddling, I got it working by explicitly setting the frame rate, resolution, and pixel format:

```shell
ffmpeg -f avfoundation \
  -framerate 30 \
  -video_size 1280x720 \
  -pixel_format nv12 \
  -i "0:none" \
  -vf "select=eq(n\,60)" \
  -frames:v 1 \
  -y webcam.jpg
```

And voila! That's me.

![](https://cdn.hashnode.com/uploads/covers/6409d4783e0c5ff6cf7f7de0/32dc8962-6fb5-4a84-a0e1-c47971b33d8b.jpg align="center")

As I've said - it's a terrible webcam. On the other hand, you can clearly tell there is someone in the picture 😀 It should be more than enough for this use case.

## Step 2 - Hey LLM, is anyone there?

I started by passing the image to an LLM and asking it to detect whether there's a person in it.

I used Apple' Foundation Models, available on macOS 27:

```shell
# fm respond --model system \
  --image webcam.jpg \
  "Is there a real person sitting at the desk? Answer only YES or NO."

YES
```

To make sure the model wasn't just answering "YES" to everything, I tried another picture. This time of an empty chair.

![](https://cdn.hashnode.com/uploads/covers/6409d4783e0c5ff6cf7f7de0/9474bd16-a56a-4bd6-b8ac-fa52bbab892b.jpg align="center")

And the response?

```shell
NO
```

Ta-da! I could have stopped here. The model runs locally, it does not have any dependencies and running a bash script to do this would be easy.

## Step 3 - Apple Vision framework to the rescue

But Apple also provides Vision ML models, so naturally I had to test them as well. Vision models also support human pose detection, which could be used for gesture-based microphone controls, FaceTime-style reactions and other "very useful and impactful" features nobody asked for.

I started with a simple Swift script, so I don't have to install anything.

```swift
import Foundation
import Vision

guard CommandLine.arguments.count > 1 else {
    fputs("Usage: swift detect.swift image.jpg\n", stderr)
    exit(1)
}

let url = URL(fileURLWithPath: CommandLine.arguments[1])

// Simple person detection without posture
let request = VNDetectHumanRectanglesRequest()
// Checking just the upper body, because I'm sitting most of the time
request.upperBodyOnly = true

let handler = VNImageRequestHandler(url: url, options: [:])

do {
    try handler.perform([request])

    let people = request.results ?? []

    print("""
    {
      "personDetected": \(people.count > 0),
      "count": \(people.count)
    }
    """)
} catch {
    fputs("Vision error: \(error)\n", stderr)
    exit(1)
}
```

And when executed:

```shell
# swift detect.swift webcam.jpg

{
  "personDetected": true,
  "count": 1
}
```

The first run was extremely slow, probably because Swift had to compile it. But subsequent detections were... well, *swift*.

I also tried getting the detected person's bounding box and drawing a rectangle around them. The code got a bit longer, so I'll spare you the details. The result:

![](https://cdn.hashnode.com/uploads/covers/6409d4783e0c5ff6cf7f7de0/215a3890-2c05-4150-b8d5-9598e5673d36.jpg align="center")

## Step 4 - Getting Microsoft Teams to cooperate

Before getting too excited about detecting people with my webcam, I first checked whether I could actually update Teams status programmatically. Turns out, there are several ways to do it.

### Approach 1 - Mimic user activity

If you've ever tried to address this issue yourself, you've probably come across *mouse jigglers*. They try to mimic user activity with more or less realistic mouse movement tricking MS Teams into thinking someone is actively using the computer.

![](https://cdn.hashnode.com/uploads/covers/6409d4783e0c5ff6cf7f7de0/b575fc91-31b2-46eb-8144-bbcb89cbc41b.png align="center")

I wasn't particularly interested in moving my mouse around or pressing random keys just to keep Teams happy.

### Approach 2 - Integrate into BeZoned Sidebar

Conveniently, we already have a desktop app that can update a user's Teams presence.

As of now, [BeZoned Sidebar](https://bezoned.com/bezoned-sidebar-app?utm_source=malja&utm_source=blog&utm_campaign=presence_detector) requires the user to click the Presence menu and set it manually.

![](https://cdn.hashnode.com/uploads/covers/6409d4783e0c5ff6cf7f7de0/76ec3ea6-c35c-44bf-840f-b90d380fd4fc.png align="center")

I could extend the Sidebar app to have this "presence detection" functionality baked in, or publish an API to other desktop apps. So anyone could extend the Sidebar functionality.

If I were building this as a real product feature, integrating it would probably be the best option. But this was just a weekend experiment, and I was the only user. Modifying an app everyone else relies on is a no-go.

### Approach 3 - Direct API access

What's left? Talking to the API directly. I could use the very same Graph API that BeZoned uses and set presence there.

I'd need an Entra app registration, a few API permissions and some auth logic. But that would let me update my own Teams presence without touching the BeZoned app or affecting anyone else.

# Putting it all together

I decided to stick with Swift. It's already available on my Mac and I wanted to play with it for a long time.

Additionally, it's easy to share the codebase with my colleagues if they ever want to run it as well.

When the script starts, it authenticates through Entra using the [Device code](https://learn.microsoft.com/en-us/entra/identity/conditional-access/concept-authentication-flows#device-code-flow) flow. I just open the auth page, enter the displayed code and sign in.

A deeper integration with Keychain to preserve credentials is a problem for future me.

To avoid constantly switching between Available and Away, I settled on a simple *state machine*:

![](https://cdn.hashnode.com/uploads/covers/6409d4783e0c5ff6cf7f7de0/d4e3830b-7c45-45f2-9a3a-63bf658e58b5.png align="center")

In short, the script requires two consecutive detections of the same result to set the status to Away or Available in Teams.

The final piece was connecting everything to Teams. I used three endpoints:

- [setPresence](https://learn.microsoft.com/en-us/graph/api/presence-setpresence?view=graph-rest-1.0&tabs=http) updates the presence using a session ID, which is later used to clear the presence override
- [getPresence](https://learn.microsoft.com/en-us/graph/api/presence-get?view=graph-rest-1.0&tabs=http) returns the current presence - I run this before every "switch" to ensure I'm not just hammering API with unnecessary "Change from Away to Away" requests.
- and finally [clearPresence](https://learn.microsoft.com/en-us/graph/api/presence-clearpresence?view=graph-rest-1.0&tabs=http) clears the presence configured from this script if I terminate it or when I lock the screen. Teams is then free to set it according to normal rules.

Just one delegated permission is needed for all three of them: `Presence.ReadWrite`

And that's it. When I start my computer, the Swift script launches, wakes up the old webcam and starts checking whether I'm at my desk. My colleagues are updated in real time. Away finally means I'm actually away.