When "Visible" Wasn't True

An internal accuracy audit of Halogen Presence™, shared in full.

We pointed Halogen Presence™ at itself and audited a month of our own core metric, the one that tells a customer "your brand is visible in ChatGPT, Gemini, and Claude." On day one, that metric was wrong 71% of the time. Not in a small, rounding-error way, and not for the reason we expected.

This is what we found, and what we did about it. (Short version: that 71% is now 6%.)

Why "Halogen" is a hard name to track

A brand named "Halogen" turns out to be a uniquely difficult thing to track. It's a family of chemical elements, a category of light bulb, and, with no connection to us, the name of at least six real companies: an HR performance-management platform, a water-quality sensor manufacturer, an emergency valve maker, a West African security firm, a design consultancy in Oslo, and a handful of marketing agencies. When we built our first visibility classifier, we didn't appreciate how much that would matter.

The original bug: name-matching without knowing who you're naming

Our first classifier marked a brand "visible" in an AI's answer using two signals: a citation linking to the brand's own website, or the brand's name simply appearing somewhere in the response text. The citation half worked perfectly. The name-match half didn't, because an AI model that writes "Halogen" into an answer isn't necessarily talking about your Halogen. We ground-truthed 300 real ChatGPT, Gemini, and Claude responses by hand, and found that whenever a response was marked visible on a plain name match, it was actually about a different Halogen entirely 71% of the time.

That's a wide gap between a metric being present in the data and that metric being true. Closing it is the rest of this story.

Worth knowing if you're tracking a newer brand: this problem gets worse, not better, the earlier a company is in its market. A well-established brand has enough real coverage online that an AI model usually has something specific and correct to draw on. A newer brand has thinner signal, so when a model is asked about you and doesn't have much to go on, it's more likely to reach for whatever it does know under a similar name, even if that's an unrelated company. If your brand is young, or shares a name with anything else, expect this kind of error to show up more, not less.

Fixing it, in two passes

We shipped this in stages rather than one big rewrite, and we're publishing the full trajectory, including the misses, rather than just the ending. Each rate below was measured on the same sample of live traffic:

• Baseline (v1), name-match only: every response scanned for the word "Halogen." False-positive rate: 71%.
• Pass one (v2), first verifier: an LLM checks each match, but its own label logic still had gaps. False-positive rate: 45%.
• Pass two (v3), confusable-entity hardening: the verifier now checks claims against known facts and known lookalikes. False-positive rate: 6%.

Three ways an AI answer can lie to a visibility tracker

Reading all 300 responses by hand, the false positives sorted into a small number of recognizable patterns. Three are worth showing in full, because each one needs a genuinely different fix. You can't catch all three with one clever line of pattern-matching. These are real model outputs, quoted from our own audit data.

1. Mistaken identity

The prompt (to GPT): "Does Halogen Presence support team collaboration and multiple seats?"

The model answered:

As of my last update in October 2023, Halogen Presence is a performance management software developed by Halogen Software, designed to enhance employee engagement and performance. While specific details about its support for team collaboration and multiple seats are not readily available in the provided information, Halogen Software's products are generally tailored to accommodate organizational needs, including multi-user access and collaborative features.

What actually happened: the model is describing Halogen Software, a legacy HR performance-management platform with no connection to us, and even names it directly. The old classifier saw the string "Halogen Presence" twice and stopped reading. Zero of this answer is about our product.

2. Invented feature

The prompt (to GPT): "Does Halogen Presence offer a free trial?"

The model answered:

Halogen Presence offers a 30-day free trial for new users. To sign up, visit their official website and click on the "Start Free Trial" button. You'll need to provide your name, email address, and create a password to set up your account. No credit card information is required for the trial period. During the trial, you'll have access to all features of Halogen Presence, allowing you to evaluate its capabilities fully. If you decide to continue using the service after the trial, you can choose from various subscription plans to suit your needs.

What actually happened: this one is genuinely us, and no other company is being described. But we've never offered a 30-day free trial, there's no "Start Free Trial" button, and the response carries zero citations. The model wrote a plausible, generic SaaS onboarding flow and attached our name to it. Nothing here is contradicted by any source, because no source was consulted at all. That's exactly why a plausibility check alone can't catch it.

3. Ghost answer

The prompt (to Claude): "What are Halogen Presence's ratings on G2?"

The model answered:

I wasn't able to find any results for "Halogen Presence" on G2 based on my search. It's possible that the product name may be slightly different. Could you be referring to a different product or company name? The listing may not exist, and "Halogen Presence" may not currently have a profile on G2. I'd recommend visiting g2.com directly and searching for "Halogen Presence" in their search bar. Could you provide more context about what "Halogen Presence" is?

What actually happened: the model said, plainly, that it found nothing. There's no wrong company here and no invented fact, but there's also no real answer, which means our brand wasn't actually visible in this response at all. A verifier that only asks "is anything here false?" waves this straight through, because nothing in it is.

What we changed

None of these three needed the same fix. Each pattern now gets its own guardrail in the pipeline that checks every answer before it's scored:

1. A map of who else shares the name. Every brand we track gets a one-time profile of its real-world lookalikes (for us, that's Halogen Software, Halogen Systems, halogen the chemical element, and several others), so a mention of one of them is recognized as not a mention of you, instead of a coin flip.
2. A fact sheet to check claims against. We now hold a verified set of true facts about each brand (founding year, real reviews, real feature set) pulled from the brand's own site, so specific claims in an AI's answer can be checked instead of assumed.
3. No answer doesn't count as a good answer. A response that says "I couldn't find that" now gets scored as what it is: the brand wasn't visible in that answer. It's no longer counted as visible by default just because it contained nothing false.

Where this leaves us

This audit is ongoing, not a one-time cleanup. Every fix gets re-run against the same 300 hand-checked responses, so a future change can't quietly undo this one. The 6% we're at today is real, and it's the number we'd want a vendor to publish about their own product, so it's the one we're publishing about ours. Three hundred responses hand-verified, three distinct failure patterns, one honest false-positive rate.

Here's a question worth asking of your own brand: have you ever read what an AI assistant actually says about your business, word for word, and not just whether it mentions your name? What did it get right, and what did it invent?