In a previous post on the Visibility Gap, I mentioned automated discovery as the sustainable alternative to manually maintained catalogues. Several people came back with the same question: what does that actually mean in practice?
It is a fair question. The phrase gets used loosely, and the tooling landscape is crowded enough to be confusing. This post gives a clear and honest picture of what automated discovery is, the four distinct forms it takes, where each one applies, and just as importantly, where it does not.
Most architecture teams that attempt a service catalogue start with the same approach: ask teams to register what they own. Fill in the form. Keep it updated.
It never stays current. Not because teams are careless, but because under delivery pressure, updating a catalogue feels like overhead with no immediate payoff. It gets deferred. Then forgotten. Then a new team joins and has no idea the form exists.
The result is a catalogue that reflects what teams thought existed at the time they last updated it, which is not the same as what actually exists now. In fast-moving environments with microservices, that gap opens quickly.
Automated discovery flips the model. Instead of asking humans to tell the system what exists, the system goes and finds out for itself.
At its core, automated discovery means a tool scans your actual infrastructure, including git repositories, Kubernetes clusters, cloud accounts, API gateways, CI/CD pipelines, and databases, and builds or updates a catalogue from what it finds there, rather than from what people remember to register.
The distinction that matters: manual registration is aspirational governance. Automated discovery is operational governance. One depends on human behaviour under delivery pressure. The other does not.
There are four distinct types, and they apply in different contexts.
This is the most direct answer to the dark matter problem, which refers to services, deployments, and infrastructure components that exist in production but have no entry in any catalogue.
Tools in this space scan git repositories and Kubernetes namespaces to surface what is actually running. When a team deploys a new microservice, it appears in the catalogue automatically with no registration step required. When a service is decommissioned, it disappears. The catalogue stays current because it is reading from the same source of truth as the deployment pipeline.
The practical implication is significant. One tool in this space was able to import 315 services, representing 97% of a full service architecture, in nine minutes, immediately after integrating with a git provider. That is not a documentation exercise. That is a functioning catalogue of what genuinely exists.
Where this applies best: Organisations running microservices on Kubernetes or cloud infrastructure, with multiple teams deploying independently. The more teams there are and the faster they move, the more valuable automated ingestion becomes relative to manual registration.
Tools commonly used here include Backstage (Spotify/CNCF), OpsLevel, Port, Cortex, and Atlassian Compass, each with different tradeoffs between flexibility, speed of deployment, and engineering overhead required to maintain them.
APIs present a specific and often more serious visibility problem. They get created for good reasons, evolve without governance, and sometimes continue to be called in production long after the team that built them has moved on or the system they were built for has changed.
Shadow APIs, those deployed outside any formal governance process, are particularly insidious. They operate without security controls, without versioning discipline, and without anyone tracking who depends on them. According to Gartner's 2025 API Management Forecast, over 60% of enterprises experience outages or breaches annually due to unmanaged or misconfigured APIs. Most of those APIs were not rogue deployments. They were simply never properly catalogued.
Automated API discovery works at two levels. The first is gateway-level: if traffic flows through a managed API gateway, the gateway sees every call and can surface what APIs exist. The second is network-level scanning, which can detect APIs being called even if they bypass a central gateway. This is a more complex approach, but necessary in environments with significant legacy or ungoverned traffic.
Where this applies best: Organisations that already have a managed API gateway get significant discovery almost for free, since the gateway sees the traffic. Organisations without a centralised gateway have a harder problem, and the honest answer is that discovery will be partial until a gateway is in place.
Event-driven architectures, built on Kafka, Pulsar, or similar platforms, introduce a visibility problem that is subtler and often more damaging than uncatalogued services.
Schemas drift. A producer changes the structure of a message without notifying downstream consumers. The consumers continue running. For a while, nothing breaks visibly. Then something downstream starts producing corrupt data, or silently dropping events that no longer parse correctly. By the time it surfaces, tracing the root cause is extremely difficult.
Schema registries address this directly. They provide a central repository where every event schema is registered, versioned, and enforced. Producers cannot publish a schema change without it being recorded. Consumers can detect when a schema they depend on has changed. The registry becomes the governance layer for the event contract.
The discovery aspect here is slightly different from the service case. Rather than passively finding things that exist, schema registries actively enforce that all schemas are registered before being used, making invisible drift structurally impossible rather than just unlikely.
Where this applies best: Any organisation running event-driven architecture at scale. The larger and more distributed the producer-consumer landscape, the higher the risk of silent schema drift and the higher the value of automated governance.
Without a data catalogue, architecture leaders cannot answer basic questions: where does this data come from? Who owns it? What depends on it? What is PII?
For organisations subject to GDPR, financial services regulations, or sector-specific data requirements, the inability to answer those questions is not just an operational problem. It is a compliance failure waiting to materialise, and regulators increasingly expect demonstrable evidence of data oversight, not just assertions of it.
Data discovery tools crawl database schemas, data warehouses, and data pipelines automatically, cataloguing tables, columns, data types, and relationships, and flagging potential PII based on field names, patterns, and content sampling. Rather than asking data owners to document what they hold, the crawler surfaces it. Rather than maintaining lineage manually, the tool traces data flows from source to consumption automatically.
Where this applies best: Regulated industries with complex data environments including financial services, healthcare, and any organisation handling significant volumes of personal data. Also particularly valuable when an organisation cannot confidently answer the question "where is our customer data stored?", which is more common than most leaders would want to admit.
This is worth being clear about, because the tooling landscape tends to oversell universality.
Automated discovery works well when services are containerised, cloud-hosted, and running through modern CI/CD pipelines. The tools are designed to read from git, Kubernetes, and cloud provider APIs, which are environments that expose structured, queryable metadata.
They work less well with legacy on-premises systems that do not expose the metadata these tools expect to find, mainframe environments, older monolithic applications where service boundaries do not exist in a discoverable form, and SaaS applications managed by vendors that sit outside the governance perimeter entirely.
Many large UK enterprises, particularly in government and financial services, have significant legacy estates where automated discovery gives a partial picture at best. In those environments, the honest answer is a hybrid approach: automated discovery for the modern estate, and structured manual cataloguing for the legacy one.
One concern that comes up regularly: if the tool discovers everything automatically, does anyone actually understand what they own?
The answer is that automation removes data entry, not decision-making. The typical flow is that the tool discovers and surfaces candidates, flags them for review, and a human confirms ownership and adds context. From that point forward the catalogue stays current automatically, but the accountability structure remains human.
Automated discovery does not mean ungoverned discovery. It means governance is applied at the right moment, by the right person, rather than being distributed across hundreds of developers who may or may not remember to register things at the right time.
When architecture leaders talk about carrying operational risk they cannot see, the root cause is almost always some version of this problem: a landscape that has grown faster than the governance designed to track it.
Automated discovery does not solve the governance problem on its own. But it removes the single most common reason governance fails, which is the dependence on human memory and human discipline under delivery pressure to maintain the picture.
When the catalogue is automated, the architecture leader can spend their attention on what it reveals, including the ownership gaps, the dependency risks, and the compliance exposures, rather than on the effort of assembling it in the first place.
That is the shift. Not more governance. Governance that actually keeps up.