Skip to content

Feature: seed cluster() with a previous partition, so re-clustering does not shatter stable communities #2452

Description

@duncancrawford

The problem

graphify.cluster.cluster() runs community detection from scratch on every call. On a large graph that is refreshed incrementally, this makes re-clustering far more destructive than the underlying change.

Measured on a 793k-node, 1.77M-edge multi-repo graph where a source refresh changed roughly 1.5% of nodes:

  • 88 previously-stable communities shattered into 2–8 fragments each;
  • 349 of 418 successor communities were compositionally pure (median 95% of members from one predecessor) yet held less than half of that predecessor's members — splits, not renames;
  • downstream, every per-community authored artefact keyed to those communities had to be regenerated, despite remap_communities_to_previous correctly preserving ids for the communities that did survive intact.

remap_communities_to_previous fixes labelling after the fact, but nothing steers the partition itself toward the previous one, so unchanged regions of the graph can land on a different local optimum every run.

The ask

An optional seed for cluster():

cluster(G, resolution=1.0, exclude_hubs_percentile=None,
        seed_partition: dict[str, int] | None = None)

initialising Louvain/Leiden from the previous membership rather than from singletons. Both algorithms support warm-start partitions; python-louvain exposes it directly (community_louvain.best_partition(G, partition=seed)), and leidenalg via initial_membership. Nodes absent from the seed start as singletons; nodes in the seed start in their previous community and move only if modularity actually improves.

This composes naturally with remap_communities_to_previous — seeding reduces how much there is to remap, and the existing function still handles genuine drift.

Why it matters beyond us

Any consumer attaching content to communities (summaries, labels, docs) pays the shatter cost on every refresh. The LLM community labels graphify itself generates are an in-tree example: our re-cluster invalidated 22,507 of 28,004 saved labels by membership signature, almost all of which would have survived a seeded run.

Related context: #2436 (the cluster-only persistence report) came out of the same refresh.

Versions: graphify 0.9.30, networkx 3.6.1, Python 3.13.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions