In the Hierarchical Metric Flow on Data Graphs (HMFoDG) implementation, an interesting analytical feature emerged. It enables auditing of RAG systems (and embeddings or any vectorized data - biotech, medical, finance, physics, etc.) by tracing the selected associations in a deterministic way (without heuristics). This means deterministically tracking what the transformer actually used during the reinforcement process that caused one token to lie close to another. These are very concrete references (e.g., author, specific tokens from the text that in sufficient quantity allow source identification).
Yoshihiko Yoshimatsu After winning the All-Japan Championships in 1952 (photo: wikipedia)
In the output listing I get three columns:
Ultrametric (according to HMFoDG),
Cosine Similarity (everyone knows how to compute this),
Metric distance (distance between raw vectors).
That is, the same vector presented in p-adic, normalized, and raw form - and all of it sorted.
In the two known approaches the data is obvious. These are tokens from the clustering: who lies close and who lies in the same direction from the center of the embedding (I can dynamically move the "center" to a chosen token or any position if we want to manually add a new token - and we can do it without retraining, though it's worth offloading the exact placement calculations to the machine).
From the ultrametric column we get a somewhat different list than from the other two. These are the tokens the transformer had in context when it was performing reinforcement. This is not what I expected, but it is extremely useful.
The phenomenon revealed itself when I ran a test with the word "dog". I expected what everyone else does (consistent with the literature and our idealized imagination - which turned out to be wrong): some hierarchical structure of belonging, something we would call the model's "world graph." That mental image is a mistaken idealization. The learning process (whether in children or machines) is tied to what materials we present and in what order.
"Dog" lies close to "pluto", which in turn lies close to "planet", "hubble", and names of astronauts. But there was also a highly positioned name that didn't fit at all: the token "yoshimatsu". It appeared in the listing both with "doggy" and with astrophysical terms. There were also other strange tokens (including "sepraphotography").
I started narrowing the foliation to these weird phrases and looking for loops (manually, because I was only starting to automate this - it was a test listing). The transformer associated dog, pluto, and astrophysics based on the soundtrack to Takashi Yoshimatsu’s Astro Boy, where a robot named Pluto apparently appears. Along the way, Disney (Pluto the dog) and fan texts with photos (Sepra Photography) surfaced. The obvious Hubble telescope paths were so obvious (and highly associated) that I saw no reason to investigate them further (those were the expected results).
Because "dog" in cosine similarity turned out to be associated with "pig", "bear", and "fox", I checked why. The main binding token was data from SmartSurvey (as expected), and right after that - OneOpinion surveys and… Animal Crossing. I’ll skip what "pig" was connected to, because that’s where "adult content" started appearing.
Of course, thanks to SmartSurvey, tokens related to toxicity, diseases, and statistics also appeared.
This is not an exception - it’s the rule. Most of the "knowledge" in LLMs is contaminated by pop culture, memes, SEO, surveys, and random coincidences in the data. Just knowing that this is the case does not give us a solution for "what to do about it." Noise is necessary so the transformer can prepare the data for the model. But the analytical tool that emerged for me (it doesn’t have a name beyond the HMFoDG note) makes it possible to return information about which data was present in the process. From this data we can clean the training process (essentially in a near-automatic loop, because MoE immediately recognizes that these data are positioned too high) or strengthen it (where appropriate). In other words, we can prepare a "reading list in reading order" for the transformer.
This is important because at the beginning vectors are assigned generously and with strong expression, while dumping occurs in regression.
It’s as if we were thinking about the education process… of a child? Of course you can give a fourth-grader 1984 or Animal Farm, but how they internalize it into a child’s worldview can be unexpected. We don’t start mathematics with Gödel - we start with counting apples.
And when building RAG or embeddings, do we just throw everything in? Cartoons, scientific data, and adult websites? Everything the scraper found?
I know that training data is currently filtered, but which data to filter, in what order, and why requires some mathematical justification - not just the heuristic "because it works better." By chance, I stumbled upon such a deterministic method.
Naturally, any vectorized data can be analyzed this way (medical, pharmacodynamic relations, contracts and documents, stellar spectra).
But the analysis of a single person’s idiolect (i.e., a personal RAG) allows us to discover what they read, what they associated with what, what strongly influenced their worldview, and to compare to what extent two people operate with the same conceptual apparatus in selected domains.
And what is probably most practical - we can deterministically audit models this way.
Naturally, I immediately checked how popular models react to such data sequences. Each had the same context (garbage data), and despite context censoring (i.e., matching to what the user meant or might have meant based on patterns), even GPT and Grok reveal the name Takashi in the Yoshimatsu chain, even though they are completely unable to explain why they gave that particular name (it’s not a common surname - about 3,000 families have it). The chain "dog" → "pluto" → "hubble" simply projects onto Takashi, not onto the judoka Yoshihiko or the model/actress Ikumi. Only when listing further does the context of "dog" return (because there’s a "doggy" in the anime) and Pluto (supposedly some robot).
Ikumi Yoshimatsu in not sure (photo: wikipedia)
We can therefore trace why the model has very strong ultrametric shortcuts between "cute dog" and "dwarf planet" and "space telescope", but these shortcuts are garbage from the point of view of a coherent world model. Because these are the ones that surface when the model hits a wall and has to generate something - i.e., reach for increasingly distant associations after changing the foliation with context.
The data we have in training is optimized for "any plausible continuation" and "high likelihood on next token", not for building a coherent, hierarchically ordered model of reality. During training, entanglement occurs through garbage paths because there is no context-for-context mechanism - there is no ontological distinction between "Disney Pluto" and "astronomical Pluto"; only statistical: if other tokens appear (foliation) like Hubble, Aldrin, etc., then Pluto becomes "less of a dog and more of an astrophysics issue."
Models can answer school questions correctly, but their internal representation is full of such dirty shortcuts.
If we gave children such data as the basis for building a world model, the child would learn that:
Pluto the dog is connected to the planet through the Astro Boy robot,
Fox and bear are strongly linked to survey tools and Animal Crossing,
Knowledge is mainly a collection of loose, heavily weighted associations rather than clean relationships.
And yet the common imagination sees it as a hierarchical graph? Apparently the process does not work that way. And parents have a good nose when they point out a list of "works forbidden at a given stage of development." Building an anatomical model using adult magazines is generally recognized as not purposeful?
In the construction of language corpora for models, this was not taken into account. Because there was no feedback loop that determined how strongly these data influenced relations across the entire embedding system. Now such a loop exists.
The problem that appears in synthetic concepts (knowledge graph, context graph) is the a priori determination of a metric that is, by definition, too coarse - because it is based on the imagination of "how it should work, because we think so", rather than "we measured that it works this way." I measured it and I am perplexed, because "I thought it would be different," but it is as it always is. I could have just read some developmental psychology and there would probably be something there about the order of dosing knowledge, and surely something from pedagogy, but I’m not familiar with that.
Attempts to add explicit audit at decision boundaries using postulated metrics, rather than metrics residing in the models, are too primitive to catch this kind of dirty entanglement.
But the same can be applied to any research data, and we regularly encounter analytical results where some number of postulated parameters describe the same variable (g-factor).
Semantic intuition would suggest that a cartoon should not have a significant impact on the model in linking Pluto the dog and the dwarf planet (when I was studying astrophysics it was still a planet, but that was a long time ago). And yet it does.
So we have a trace of the training process identified and… now someone will probably come up with auditing operational models, and then questions about copyright will arise: "why am I on the list even though I didn’t consent and didn’t receive remuneration?"
"Here is an ultrametric path that proves that your model strengthened the association X based on my text Y, published in Z."
MoE somewhat masks the problem (prepared training data), but for human communication they are heavily contaminated (models that are not contaminated are "unpopular," like Wolfram Alpha).
The context window suffers when a large part of its capacity is used to disentangle such garbage associations instead of pure inference. Reasoning is noisy, and when you ask about a serious topic, the model has to "sift" through songs, poems, marketing surveys, Animal Crossing animals, SEO bullshit, etc. Sometimes it manages well, sometimes it produces elegant hallucinations based on these shortcuts.
The lack of hierarchical order in training means the lack of a coherent world model. You get an agglomeration of associations optimized for likelihood from the source. Trash in → trash out.
A fundamental feature of current scaled transformers + MoE is that they are trained on a dumpster without a curriculum. It works surprisingly well for "any plausible answer", but internally they are full of such ultrametric garbage.
HMFoDG (and ultrametric | cosine audit) is a tool that unmasks this - showing how the model really connects concepts, not how nicely it talks about it in the response. Because that is filtered through censoring tools.
Such entanglement analysis provides a tool for auditing the training corpus at a level that most LLM training teams still don’t have.
Every sequence/document in the corpus can be evaluated for:
How much it strengthens garbage entanglements vs. clean, hierarchical relations.
What weight it has in Fisher information flow for a given topic (astronomy, biology, physics, etc.).
Whether it introduces “context collapse” or ultrametric shortcuts that contaminate the world model.
Debiasing / reweighting / pruning:
Remove documents that are the dominant source of garbage paths (e.g., survey sites, meme forums about Pluto the dog + anime, low-quality scrapes).
Lower the weight of partially useful but heavily contaminated data.
Increase the weight of high-quality sources (scientific papers, textbooks, curated curricula, Class A Wikipedia, arXiv, etc.).
Create curriculum stages: first a clean, hierarchically ordered corpus → then controlled noise → only then the full internet.
Self-audit of the model (or MoE router) that can access this metric and evaluate its own activations/entanglements during inference or fine-tuning - in a loop. This opens the way to online cleaning or guided training, where the model actively signals “this association is garbage, it comes from anime/marketing.” The model will then be able to improve its iterations by identifying garbage data, reweighting the corpus, and obtaining a cleaner, hierarchical context model in the next iteration. In theory, moving toward what we expect from a world model, context graph, and knowledge graph.
And if you’re wondering where models get their knowledge about drug actions from - quite high up (below pharma tools and mixed with names of online pharmacies selling drugs) are… soap operas. Blackhole is mainly associated with clickbait keywords from pop culture.
The transformer, while building the corpus, has no world model - for it, a cartoon is as important as NASA data. A soap opera script dialogue weighs as much as a pharmacokineticist’s opinion.
Of course, forget about training on clean (scientific) data only. If the model started communicating in the style of Bourbaki (about a cheese sandwich):
"Thus, having two slices of bread and a cheese-like substance, we apply the operation of superimposition… from which it follows that the sandwich is constructed." - you would throw these tools in the corner.
List of references used:
I used the most garbage embedding (GLOVE300D), because it shouldn’t work on ideal data but on the worst possible noisy scrape. If it works on a dumpster, it just works. I tested it last year on PDG (particles), but there everything connects nicely, which is obvious because it’s a clean dataset. Clean data is not a good example for testing the filter.
Implementation base:
Examples (part of listing):
@searching for word : dog
@searching for code : 010001110103 | word : dog
full_path : E:\glove_QCO_NN/words\01\00\010001110103
json parsed correct;
----------------------------------------------
n2gram : [ "⟨d","do","og","g⟩" ]
----------------------------------------------
n3gram : [ "⟨⟨d","⟨do","dog","og⟩","g⟩⟩" ]
----------------------------------------------
n4gram : [ "⟨⟨⟨d","⟨⟨do","⟨dog","dog⟩","og⟩⟩","g⟩⟩⟩" ]
----------------------------------------------
leave_at is leave; //I mean leaf in p-addic tree;
att lvl : 17 | in category : E:\glove_QCO_NN\U_categories/4f4f4h\4e4f4i\4f4g4b\b4g4h4\4f4f4h4e4f4i4f4g4b4g4h4i4h4h4d4h4e4i4c4h4h4h4i4f4j4g4d4c4g4i4h4g4i4g4g_18
My comment:
It means that it exists in range 0-68 at level 17, so it belongs to semantic word with ambiguity. This metric is calculable, I gonna add it to app in few days, so its gonna reffer precisely to numbers.
It is leaf mean that it is not a noise filler. It is found automagicly, check fixed_full_raport_paddic_analyse.txt
category_belonging belong to category;
E:\glove_QCO_NN\U_categories/4f4f4h\4e4f4i\4f4g4b\b4g4h4\4f4f4h4e4f4i4f4g4b4g4h4i4h4h4d4h4e4i4c4h4h4h4i4f4j4g4d4c4g4i4h4g4i4g4g_18
E:\glove_QCO_NN\U_categories/4f4f4h\4e4f4i\4f4f4e\e4e4g4\4f4f4h4e4f4i4f4f4e4e4g4h4g4f4g4h4e4d4e4h4e4h4e4g4d4h4g4h4e4g4g4h4g4f4f_16
My comment, category where it is a leaf is obvious, but in next cattegory it fall into noise filler (there are different leafs, so lower in semantics the are words less "special" that make "dog" araise from the noise.
This table is not ready yet (I did around 10% of 42milion operations on 1.2milion tokens as far; dog is done by coincidence, hence it came out in testing);
Nearest table :
position| Ultrametric | CosineSimiliarity | EuclidianDistance
0 | unrivalled | retriever | retriever
1 | crossbred | pigs | belong
2 | servics | car | pigs
3 | wrangle | backyard | car
4 | dawdlers | belong | backyard
5 | togeder | garden | demands
6 | naugrim | anxiety | garden
7 | 따르면 | demands | occupied
8 | lightposts | claims | claims
9 | quake3 | bugs | battling
10 | ruedas | why?--> planet | sheds
11 | nariman | moon | moon
12 | kilobuck | occupied | planet
...
511
Now we check planet...
Nearest table :
position| Ultrametric | CosineSimiliarity | EuclidianDistance
0 | egyptians | moon | moon
1 | 1257 | humanity | humanity
2 | giacomelli | many | many
3 | spruces | darkness | dawn
4 | manifest | dawn | forefront
5 | darkness | book | occupied
6 | leasing | intergalactic | battling
7 | samir | occupied | darkness
8 | grocery_or_supermarket | battling | intergalactic
9 | changd | forefront | manifest
10 | lambada | beauty | book
11 | cherif | neptune | demands
12 | rainis | manifest | neptune
13 | adrover | reading | posited
14 | posited | demands | claims
15 | battling | claims | beauty
17 | defied | why?--> dog | lifetimes
...
511
Now we check pluto (I marked what is found in crosschecks):
Nearest table :
position| Ultrametric | CosineSimiliarity | EuclidianDistance
0 | secci | hubble | begun
1 | vripack | begun | hubble
2 | loyalists | ambitious | venture
3 | misael | seiya | ambitious
4 | anglotopia | venture | decades
5 | sepra | aldrin | rechristened
6 | roald | decades | feeds
7 | l’ancien | euclid | rebranding
8 | crasher | polaris | polaris
9 | funnymen | landsat | obscured
10 | makus | daily | seiya
11 | -->>here--> yoshimatsu| owned | owned
12 | yams | feeds | euclid
13 | beilin | trust | spotted
14 | base. | speculation | speculation
15 | rathgeb | anxious | vindicated
16 | siq | projects | fortunes
17 | firstbuild | films | majorly
451 | paddington | kammu | yoshimatsu //again
Now hubble:
Nearest table :
position| Ultrametric | CosineSimiliarity | EuclidianDistance
0 | huckster | pluto | pluto
1 | crankiest | landsat | decades
2 | decentral | aldrin | begun
3 | aaboud | radiometer | undertaken
4 | seen. | decades | spotted
5 | decades | spotted | vindicated
6 | undertaken | undertaken | landsat
7 | begun | projects | unearthed
8 | unearthed | begun | aldrin
9 | nima | films | venture
10 | projects | unearthed | rebranding
11 | funded | papers | projects
12 | vindicated | funded | papers
13 | spotted | inspections | indebted
14 | ambitious | venture | ambitious
15 | landsat | ambitious | nima
...Only when listing further does the context of "dog" return (because there’s a "doggy" in the anime) and Pluto (supposedly some robot).
Searching for this manually would be a real pain, but the machine:
241 | gillars | yoshimatsu | kriel
For Yoshimatsu, the ultrametric pulls the context extremely strongly - anime/soundtrack Astro Boy + Pluto + Sepra Photography + various niche sites. Cosine similarity is more "blurry" and mainly catches general names/people. This is a classic case of messy pop-cultural entanglement.
Here's an example of two more meanings that are illustratively "polluted" in the embedding (and in language in general) - which is exactly why it's so hard to communicate what something actually means.
"grok" - a very interesting and funny result.
The ultrametric strongly reveals the memetic-internet noise: debug, install, ubuntu, npm, arstechnica, memelord, etc. A lot of technical slang, forums, and old posts. This shows that the word "grok" (from Heinlein) has been heavily "hijacked" by programmer/hacker culture and memes. There is no clean, literary path - internet noise dominates. Training on trash distorts the original meaning.
"consciousness" - here we see a mixture:
Philosophy/psychology (worldviews, mysticism, cybernetic, dissociation, monadic, shakyamuni)
But also a lot of noise (thewrap.com, ensembleiq, homebodies, gigantor, various niche numbers and names).
Ultrametric pulls out more "human"/philosophical contexts than cosine. Still, there's plenty of garbage - which shows that even such an abstract concept is heavily polluted.
Ultrametric consistently better captures the real training contexts (the paths the transformer actually walked), while cosine similarity more often gives "flat" semantic similarity. The tool unmasks the true structure of the model's embedding, not just its pretty surface.
So I wouldn't have detected this manually, but the machine made a loop through the data - "what doesn't fit here" - and can list many answers explaining "how it ended up in the embedding in this neighborhood." And let's keep in mind that the machine can communicate with us in a way that we understand. That is, we use language that contains such associations which are absurd and non-obvious to us. Ontologically, we would not agree with them, but in the data they exist.
I looked at the test results and accepted them. To this absurdity I can only comment:
Praise A’Tuin, for the Turtle is moving!
-- Jack Kowalski.