What AI Systematically Cannot Replace

AI systems are powerful interpolators. They synthesize within their training distribution extremely well. But several categories of expert value remain structurally outside that reach:

AI as a Force Multiplier for Specialists

The more productive frame is not “AI vs. experts” but AI-augmented experts vs. unaugmented experts. The productivity differential between those who can direct AI precisely and those who cannot is widening rapidly.

Four concrete moves follow from this framing:

1. Compress routine deliverables. Reports, summaries, code scaffolding, documentation — tasks that once took days now take hours. This frees capacity for higher-value engagement.

2. Scale proprietary knowledge. Domain expertise becomes a durable moat only when embedded in systems, workflows, and products — not just in a person’s head. AI makes this extraction tractable.

3. Shift from time-billing to outcome-pricing. When execution time falls, the old hourly model erodes. The new model prices the value of the outcome — insight, decision quality, risk reduction — not the hours spent.

4. Build AI-native products around expertise. The highest-leverage play: encode specialist knowledge into a product or platform that scales independently of billable time. This is the transition from practitioner to product company.

Where Graph Expertise Is Uniquely Defensible

The graph domain sits at a particularly interesting intersection. As AI systems grow more capable, their own internal reasoning is increasingly structured as graphs — knowledge graphs, semantic networks, entity stores. The infrastructure to build, validate, visualize, and query these structures requires expertise that is sparse and deeply technical.

As a person or as a company, if you invest considerable amounts of time and money in something or if the strategic position of your product needs consideration you will want to talk to a human being. If you have a terminal disease, you want to talk to a person and not AI. Trust needs eye contact and a hand-shake. People are complex beings and the things that matter are always a combination of ratio, emotions and intuition. Many challenges don’t have a clear formulation or answer and dealing with the undefined aims (and the things between the lines) is what makes consulting and human contact as important as ever.

Personally, I think business is about people. Life is about people. Having a coffee or a beer with AI is not around the corner and the covid years have proven we are more than information processors.

Three Moves (Still) That Matter

Productize

Convert expertise into repeatable, scalable products. Every engagement that requires the same deep knowledge is a product waiting to be built. The goal is revenue that does not require proportional time investment.

Specialize Deeper

Generalism is more exposed. The more specific and verifiable your domain mastery, the less substitutable you are. Breadth is available via AI; depth is not. Go narrow and go deep.

Adopt Aggressively

Use AI across every workflow. The goal is to work at a pace and scope that was previously impossible for a one-person operation. The one-person firm that builds AI-augmented products is more competitive today than a five-person firm that doesn’t.

A Considered, Positive Stance

The framing of “AI replacing skills” is both true and misleading. True, in that a specific set of execution-oriented tasks are now cheaper and faster to produce. Misleading, in that it obscures what is growing in value: judgment, synthesis, accountability, and the ability to direct AI toward problems that actually matter. Talking matters, trust comes with communication and time.

For deep specialists — particularly those who can reason across technical and business domains — this is one of the more favorable environments in a generation. The barriers to building products have dropped. The demand for structured, trustworthy knowledge systems is rising. The combination of deep expertise and AI leverage is genuinely rare. The threshold to build amazing things has lowered. If you are creative and/or innovative these are golden times to go where no one has bone before.

The professional attitude for this moment is neither dismissal nor alarm. It is clear-eyed adaptation: understanding precisely which parts of your work are being commoditized, doubling down on what is not, and using the tools available to operate at a scale that was previously inaccessible.

Concepts and dimensions

The ingredients of a RAG solution are fairly simple and if you take a step back you can observe the following elements:

• documents
• chunks
• metadata (doc, chunk, node…)
• entities (links, graphs…)

Each of these elements can be tuned or improved via diverse techniques. Together they form a structural dimension along which you can experiment.

There are also three processing dimensions in a RAG pipeline:

• ingestion
• query
• feedback (optional)

Finally, you can also play with the more technical aspects in a pipeline:

• chunking strategy (semantic, size…)
• vector comparisons, embedding models
• indexing techniques and (re)ranking (BM25, neural reranking…)

The twenty techniques highlighted below can all be place within these three dimensions. I call them dimensions because each has a certain level of complexity and sometimes incremental. Whether you need one or the other is more an art than science, many factors can play a role in deciding what to use:

• the business context
• the corpus or repository
• the budget
• the accuracy of the end-result
• the type of user-experience
• the time allowed for processing queries
• how dynamic the corpus is and the necessity for updates
• whether knowledge has to be structured (hierarchic, graph, visualizations…)

You don’t necessarily need a knowledge graph, ontology and agentic frameworks to be successful. Many RAG projects can do without a feedback pipeline or explainable AI. What matters is to understand the business case and the options at your disposal.

Take a look at our article regarding QA generation for DeepEval which highlights the process of evaluating a RAG solution.

I Document compressors

• LangChain contextual compression
• LlamaIndex research on compression
• CompAct: Compressing Retrieved Documents Actively for Question Answering

II Header augmentation (aka out of context chunks, contextual chunk headers or CCH)

• LangChain implementation
• Mix-of-Granularity: Optimize the Chunking Granularity for Retrieval-Augmented Generation
• Searching for Best Practices in Retrieval-Augmented Generation
• Financial Report Chunking for Effective Retrieval Augmented Generation
• Introducing a new hyper-parameter for RAG: Context Window Utilization
• RAGulator: Lightweight Out-of-Context Detectors for Grounded Text Generation

III Query augmentation (aka query expansion)

• LangChain query expansion via a multi-query retriever
• LlamaIndex query transformation cookbook

IV Hypothetical document embedding

• Precise Zero-Shot Dense Retrieval without Relevance Labels
• LangChain HypotheticalDocumentEmbedder
• LlamaIndex query transformations (including HyDE)

V Graph RAG

• An extensive compilation of graph RAG research papers
• Microsoft Graph RAG
• Formerly Neo4j GenAI but now called Neo4j GraphRAG for Python
• Nano Graph RAG
• LightRAG
• Fast Graph RAG
• TrustGraph

VI Adaptive RAG

This techniques introduces a classification of a given question prior to one of several specialized RAG pipelines. For instance, a question can be cataloged as “factual” or “opinion” and for each an appropriate prompt handles the question accordingly. This can also mean that the question is multiplied into multiple question. If it’s an “opinion” this would involve asking the LLM first what aspects are relevant and generating a separate question for each perspective.

In essence, this is prompt tuning and not so much RAG tuning but the difference is sublte and can lead to better answers.

• Our own take on adaptive RAG
• Adaptive RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity
• Self-adaptive Multimodal Retrieval-Augmented Generation (SAM-RAG)
• MBA-RAG: A Bandit Approach for Adaptive Retrieval-Augmented Generation through Question Complexity
• CtrlA: Adaptive Retrieval-Augmented Generation via Probe-Guided Control
• SeaKR: Self-aware Knowledge Retrieval for Adaptive Retrieval Augmented Generation
• RetrievalQA: Assessing Adaptive Retrieval-Augmented Generation for Short-form Open-Domain Question Answering
• LangGraph implementation
• LlamaIndex implementation

VII Context Enrichment

• Extracting Metadata for Better Document Indexing and Understanding

VIII Corrective RAG (aka CRAG)

• LangGraph CRAG
• Corrective RAG
• LlamaIndex CRAG

IX Explainable RAG

• RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems
• RAG-based Explainable Prediction of Road Users Behaviors for Automated Driving using Knowledge Graphs and Large Language Models

X Fusion retrieval

• RAG-Fusion: a New Take on Retrieval-Augmented Generation
• LlamaIndex simple fusion retriever

XI Hierarchical RAG

• LlamaIndex Structured Hierarchical Retrieval
• RAG Strategies - Hierarchical Index Retrieval

XII Propositional chunking

Dense retrieval performance is significantly impacted by the choice of retrieval unit, particularly when using propositions, which are atomic expressions encapsulating distinct factoids. Fine-grained retrieval units, such as propositions, outperform passage-level units in retrieval tasks and improve downstream QA tasks. Propositional chunking involves breaking down text into atomic units called propositions, each representing a distinct fact or idea.

• Meta-Chunking: Learning Efficient Text Segmentation via Logical Perception
• Dense X Retrieval: What Retrieval Granularity Should We Use?

XIII Query rewriting

• Query Rewriting for Retrieval-Augmented Large Language Models
• RePhraseQuery LangChain retriever
• An Introduction to LlamaIndex Query Pipelines

XIV Raptor

• LlamaIndex Raptor implementation
• LangChain Raptor cookbook
• RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval

XV Relevant segment extraction

Relevant Segment Extraction (RSE) is an optional (but strongly recommended) post-processing step that takes clusters of relevant chunks and intelligently combines them into longer sections of text that we call segments. These segments provide better context to the LLM than any individual chunk can.

• Introducing Relevant Segment Extraction (RSE)

XVI Reliable RAG

The “Reliable-RAG” method improves RAG by incorporating layers of validation and refinement to enhance the accuracy and relevance of retrieved information. The method incorporates checks for document relevance, hallucination prevention, and highlights the exact segments used in generating the final response.

• Trustworthy RAG with the Trustworthy Language Model
• TrustLLM: Trustworthiness in Large Language Models
• Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models’ Alignment

XVII Reranking

• FlashRank reranker
• RAG Workflow with Reranking

XVIII Feedback loop

XIX Self RAG

• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
• Self Correcting Query Engines - Evaluation & Retry
• LangGraph SelfRAG

XX Semantic chunking

• LlamaIndex semantic chunker
• How to split text based on semantic similarity

NetworkX Exploration

The above can be easily explored in NetworkX. Let’s take the classic Karate Club graph

import networkx as nx
import numpy as np

g = nx.karate_club_graph()

# the adjacency matrix
A = nx.adj_matrix(g)

# the Laplacian
Delta = nx.laplacian_matrix(g)

# from the formula we can get the degree matrix
D = A.toarray() + Delta.toarray()

# to confirm, you can take the degrees explicitly
print(np.diag(D))
print([nx.degree(g, u) for u in g.nodes])

[16  9 10  6 ... 17]
[16  9 10  6 ... 17]

The eigenvalues can be obtained via the laplacian_spectrum method:

eigenvalues = nx.laplacian_spectrum(g)
np.round(eigenvalues,2)

array([-0.  ,  0.47,  0.91,  1.13,  1.26,  1.6 ,  1.76,  1.83,  1.96,
    2.  ,  2.  ,  2.  ,  2.  ,  2.  ,  2.49,  2.75,  3.01,  3.24,
    3.38,  3.38,  3.47,  4.28,  4.48,  4.58,  5.38,  5.62,  6.33,
    6.52,  7.  ,  9.78, 10.92, 13.31, 17.06, 18.14])

You can see that the first eigenvalue is indeed zero and that it has multiplicity one, meaning that the graph is connected. The Karate Club has one (weakly) connected component.

The eigenvectors are not directly available in NetworkX but you can use Scipy for this:

from scipy.linalg import eigh
# Calculate the eigenvalues and eigenvectors of the Laplacian
eigenvalues, eigenvectors = eigh(Delta.toarray())
np.round(eigenvectors, 1)

array([[-0.2, -0.1, -0.1, ...,  0.1,  0.9, -0.2],
   [-0.2, -0. , -0.1, ..., -0.1, -0.1, -0. ],
   [-0.2,  0. , -0. , ...,  0.3, -0.1, -0. ],
   ...,
   [-0.2,  0.1,  0. , ...,  0.1, -0.1,  0.1],
   [-0.2,  0.1,  0. , ..., -0.9,  0.1,  0.1],
   [-0.2,  0.1,  0. , ..., -0. , -0.2, -0.9]])

Note that the first column is the unit (up to an irrelevant scaling factor) corresponding to the zero eigenvalue.

Introduction: The Challenge of Limited Context in LLMs

At its core, the issue we address with RAG stems from the limitations of LLMs. These models are trained on a fixed corpus of data at a specific point in time. This creates two major constraints:

1. Data Scope: LLMs cannot access proprietary, specialized, or private data unless explicitly provided.
2. Time Sensitivity: Models become outdated as new information emerges.

The good news is that LLMs are designed to incorporate additional information when it’s included in a prompt. This is the essence of Retrieval-Augmented Generation (RAG). By retrieving and supplying relevant context to the LLM during inference, companies can utilize LLMs with their private or proprietary data without retraining or exposing sensitive information.

However, RAG introduces its own challenges:

• Token Limits: LLMs have finite prompt sizes, limiting the amount of contextual information you can provide.
• Context Relevance: How do you identify and retrieve the most relevant information to include in the prompt?

These challenges divide the problem into two main areas:

1. Retrieval: Identifying the data relevant to a query.
2. Prompting: Formatting the retrieved data to maximize LLM performance.

The development of prompt engineering has emerged as a distinct discipline, focused on crafting input formulations that elicit accurate and predictable responses from LLMs. This process is inherently model-specific, with each LLM exhibiting unique characteristics that render universal prompting strategies ineffective. Consequently, efforts to codify and standardize prompt engineering have been undertaken (see e.g. DSPY), albeit with varying degrees of success. Notwithstanding these challenges, the efficacy of well-crafted prompts in yielding high-quality responses remains a consistent truth, regardless of the context from which they are derived.️

How to retrieve (business) data relevant to a question is where knowledge graphs enter the picture. The problem is in fact independent of LLMs: given a question, how to query and retrieve information from a repository relevant to the question? Systems like Confluence, Sharepoint and others index documents and much like a table of contents or the index in a book, they return a list based on indexing and keyword matches. Frameworks like Apache Lucene have been integrated in countless systems to this end. This is the approach from the 20th century and has been superseded by vectors and embeddings.

The advent of machine learning has introduced a paradigm shift in natural language processing (NLP), predicated upon the concept of utilizing vectors to encapsulate linguistic information. While traditional NLP methodologies have long employed techniques such as chunking and tokenizing, which involve dissecting text into constituent parts, this approach remains relevant within the context of large language models.

In contrast, the notion of text vectors represents a novel approach to NLP, wherein the focus shifts from analytical examination of syntax, grammatical structure, coreference, and other linguistic phenomena to a more holistic treatment of language as a classification and feature engineering problem. By creating comprehensive vectors for words, this methodology ensures that pertinent information is preserved. The underlying principle is that text can be regarded as a numerical construct, analogous to an image being represented by a sequence of RGB values. Furthermore, the resultant vectors exhibit a degree of independence from their original textual context, capturing the essence of language in a manner akin to how children absorb and later specialize in linguistic terminology throughout their lives.️

The process of converting linguistic inputs into vector representations enables facile comparison and analysis. This methodology allows for the identification of semantic relationships between words, such as those linking “child” and “parents”, and facilitates the quantification of their proximity. In contrast, direct comparisons of words are often impractical due to their inherent complexity. Prior to the advent of LLMs, knowledge graphs and bases were constructed by domain experts to establish connections between concepts. However, this approach was labor-intensive and prone to errors, as it relied on manual curation. The development of biochemical and oncological knowledge bases exemplifies this challenge, requiring painstaking effort from human experts. These knowledge bases, while valuable, are often difficult to query and maintain due to their complexity. In response, a distinct academic discipline emerged, encompassing knowledge management, ontologies, SPARQL, and SHACL. Notably, the advent of LLM-based approaches has largely supplanted these traditional methodologies, offering a more efficient and effective means of information representation and retrieval.️

The advent of vectorized knowledge enables the comparison of semantic representations, thereby facilitating the identification of related concepts and entities. This approach obviates the need for explicit knowledge graphs or indexing systems, instead leveraging the inherent relationships between vectors to reveal connections.

For instance, when applied to a dataset containing information on climate change, this methodology can uncover associations between CO2 emissions and sea level rise, among other topics, without the necessity of manual graph construction. This technique underlies the transformative potential of artificial intelligence, empowering innovative applications and insights.

With vectorized knowledge in place, it becomes straightforward to extract relevant context from an information base to support a LLM. The process involves:

1. Utilizing a vast collection of vectors representing various pieces of information.
2. Generating a vector for the query or question at hand.
3. Comparing the query vector with the existing information-based vectors.
4. Selecting a set of similar words, paragraphs, or sentences, commonly referred to as k-nearest vectors/words/paragraphs.
5. Incorporating these contextual elements into the LLM prompt to inform and enhance its responses.️

Transcending traditional NLP methodologies involves seamlessly transitioning between text and vectors, thereby circumventing the need for programmatic or analytical processing. This approach enables rapid and accurate information retrieval. The technical complexities associated with this paradigm are multifaceted; however, they can be effectively managed through a simple download and minimal coding requirements, allowing for efficient text analysis via vectorization.

While this article primarily focuses on text-based applications, it is essential to note that the concept of vectorizing information is inherently generic. Recent advancements have enabled the embedding of various forms of data, including audio, video, and protein structures, into vectorized formats. Furthermore, multi-modal models have been developed, which can uniformly process and analyze all media types, thereby expanding the scope of this technology.️

The concept of retrieving context via text embedding is referred to as Basic Retrieval-Augmentation-Generation (RAG). This approach can be implemented in a relatively short code snippet and has been extensively modified and adapted by numerous developers. The storage and comparison of vectors have also given rise to an entire industry, with various vector databases and database systems that incorporate vector capabilities being widely available.

Most relational database vendors have integrated vector indexes or developed plugins to support this functionality. Furthermore, the NoSQL movement’s emphasis on multi-modal data stores (key-value, document, graph) has led to the incorporation of vectors and embeddings into these systems.

In conjunction with bot and agentic frameworks, the RAG mechanism has enabled the creation of numerous corporate bots and applications that allow users to access information bases via natural language. However, it has been observed that this approach does not function optimally, primarily due to the inherent ambiguity of words and sentences.

The term “apple,” for instance, can refer to a fruit, a corporate entity, or stock information, among other possibilities. Consequently, when retrieving information related to “apple” from a typical database, users may receive an excessive amount of data on both the fruit and the stock market. This has resulted in confusing responses from corporate bots, not to mention issues with hallucinations.️

The crux of the matter lies not in the contextual information provided to a LLM, but rather in the context of the query itself. To furnish an LLM with the requisite context, one must first comprehend the underlying intent behind the question. For instance, a query regarding the caloric content of an apple would necessitate an understanding that this pertains to fruit, whereas a request for pricing information likely relates to stock valuation.

This dichotomy underscores two primary concerns: how to store contextual information and how to retrieve it effectively. It is at this juncture that knowledge graphs and graph-based Retrieval Augmented Generation (RAG) come into play, offering a solution to the accuracy and context-related issues inherent in naive RAG approaches.

Studies and commercial entities have substantiated the subjective experience of bots providing inaccurate responses, often with vested interests tied to the promotion of graph databases. The efficacy of text-to-SQL, naive RAG, graph RAG, and their variants can be quantitatively measured, underscoring the commercial appeal of these technologies.

The utility of graphs in this context may seem counterintuitive at first, as one might expect vectorization to provide a suitable solution. However, attaching context to vectors proves challenging, as words like ‘apple’ assume unique vector representations independent of their contextual usage. This leads to a situation where the same word can be situated within multiple topic clusters, necessitating the capture of entities or concepts and the attachment of contextual information.

Graphs excel in this regard by capturing the affinity between entities while also highlighting the importance of context. The question of how to create knowledge graphs and interrogate them via LLMs is the central focus of this article.️

What is a knowledge graph?

The concept of a knowledge graph can be succinctly defined as an organizational framework for structured data, represented in a graph-like structure. This differs from relational databases, which implicitly create graphs but do not explicitly utilize them.

However, the notion of a knowledge graph is more complex and encompasses various knowledge cultures, including semantic stacks, property graphs, strongly typed graphs, ontologies, document stores, and others. These different approaches enable diverse methods for storing and organizing data as graphs, each with its own strengths and weaknesses.

Key elements that distinguish knowledge graphs from other data storage systems include:

• The concept of traversals: Knowledge graphs prioritize the traversal of data as a graph, returning results in a graph format. In contrast, relational databases store and query data as tables.
• Graph-specific metrics: Centrality, hops, connected components, and similar measures are unique to knowledge graphs and not found in key-value stores or document stores.

Additional concepts that are relevant to knowledge graphs include:

• Taxonomies and ontologies, which can be beneficial but also burdensome
• Hypergraphs, temporal graphs, geospatial graphs, and other specialized graph types
• The various methods for storing and scaling graph storage
• Querying speed and flexibility
• Combining storage dimensions, such as vectors, documents, and graphs️

The significance of various features and functionalities often varies depending on the specific requirements and objectives of a particular business, research initiative, or project. Database vendors typically emphasize the capabilities that are most relevant to their industry, while academic researchers focus on aspects that contribute to the advancement of knowledge or enhance their professional reputation.

Similarly, cloud service providers tend to highlight the scalability and speed of their offerings. However, when it comes to developing a proof-of-concept (POC) or minimum viable product (MVP), the need for sophisticated infrastructure is not always necessary.

In fact, a POC or MVP can be effectively developed without incurring significant expenses. The following section outlines some essential Python components that can be used to establish a solid graph-based reasoning and analytics (RAG) solution, thereby facilitating the creation of a functional POC at minimal cost.

For the purposes of this discussion, I will refer to a knowledge graph as a data structure comprising named entities and relationships between them. This definition is independent of how such a graph is stored or queried, and it provides a useful framework for exploring various operationalization and implementation strategies.️

Creating Knowledge Graphs

The recipe for ingesting data into a knowledge graph consists of the following steps:

1. Scraping: raw information, typically in the form of PDFs, is extracted and parsed into markdown format. Each raw source is referred to as a document.
2. Chunking: documents are subsequently divided into smaller components, known as chunks or nodes (although this terminology does not yet relate to graph structures).
3. Entity extraction: the named entities within each chunk, along with their relationships, are identified and extracted. This process involves consolidating duplicate entity references across multiple chunks.
4. Graph consolidation: the resulting entities and relationships form the foundation of the knowledge graph, with metadata retained for referencing original chunk and document sources.
5. Semantic consolidation: as new chunks are extracted, the knowledge graph expands accordingly, with descriptions truncated if necessary to maintain data integrity.
6. Vectorize everything: the nodes, edges, and chunks within the knowledge graph are then vectorized, enabling cross-referencing and tracking of information origins.
7. Variations on this process may necessitate community detection algorithms, such as the Louvain algorithm, which is discussed in further detail in the appendix.

The following sections will provide a detailed explanation of each step in this process.️

Querying Knowledge Graphs

Interactions

The initial introduction of a dataset into a system or process is a critical phase; however, a thorough examination of the subsequent long-term interactions and consequences is equally essential for a comprehensive understanding.️

Any ingestion will lead to errors and issues. The end-user or domain expert needs to be able to adjust and correct things in the KG. This can take many forms:

• entities and relationships can be deleted
• entities can be added or existing ones adorned (change the description, add keyword)
• chunks might not contain relevant information
• weights can be altered to increate the importance of some relationships.

Each of these actions should update the vector storage, the KG and the chunk/document store:

• when a description is changed, the entity/relationship should be vectorized again
• if a chunk is removed or changed, all stores should get updated

and so on. The central idea here is data integrity or change propagation. On a code level this means that various bits used by the ingestion should be triggered in other ways as well. The CRUD (create/read/update/delete) has to be in place on all levels: document, chunk, node and relationship. Once again, if you understand conceptually how graph RAG hangs together it’s just a lot of coding and unit testing. I mean, it’s not algorithmically demanding, just correct plumbing and wiring of methods.

Minimal setup

In this section I describe a typical POC/MVP, running fully local.

You need three types of data storage:

• for chunks and raw documents (text) you can use JSON files or, more advance, some document database like MongoDB.
• for vectors the ChromaDB is the easiest of them all but any of the ones listed on Wikipedia will do
• for the knowledge graph you can use NetworkX and a file (e.g. GraphML) and a more advanced setup would entail Memgraph, Neo4j, AWS Neptune and many more.

A minimal setup based on files is obviously not going to scale well but can go a long way. You can combine everything in one and the same storage. For instance, Neo4j can store vectors graphs and text at the same time. There are also some open source options like Unum or UStore.

The storage and querying part is straightforward, but depends on the underlying API. You can also explore the wrappers of LlamaIndex and LangChain for inspiration.

Tokenizing can be based on Tiktoken or the Huggingface tokenizer:

ENCODER = tiktoken.encoding_for_model(settings.tokenize_model)
tokens = ENCODER.encode(content)

Chunking of documents can be done via tokenizing by taking a fixed amount of tokens each time, or you can use something smarter like semantic chunking. The size of a chunk and what works best depends on the type of data. You don’t need a fixed chunk size, using an amount of sentences can work too. Some advocate the need to have overlapping chunks, with a certain amount of tokens in adjacent chunks.

The most basic fixed token length approach looks like this:

tokens = encode(content)
if len(tokens) <= max_token_size:
    return content
else:
    return decode(tokens[:max_token_size])

The original documents need to be stored with a unique identifier and each chunk has to point to this unique identifier so that grounding and showing references can be done with each request.

Once chunking is done you need to feed each chunk to the ingestion process:

• the entities are extracted with some relations
• every chunk produces a little graph (it can be a single entity, nothing at all or a small graph with a dozen nodes and edges)
• this little graph has to be merged into the global KG

In addition: - every entity and relationship is vectorized - every chunk is vectorized - all the entities and relationships need to point to the chunk where they were found.

The most difficult part here is how to extract this ‘little graph’ from a chunk. As mentioned earlier, the magic resides in a sophisticated prompt. In essence, you ask very explicitly and by giving examples to the LLM what you need. This can look something like this:

--Goal--

You get a piece of text and a list of entity types. Identify all entities of those types from the text and all relationships among the identified entities.

--Steps--
1. Identify the entities:
- the entity name
- the entity type
- a description of this entity
Format each entity as follows:
(entity name, entity type, entity description)
2. From the entities in step 1 identify all pairs that are related.
For each pair (source_entity, target_entity) you extract the following information:
- source_entity: the name as identified in step 1
- target_entity: the name as identified in step 1
- description: a description of this relationship
3. Identify high-level keywords that summarize the essence of the given text.

--Examples--
<give concrete example here>

--Given text--
<inject the chunk here>

The magic sauce here is the neatly structured format and the very explicit guidelines and examples.

There are various models which focus on named entity extraction but they typically will not generate the high-level keywords and the description. These are, however, essential for the querying part and for the vectorization. If you leave out these you will generate a typically triple store but the incremental aspect will be much harder.

What happens when the same entity appears in multiple places with a different description? This is the smart merging part. The LLM is used once again to create a comprehensive merge of the description. This idea also applies to the description and keywords on relationships. The trickier part is to identify an entity within a context. For example, if two paragraphs contain the word ‘apple’ and one refers to the company while the other refer to fruit, the merged description will not make sense. There are a few ways to proceed: - you can ask the LLM very clearly whether the two entities with the given descriptions refer to the same thing. - you analytically look at the similarity of the node’s neighborhood - you can use the neighborhood as context to the question for the LLM.

If the LLM thinks the two are the same you can proceed and if not you need to store the entity separately. This could mean that the unique key for a node in the KG is not the name but the combination (name, type) or possibly more. Whether this is necessary depends on the knowledge domain. For example, technical terms like ‘cyanobacteria’ or ‘synechococcus’ are unlikely to refer to more than one thing, but names like ‘phoenix’ or ‘apple’ lead to disambiguation modeling.

The little chunk graphs can be merge in the KG and in addition: - every entity is vectorized together with the description - every relationship is vectorized together with the description and the keywords

All of this can fit in a single Python file. A scalable solution would entail some queues and separate processes. A sophisticated implementation is TrustGraph for instance. A codebase like TrustGraph contains a lot of plumbing code related to Apache Pulsar and other things, meaning that the essence of a graph RAG pipeline is hard to discover. It helps tremendously to have a minimal solution as a map to develop something like TrustGraph.

In any case, at this point the KG is ready to be queried. As mentioned above there are four types and each type has been described. The implementation is quite straightforward, if you understand conceptually what needs to be assembled.

For instance, the local query requires: - fetching the most important nodes via the vector databases - collecting for these nodes the relationships attached (edge neighbors) - collecting the unique chunks - sorting the chunks on the basis of some order, typically the edge degree - using these chunks as context for the question.

It means you need to use the NetworkX API, the ChromaDB API and so on. The fact that these API are very plain makes it easy to see what is graph RAG specific and what is storage specific. Here again, if you use Neo4j, Pinecone, MongoDB and whatnot, you likely will get lost in API’s. Start simple and grow from there.

Appendix

Simplistic

The most basic version of label propagation could be like this:

import networkx as nx
import numpy as np

def label_propagation(G, labels, max_iter=100):
    nodes = list(G.nodes())
    for _ in range(max_iter):
        new_labels = labels.copy()
        for node in nodes:
            if labels[node] is None:
                neighbor_labels = [labels[neighbor] for neighbor in G.neighbors(node) if labels[neighbor] is not None]
                if neighbor_labels:
                    new_labels[node] = max(set(neighbor_labels), key=neighbor_labels.count)
        if new_labels == labels:
            break
        labels = new_labels
    return labels

If we apply this to the karate graph and assign None to all nodes except to number 33 and 0:

G = nx.karate_club_graph()
initial_labels = {node: None for node in G.nodes()}
initial_labels[0] = 0  # Label node 0 as class 0
initial_labels[33] = 1  # Label node 33 as class 1

final_labels = label_propagation(G, initial_labels)
print(final_labels)

{0: 0, 1: 0, 2: 0, 3: 0, 4: 0, 5: 0, 6: 0, 7: 0, 8: 0, 9: 1, 10: 0, 11: 0, 12: 0, 13: 0, 14: 1, 15: 1, 16: 0, 17: 0, 18: 1, 19: 0, 20: 1, 21: 0, 22: 1, 23: 1, 24: 0, 25: 0, 26: 1, 27: 1, 28: 1, 29: 1, 30: 1, 31: 0, 32: 1, 33: 1}

This can be visualized for instance with PyVis:

from pyvis import network as net
g=net.Network(notebook=True, cdn_resources='in_line')
g.from_nx(G)
def set_options(net_g):
    options = """
var options = {
   "configure": {
        "enabled": true
   },
  "edges": {
    "color": {
      "inherit": false
    },
    "smooth": true
  },
  "physics": {
    "barnesHut": {
      "gravitationalConstant": -3000.0
    },
    "minVelocity": 0.75
  }
}
"""
    net_g.set_options(options) 


set_options(g)
g.show("simple.html")

simple.html

By setting the size of the initial propagating nodes we get:

g.nodes[0]["color"]="#00ff00"
g.nodes[0]["size"]=20
g.nodes[33]["color"]="#ff0000"
g.nodes[33]["size"]=20
for i in final_labels:
    if final_labels[i]==1:
        g.nodes[i]["color"]="#ff0000"
    else:
        g.nodes[i]["color"]="#00ff00"
    g.nodes[i]["title"]=str(i)
g.show("example.html")  

example.html

You can see that this simplistic propagation algorithm ain’t very good. There are some nodes with seem to be isolated.

Label propagation

A more rigorous approach can be seen in Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propagation. Technical Report CMU-CALD-02-107, Carnegie Mellon University, 2002 and the algorithm there can be summed up as:

with a graph where initially some nodes are labelled and the propagation assigns labels via diffusion.

Below you can find a label spreading algorithm and we’ll base both algorithm on the same base class:

from abc import abstractmethod
import torch

class BaseLabelPropagation:   
    def __init__(self, adj_matrix):
        self.norm_adj_matrix = self._normalize(adj_matrix)
        self.n_nodes = adj_matrix.size(0)
        self.one_hot_labels = None 
        self.n_classes = None
        self.labeled_mask = None
        self.predictions = None

    @staticmethod
    @abstractmethod
    def _normalize(adj_matrix):
        raise NotImplementedError("_normalize must be implemented")

    @abstractmethod
    def _propagate(self):
        raise NotImplementedError("_propagate must be implemented")

    def _one_hot_encode(self, labels):
        # Get the number of classes
        classes = torch.unique(labels)
        classes = classes[classes != -1]
        self.n_classes = classes.size(0)

        # One-hot encode labeled data instances and zero rows corresponding to unlabeled instances
        unlabeled_mask = (labels == -1)
        labels = labels.clone()  # defensive copying
        labels[unlabeled_mask] = 0
        self.one_hot_labels = torch.zeros((self.n_nodes, self.n_classes), dtype=torch.float)
        self.one_hot_labels = self.one_hot_labels.scatter(1, labels.unsqueeze(1), 1)
        self.one_hot_labels[unlabeled_mask, 0] = 0

        self.labeled_mask = ~unlabeled_mask

    def fit(self, labels, max_iter, tol):
        """Fits a semi-supervised learning label propagation model.
        
        labels: torch.LongTensor
            Tensor of size n_nodes indicating the class number of each node.
            Unlabeled nodes are denoted with -1.
        max_iter: int
            Maximum number of iterations allowed.
        tol: float
            Convergence tolerance: threshold to consider the system at steady state.
        """
        self._one_hot_encode(labels)

        self.predictions = self.one_hot_labels.clone()
        prev_predictions = torch.zeros((self.n_nodes, self.n_classes), dtype=torch.float)

        for i in range(max_iter):
            # Stop iterations if the system is considered at a steady state
            variation = torch.abs(self.predictions - prev_predictions).sum().item()
            
            if variation < tol:
                print(f"The method stopped after {i} iterations, variation={variation:.4f}.")
                break

            prev_predictions = self.predictions
            self._propagate()

    def predict(self):
        return self.predictions

    def predict_classes(self):
        return self.predictions.max(dim=1).indices

The Zhu and Ghahramani algorithm is now:

class LabelPropagation(BaseLabelPropagation):
    def __init__(self, adj_matrix):
        super().__init__(adj_matrix)

    @staticmethod
    def _normalize(adj_matrix):
        """Computes D^-1 * W"""
        degs = adj_matrix.sum(dim=1)
        degs[degs == 0] = 1  # divide by zero
        return adj_matrix / degs[:, None]

    def _propagate(self):
        self.predictions = torch.matmul(self.norm_adj_matrix, self.predictions)

        # Put back already known labels
        self.predictions[self.labeled_mask] = self.one_hot_labels[self.labeled_mask]

    def fit(self, labels, max_iter=1000, tol=1e-3):
        super().fit(labels, max_iter, tol)

Label spreading

The Label Spreading algorithm from Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston, Bernhard Schoelkopf. Learning with local and global consistency (2004) is as follows:

Note that the Laplacian is used here rather than the odd matrix above. The appearance of the Laplacian mimics closer the continuous diffusion equation.

class LabelSpreading(BaseLabelPropagation):
    def __init__(self, adj_matrix):
        super().__init__(adj_matrix)
        self.alpha = None

    @staticmethod
    def _normalize(adj_matrix):
        """Computes D^-1/2 * W * D^-1/2"""
        degs = adj_matrix.sum(dim=1)
        norm = torch.pow(degs, -0.5)
        norm[torch.isinf(norm)] = 1
        return adj_matrix * norm[:, None] * norm[None, :]

    def _propagate(self):
        self.predictions = (
            self.alpha * torch.matmul(self.norm_adj_matrix, self.predictions)
            + (1 - self.alpha) * self.one_hot_labels
        )
    
    def fit(self, labels, max_iter=3000, tol=1e-5, alpha=0.5):
        """
        Parameters
        ----------
        alpha: float
            Clamping factor.
        """
        self.alpha = alpha
        super().fit(labels, max_iter, tol)

Testing models on synthetic data

Let’s apply these two propagations to our karate graph:

G = nx.karate_club_graph()
adj_matrix = nx.adjacency_matrix(G).toarray()
# Create labels
labels = np.full(len(G.nodes), -1.)
labels[0]=0
labels[33]=1

adj_matrix_t = torch.FloatTensor(adj_matrix)
labels_t = torch.LongTensor(labels)

# Learn with Label Propagation
label_propagation = LabelPropagation(adj_matrix_t)
print("Label Propagation: ", end="")
label_propagation.fit(labels_t)
label_propagation_output_labels = label_propagation.predict_classes()

# Learn with Label Spreading
label_spreading = LabelSpreading(adj_matrix_t)
print("Label Spreading: ", end="")
label_spreading.fit(labels_t, alpha=0.8)
label_spreading_output_labels = label_spreading.predict_classes()

Label Propagation: The method stopped after 39 iterations, variation=0.0008.
Label Spreading: The method stopped after 31 iterations, variation=0.0000.

The label propagation can be seen via this rendering:

prop_labels = label_propagation_output_labels.numpy()
g=net.Network(notebook=True, cdn_resources='in_line')
g.from_nx(G)
set_options(g)
g.nodes[0]["color"]="#00ff00"
g.nodes[0]["size"]=20
g.nodes[33]["color"]="#ff0000"
g.nodes[33]["size"]=20
for i,v in enumerate(prop_labels):
    if prop_labels[i]==1:
        g.nodes[i]["color"]="#ff0000"
    else:
        g.nodes[i]["color"]="#00ff00"
    g.nodes[i]["title"]=str(i)
g.show("propagation.html") 

propagation.html

The label spreading gives:

spread_labels = label_spreading_output_labels.numpy()
g=net.Network(notebook=True, cdn_resources='in_line')
g.from_nx(G)
set_options(g)
g.nodes[0]["color"]="#00ff00"
g.nodes[0]["size"]=20
g.nodes[33]["color"]="#ff0000"
g.nodes[33]["size"]=20
for i,v in enumerate(spread_labels):
    if spread_labels[i]==1:
        g.nodes[i]["color"]="#ff0000"
    else:
        g.nodes[i]["color"]="#00ff00"
    g.nodes[i]["title"]=str(i)
g.show("spreading.html") 

spreading.html

If you apply these to the caveman graph where the propagation reflect the community topology:

import pandas as pd
import numpy as np
import networkx as nx


# Create caveman graph
n_cliques = 4
size_cliques = 10
caveman_graph = nx.connected_caveman_graph(n_cliques, size_cliques)
adj_matrix = nx.adjacency_matrix(caveman_graph).toarray()

# Create labels
labels = np.full(n_cliques * size_cliques, -1.)

# Only one node per clique is labeled. Each clique belongs to a different class.
labels[0] = 0
labels[size_cliques] = 1
labels[size_cliques * 2] = 2
labels[size_cliques * 3] = 3

# Create input tensors
adj_matrix_t = torch.FloatTensor(adj_matrix)
labels_t = torch.LongTensor(labels)

# Learn with Label Propagation
label_propagation = LabelPropagation(adj_matrix_t)
print("Label Propagation: ", end="")
label_propagation.fit(labels_t)
label_propagation_output_labels = label_propagation.predict_classes()

# Learn with Label Spreading
label_spreading = LabelSpreading(adj_matrix_t)
print("Label Spreading: ", end="")
label_spreading.fit(labels_t, alpha=0.8)
label_spreading_output_labels = label_spreading.predict_classes()

Label Propagation: The method stopped after 73 iterations, variation=0.0010.
Label Spreading: The method stopped after 39 iterations, variation=0.0000.

spread_labels = label_spreading_output_labels.numpy()
g=net.Network(notebook=True, cdn_resources='in_line')
g.from_nx(caveman_graph)
set_options(g)
g.nodes[0]["color"]="#00ff00"
g.nodes[0]["size"]=20
g.nodes[33]["color"]="#ff0000"
g.nodes[33]["size"]=20
color_map=["#ff0000","#00ff00","#0000ff","#ff00ff"]
for i,v in enumerate(spread_labels):
    g.nodes[i]["color"]=color_map[spread_labels[i]]
    g.nodes[i]["title"]=str(i)
g.show("caveman.html") 

caveman.html

Aside from the three anomalies this gives indeed a good result.

Arithmetic

Another fun example of LSTM is simple additions, it’s an example we also assembled in Wolfram. It’s interesting to note that the LSTM layer is sufficient, nothing dense, dropout or anything.

import numpy as np
from keras.models import Sequential
from keras.layers import LSTM, Dense

# Generate data for addition
def generate_addition_data(num_samples, max_value):
  """Generates data for addition task."""
  X = np.random.randint(0, max_value + 1, size=(num_samples, 2))
  y = np.sum(X, axis=1)
  return X, y

# Generate training and testing data
num_samples = 10000
max_value = 100
X_train, y_train = generate_addition_data(num_samples, max_value)
X_test, y_test = generate_addition_data(num_samples // 10, max_value)

# Reshape data for LSTM
X_train = X_train.reshape(X_train.shape[0], X_train.shape[1], 1)
X_test = X_test.reshape(X_test.shape[0], X_test.shape[1], 1)

# Create LSTM model
model = Sequential()
model.add(LSTM(32, input_shape=(X_train.shape[1], X_train.shape[2])))
model.add(Dense(1))

# Compile the model
model.compile(loss='mean_squared_error', optimizer='adam')

# Train the model
model.fit(X_train, y_train, epochs=100, batch_size=32, validation_split=0.2)

# Evaluate the model
loss = model.evaluate(X_test, y_test)
print('Test Loss:', loss)

# Make predictions
predictions = model.predict(X_test)
print('Predictions:', predictions[:10])
print('Actual values:', y_test[:10])

Epoch 1/100

 15/250 ━━━━━━━━━━━━━━━━━━━━ 1s 8ms/step - loss: 11907.7441

250/250 ━━━━━━━━━━━━━━━━━━━━ 2s 9ms/step - loss: 11319.8467 - val_loss: 9217.2393

Epoch 2/100

250/250 ━━━━━━━━━━━━━━━━━━━━ 2s 9ms/step - loss: 8795.1885 - val_loss: 7656.9336

Epoch 3/100

250/250 ━━━━━━━━━━━━━━━━━━━━ 3s 11ms/step - loss: 7350.7256 - val_loss: 6516.9985

...

Predictions: [[ 95.97442 ]

 [103.975876]

 [ 47.84532 ]

 [144.85732 ]

 [118.88151 ]

 [157.88887 ]

 [ 89.905106]

 [ 67.921936]

 [ 47.886024]

 [ 38.99708 ]]

Actual values: [ 96 104  48 145 119 158  90  68  48  39]

NetworkX

Download and unzip, say in ~/data/cora/:

import os
import networkx as nx
import pandas as pd
data_dir = os.path.expanduser("~/data/cora")

import the edges:

edgelist = pd.read_csv(os.path.join(data_dir, "cora.cites"), sep='\t', header=None, names=["target", "source"])
edgelist["label"] = "cites"

The edgelist is a simple table with the source citing the target. All edges have the same label:

edgelist.sample(frac=1).head(5)

Create a NetworkX graph from thie edge-list:

Gnx = nx.from_pandas_edgelist(edgelist, edge_attr="label")
nx.set_node_attributes(Gnx, "paper", "label")

A typical node looks like

Gnx.nodes[1103985]

{'label': 'paper'}

The data attached to the nodes consists of flags indicating whether a word in a 1433-long dictionary is present or not:

feature_names = ["w_{}".format(ii) for ii in range(1433)]
column_names =  feature_names + ["subject"]
node_data = pd.read_csv(os.path.join(data_dir, "cora.content"), sep='\t', header=None, names=column_names)

Each node has a subject and 1433 other flags corresponding to word occurence:

node_data.head(5)

The different subjects are:

set(node_data["subject"])

{'Case_Based',
 'Genetic_Algorithms',
 'Neural_Networks',
 'Probabilistic_Methods',
 'Reinforcement_Learning',
 'Rule_Learning',
 'Theory'}

A typical ML challenges with this dataset in mind:

• label prediction: predict the subject of a paper (node) on the basis of the surrounding node data and the structure of the graph
• edge prediction: given node data, can one predict the papers that should be cited?

You will find on this site plenty of articles which are based on the Cora dataset.

For your information, the visualization above was created via an export of the Cora network to GML (Graph Markup Language), an import into yEd and a balloon layout. It shows some interesting characteristics which can best be analyzed via centrality.

Wolfram

The TSV-formatted datasets linked above are easily loaded into Mathematica and it’s also a lot of fun to apply the black-box machine learning functionality on Cora. As an example, you can find in this gist an edge-prediction model based on node-content and adjacency. The model is poor (accuracy around 70%) but has potential, especially considering how easy it is to experiment within Mathematica.

The Wofram Cora experiment notebook

Pytorch

PyTorch Geometric has various graph datasets and it’s straightforward to download Cora:

from torch_geometric.datasets import Planetoid
dataset = Planetoid(root='~/somewhere/Cora', name='Cora')

data = dataset[0]
print(f'Dataset: {dataset}:')
print('======================')
print(f'Number of graphs: {len(dataset)}')
print(f'Number of features: {dataset.num_features}')
print(f'Number of classes: {dataset.num_classes}')

print(f'Number of nodes: {data.num_nodes}')
print(f'Number of edges: {data.num_edges}')
print(f'Average node degree: {data.num_edges / data.num_nodes:.2f}')
print(f'Number of training nodes: {data.train_mask.sum()}')
print(f'Training node label rate: {int(data.train_mask.sum()) / data.num_nodes:.2f}')
print(f'Contains isolated nodes: {data.contains_isolated_nodes()}')
print(f'Contains self-loops: {data.contains_self_loops()}')
print(f'Is undirected: {data.is_undirected()}')

Dataset: Cora():
======================
Number of graphs: 1
Number of features: 1433
Number of classes: 7
Number of nodes: 2708
Number of edges: 10556
Average node degree: 3.90
Number of training nodes: 140
Training node label rate: 0.05
Contains isolated nodes: False
Contains self-loops: False
Is undirected: True

/Users/swa/conda/envs/pyg/lib/python3.10/site-packages/torch_geometric/data/dataset.py:238: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  if osp.exists(f) and torch.load(f) != _repr(self.pre_transform):
/Users/swa/conda/envs/pyg/lib/python3.10/site-packages/torch_geometric/data/dataset.py:246: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
  if osp.exists(f) and torch.load(f) != _repr(self.pre_filter):

You can also convert the dataset to NetworkX and use the drawing but the resulting picture is not really pretty.

import networkx as nx
import matplotlib.pyplot as plt

from torch_geometric.utils import to_networkx
G = to_networkx(data, to_undirected=True)
nx.draw_networkx(G, with_labels=False)
plt.show()

PyG has lots of interesting datasets, both from a ML point of view and from a visualization point of view. Below is generic approach to download the data and convert it to NetworkX and GML.

from torch_geometric.datasets import Amazon
import numpy as np
from scipy.sparse import coo_matrix
import networkx as nx
# the name can only be 'Computers' or 'Photo' in this case.
amazon = Amazon(root='~/Amazon', name="Computers");
edges = amazon.data["edge_index"];
row = edges[0].numpy();
column = edges[1].numpy();
data = np.repeat(1, edges.shape[1]);
adj = coo_matrix((data, (row, column)));
graph = nx.from_scipy_sparse_matrix(adj);
# nx.write_gexf(graph, "//graph.gml")

The COO format referred to is a way to store sparse matrices, see the SciPy documentation. As outline below, from here on you can use various tools to visualize the graph.

Neo4j

Download the two CSV files (nodes and edges) and put them in the import directory of the database (see screenshot). They can’t be in any other directory since Neo4j will not access files outside its scope.

Run the queries:

LOAD CSV WITH HEADERS FROM 'file:///nodes.csv' AS row
Create (n:Paper{id:row["nodeId"], subject:row["subject"], features:row["features"]});

LOAD CSV WITH HEADERS FROM 'file:///edges.csv' AS row
Match (u:Paper{id:row["sourceNodeId"]})
Match (v:Paper{id:row["targetNodeId"]})
Create (u)-[:Cites]->(v);

This creates something like 2708 nodes and 10556 edges. You can also directly download this Neo4j v5.2 database dump if you prefer.

Visualization

You can easily visualize the dataset with various tools. The easiest way is via the yEd Live by opening this GraphML file. There is also the desktop version of yEd.

If you wish to reproduce the layout shown, use the balloon layout and use the settings shown below

The Gephi app is a popular (free) app to explore graphs but offers fewer graph layout algorithms. Use this Gephi file or import the aforementioned GraphML file.

Finally, there is Cytoscape and if you download the yFiles layout algorithms for Cytoscape you can create beautiful visualizations with little effort.

1. Definition and Scope

• Graph Analytics:
  • Graph analytics refers to a set of techniques and algorithms used to analyze graph structures. It focuses on understanding the (topological) properties and behavior of graphs, such as the relationships between nodes, the overall structure, and specific patterns within the graph. Common graph analytics tasks include calculating centrality measures, detecting communities, finding shortest paths, and identifying cliques or connected components.
  • Graph analytics often involves predefined algorithms that are applied to a graph to extract insights or solve specific problems. The techniques are usually more deterministic and involve direct computation on the graph’s structure.
• Graph Machine Learning (GML):
  • Graph machine learning involves the application of machine learning models to graph-structured data. GML aims to learn patterns and make predictions based on the data represented in a graph. This might include predicting properties of nodes, edges, or entire subgraphs, generating embeddings for nodes or graphs, and identifying new connections within the graph.
  • Graph ML is more dynamic and involves training machine learning models on (a lot of) graph data to generalize from patterns in the data, enabling predictions or classifications that go beyond the capabilities of traditional graph analytics.

[!Summary] In essence, analytics is descriptive and deterministic. You can apply it to a graph or a set of graphs of any size. Graph ML is predictive and probabilistic. You need big data to create ML models.

2. Techniques and Algorithms

• Graph Analytics:
  • Centrality Measures: Algorithms like PageRank, degree centrality, betweenness centrality, and eigenvector centrality are used to identify the most important nodes within a graph. Centrality is all about describing which are the important nodes and edges in a graph. The definition of ‘important’ leads to different notions of centrality.
  • Community Detection: Algorithms such as modularity optimization, Louvain method, and spectral clustering are used to detect clusters or communities within a graph.
  • Pathfinding: Algorithms like Dijkstra’s and A* are used to find the shortest paths between nodes.
  • Subgraph Matching: Techniques for identifying specific patterns or motifs within a graph, such as triangles, stars, or more complex structures.
• Graph Machine Learning:
  • Graph Neural Networks (GNNs): A class of neural networks designed to operate on graph-structured data, allowing for tasks like node classification, link prediction, and graph classification.
  • Node/Edge/Graph Embeddings: Techniques like DeepWalk, Node2Vec, and GraphSAGE that learn low-dimensional vector representations of nodes, edges, or entire graphs, preserving the structure and properties of the graph.
  • Supervised and Unsupervised Learning: GML applies both supervised (with labeled data) and unsupervised learning (without labels) techniques to graphs, learning patterns and making predictions based on the graph’s structure.

Note that graph machine learning goes by various names:

• graph machine learning
• geometric deep learning
• graph neural networks (GNN)
• network embedding
• graph representation learning
• machine learning on graphs
• graph embeddings.

Graph analytics can be done in-memory with frameworks like NetworkX or iGraph. If you have a large amount you can use a graph database and vendors have custom implementations. Neo4j’s GDS has lots of graph analytics and Memgraph has strong support for NetworkX.

Graph machine learning requires special GPU/CUDA frameworks like PyTorch or DGL. Like all data science efforts, it’s hard work and comes with a lot of experimenting and it requires skills and experience to design a ML model. Graph analytics is much more straightforward, in general.

3. Goals and Applications

• Graph Analytics:
  • The primary goal is to explore and understand the structure and properties of the graph. It helps in answering questions like “Who are the key influencers in a social network?” or “What is the shortest path between two points in a transportation network?”
  • Applications include network optimization, fraud detection, social network analysis, and infrastructure management, where the main focus is on interpreting the existing structure of the graph.
• Graph Machine Learning:
  • The goal of GML is to learn from graph data to make predictions, classifications, or generate embeddings that can be used in downstream tasks. This could involve predicting future interactions in a network, classifying nodes based on their features and connections, or even generating new graph structures.
  • Applications include recommendation systems, drug discovery (predicting molecular interactions), predictive maintenance (predicting failures based on equipment graphs), and any scenario where the goal is to predict unknown information based on the graph’s structure and existing data.

4. Complexity and Flexibility

• Graph Analytics:
  • Generally involves straightforward, deterministic algorithms. The complexity depends on the size of the graph and the specific algorithms used but is often bounded by the need to directly compute on the graph’s structure.
  • Less flexible in adapting to new patterns or unseen data since the analysis is usually based on predefined metrics and algorithms.
• Graph Machine Learning:
  • More complex, as it involves training models that learn from data. The complexity also comes from the need to process graph data in a way that machine learning models can understand (e.g., through embeddings or GNNs).
  • Highly flexible and adaptable to new data, allowing models to generalize and make predictions even on unseen parts of the graph.

5. Output

• Graph Analytics:
  • Outputs are usually descriptive metrics or patterns. For example, you might get a list of nodes ranked by their centrality, clusters of nodes that form communities, or a visualization of the shortest path between nodes.
• Graph Machine Learning:
  • Outputs are predictive in nature. A model embodies learned patterns. For example, predictions about the category of a node, probabilities of new edges forming between nodes, or learned embeddings that can be used in further machine learning tasks.

SI model

The SI model is the simplest form of all disease models. Individuals are born into the simulation with no immunity (susceptible). Once infected and with no treatment, individuals stay infected and infectious throughout their life, and remain in contact with the susceptible population. This model matches the behavior of diseases like cytomegalovirus (CMV) or herpes.

In a continuous settings the model is defined via this set of coupled differential equations:

with the total population consisting of susceptible individuals and infected individuals. The factor is the diffusion rate leading to a logistic growth of the infected population.

This simple diffusion model is really straightforward on a graph. Given a connection from an infected node to a susceptible one, a random value above a threshold (the diffusion parameter) defines whether or not the susceptible node gets affected. This loop runs untile the whole network is reached and infected. It’s also obvious that in this model the whole population gets infected eventually and this corresponds to the logistic curve as seen below.

def SI(G, initial_infected:int=1, beta:float=0.01, N:int=2000, T:int=1000):
    """
        Simulates the Susceptible-Infected (SI) model on a given graph.
        Parameters:
        G (networkx.Graph): The input graph where nodes represent individuals.
        initial_infected (int): The initial number of infected individuals.
        HM (float): The probability of infection transmission between connected nodes.
        N (int): The total number of individuals in the graph.
        T (int): The number of time steps to simulate.
        Returns:
        tuple: A tuple containing three numpy arrays:
            - susceptible (numpy.ndarray): The number of susceptible individuals at each time step.
            - infected (numpy.ndarray): The number of infected individuals at each time step.
            - infection_delta (numpy.ndarray): The number of new infections at each time step.

    """
    
    susceptible = np.zeros(T) # how many susceptible at time t
    infected = np.zeros(T) # how many infected at time t
    infection_delta = np.zeros(T) # increase in infection compared to previous time
    infected[0] = initial_infected
    susceptible[0] = N - initial_infected
    infection_delta[0] = infected[0]

    for u in G.nodes():
        G.nodes[u]["infected"] = 0
        G.nodes[u]["neighbors"] = [n for n in G.neighbors(u)]
    init = random.sample(G.nodes(), initial_infected)
    for u in init:
        G.nodes[u]["infected"] = 1

    for t in range(1,T):
        susceptible[t] = susceptible[t-1]
        infected[t] = infected[t-1]
        for u in G.nodes:            
            if G.nodes[u]["infected"] == 0:
                # nb_friend_infected = [G.nodes[n]["infected"] == 1 for n in G.nodes[u]["neighbors"]].count(True)
                for n in G.nodes[u]["neighbors"]:
                    if G.nodes[n]["infected"] == 1: 
                        # with HM infect
                        if np.random.rand() < beta:
                            G.nodes[u]["infected"] = 1
                            infected[t] += 1
                            susceptible[t] += -1
                            break
        infection_delta[t] = infected[t]-infected[t-1]
    return susceptible, infected, infection_delta

This is a simple implementation without using adjacency matrices or vectorizations. It ain’t the most performant one but it shows best what is going on.

T = 100
N = 1500 
beta = 0.03
initial_infected = 2 

k = 20
G = nx.erdos_renyi_graph(N,k/N)
s_erdos, inf_erdos,infection_delta = SI(G,initial_infected,beta, N, T)
plt.plot((100/N)*s_erdos, color='b',marker='+', label="Susceptible p=0.04")
plt.plot((100/N)*inf_erdos, color='r',marker='o', label="Infected  p=0.04")
plt.xlabel("time")
plt.ylabel("Percentage of population infected")
plt.legend()
plt.show()

/var/folders/0p/dt9_dywj6rnc0wzps07dg31m0000gn/T/com.apple.shortcuts.mac-helper/ipykernel_47535/1712297490.py:28: DeprecationWarning: Sampling from a set deprecated
since Python 3.9 and will be removed in a subsequent version.
  init = random.sample(G.nodes(), initial_infected)

k = 3
T = 1000
G = nx.erdos_renyi_graph(N,k/N)
s_erdos, inf_erdos,infection_delta = SI(G,initial_infected,beta, N, T)
plt.plot((100/N)*s_erdos,"b",marker='+', label="Susceptible p=0.01")
plt.plot((100/N)*inf_erdos,"r",marker='o', label="Infected  p=0.01")

plt.xlabel("time")
plt.ylabel("% infected")
plt.legend()
plt.show()

/var/folders/0p/dt9_dywj6rnc0wzps07dg31m0000gn/T/com.apple.shortcuts.mac-helper/ipykernel_47535/1712297490.py:28: DeprecationWarning: Sampling from a set deprecated
since Python 3.9 and will be removed in a subsequent version.
  init = random.sample(G.nodes(), initial_infected)

k = 2 # the mean degree of the networks
# defining an erdos renyi network
G = nx.erdos_renyi_graph(N,k/N)
s_erdos, inf_erdos,infection_delta = SI(G,initial_infected,beta, N, T)
plt.plot(infection_delta,"r",marker='o', label="Infected  p=0.01")
plt.xlabel("time")
plt.ylabel("Number of new cases")
##

/var/folders/0p/dt9_dywj6rnc0wzps07dg31m0000gn/T/com.apple.shortcuts.mac-helper/ipykernel_47535/1712297490.py:28: DeprecationWarning: Sampling from a set deprecated
since Python 3.9 and will be removed in a subsequent version.
  init = random.sample(G.nodes(), initial_infected)

Text(0, 0.5, 'Number of new cases')

k = 20
N = 500
initial_infected = 2

G1 = nx.erdos_renyi_graph(N,k/N)
G1_demi = nx.erdos_renyi_graph(N,k/N *0.5)
G2 = nx.barabasi_albert_graph(N, k)
G2_demi = nx.barabasi_albert_graph(N, int(k*0.5))

s_ER, inf_ER,nb_inf_t_ER = SI(G1,initial_infected,beta, N, T)
s_ER_demi, inf_ER_demi,nb_inf_t_ER_demi = SI(G1_demi,initial_infected,beta, N, T)

s_BA, inf_BA,nb_inf_t_BA = SI(G2,initial_infected,beta, N, T)
s_BA_demi, inf_BA_demi,nb_inf_t_BA_demi = SI(G2_demi,initial_infected,beta, N, T)
plt.figure(figsize=(15,10))
plt.plot(nb_inf_t_ER,"r", linewidth=2, label="Network:EA Infected  p=0.04")
plt.plot(nb_inf_t_ER_demi,"r--", linewidth=2, label="Network:EA Infected  p=0.02")


plt.plot(nb_inf_t_BA,"b", linewidth=2, label="Network:BA Infected  p=0.04")
plt.plot(nb_inf_t_BA_demi,"b--", linewidth=2, label="Network:BA Infected  p=0.02")
plt.xlabel("time")
plt.ylabel("Number of new cases")
plt.xlim(0,60)

plt.legend()
plt.show()

/var/folders/0p/dt9_dywj6rnc0wzps07dg31m0000gn/T/com.apple.shortcuts.mac-helper/ipykernel_47535/1712297490.py:28: DeprecationWarning: Sampling from a set deprecated
since Python 3.9 and will be removed in a subsequent version.
  init = random.sample(G.nodes(), initial_infected)

SIR model

The SI model is not satisfactory because everyone gets infected eventually and it does not take into account recovery, survival, removal (death) and so on. The SIR model improves the SI by adding a number of removed (and immune) or deceased individuals, denoted by :

The coefficient corresponds to how fast an infection leads to deceased individuals.

def SIR(G, initial_infected, gamma, beta, N, T):
    
    susceptible = np.zeros(T)
    removed = np.zeros(T)
    infected = np.zeros(T)
    infected[0] = initial_infected
    susceptible[0] = N - initial_infected
    
    for u in G.nodes():
        G.nodes[u]["infected"] = 0
        G.nodes[u]["infectedDuration"] = 0
        G.nodes[u]["neighbors"] = [n for n in G.neighbors(u)]

    init = random.sample(G.nodes(), initial_infected)
    for u in init:
        G.nodes[u]["infected"] = 1
        G.nodes[u]["infectedDuration"] = 1
    # running simulation
    for t in range(1,T):
        susceptible[t] = susceptible[t-1]
        infected[t] = infected[t-1]
        removed[t] = removed[t-1]
        # Check which persons have recovered
        for u in G.nodes:
            # if infected
            if G.nodes[u]["infected"] == 1:
                if G.nodes[u]["infectedDuration"] < gamma:
                    G.nodes[u]["infectedDuration"] += 1
                else:
                    G.nodes[u]["infected"] = 2 #"recovered"
                    removed[t] += 1
                    infected[t] += -1
        # check contagion    
        for u in G.nodes:
            #if susceptible
            if G.nodes[u]["infected"] == 0:
                nb_friend_infected = [G.nodes[n]["infected"] == 1 for n in G.nodes[u]["neighbors"]].count(True)
                #print(nb_friend_infected)
                for n in G.nodes[u]["neighbors"]:
                    if G.nodes[n]["infected"] == 1: # if friend is infected
                        # with HM infect
                        if np.random.rand() < beta:
                            G.nodes[u]["infected"] = 1
                            infected[t] += 1
                            susceptible[t] += -1
                            break
    
    return susceptible, infected,removed

Once again, this isn’t the most performant implementation but demonstrative.

If we apply the algorithm an Erdos-Renyin graph:

np.random.seed(0)
# time of simulation
T = 100
# number of agents
N = 5000
beta = 0.03
gamma = 5
initial_infected = 2
# mean degree of the networks
k = 20
# defining an erdos renyi network
G = nx.erdos_renyi_graph(N,k/N)

s_erdos, inf_erdos,r_erdos = SIR(G,initial_infected,gamma,beta, N, T)
#plt.plot((100/N)*s_erdos, color='b',marker='+', label="Susceptible k=20")
plt.plot((100/N)*inf_erdos, color='r',marker='+', label="Infected p=0.04")
plt.plot((100/N)*r_erdos, color='g',marker='+', label="Recovered p=0.04")

#
# mean degree of the networks
k = 10
# defining an erdos renyi network
G = nx.erdos_renyi_graph(N,k/N)

s_erdos, inf_erdos,r_erdos = SIR(G,initial_infected,gamma,beta, N, T)
#plt.plot((100/N)*s_erdos,color="b",marker='o',label="Susceptible k=10")
plt.plot((100/N)*inf_erdos,color="r",marker='o',label="Infected p=0.02")
plt.plot((100/N)*r_erdos,color='g',marker='o',label="Recovered p=0.02")

plt.xlabel("time")
plt.ylabel("Percentage of population infected")
plt.legend()
plt.show()

/var/folders/0p/dt9_dywj6rnc0wzps07dg31m0000gn/T/com.apple.shortcuts.mac-helper/ipykernel_47535/935179267.py:16: DeprecationWarning: Sampling from a set deprecated
since Python 3.9 and will be removed in a subsequent version.
  init = random.sample(G.nodes(), initial_infected)

These epidemiology models are not limited to flat spaces and it’s interesting to apply it to other topologies, like a torus. In the adjacent image we used Python inside Cinema 4D to simulate infection:

Sample Data

We’ll use a network originally from Neo4j (see this Github repo) and converted in NetworkX format here.

The schema is straightforward and the main objective is to consolidate users (label: User):

The data can be fetched from the pickle back into NetworkX like this:

import networkx as nx
import pickle
with open('../data/EntityResolution.pkl', 'rb') as f:
    G = pickle.load(f)
print(G)

MultiDiGraph with 1237 nodes and 1819 edges

The data in the nodes consists of the usual suspects:

for id in G.nodes():
    print(G.nodes[id])
    break

{'labels': ['User'], 'properties': {'lastName': 'Burbidge', 'country': 'US', 'firstName': 'Dorette', 'gender': 'Male', 'phone': '834-424-8856', 'state': 'Ohio', 'userId': 1, 'email': 'dburbidge0@japanpost.jp'}}

Properties

At mentioned above, common sense dictates to look at the properties to identify similarities. On a string level you have various options and Python has the built-in SequenceMatcher. Taking random pairs of User nodes and finding the similarity of the full names goes like this:

from difflib import SequenceMatcher
import random
# take the ids of the User labels
user_ids = [id for id in G.nodes() if "User" in G.nodes(data=True)[id]["labels"]]
pairs = [random.sample(user_ids , k=2) for _ in range(10)]
for pair in pairs:
    u = G.nodes(data=True)[pair[0]]["properties"]
    v = G.nodes(data=True)[pair[1]]["properties"]
    u_name = u["firstName"] + " " + u["lastName"]
    v_name = v["firstName"] + " " + v["lastName"]
    w = SequenceMatcher(None,u_name, v_name)
    print(u_name, v_name, w.ratio())

Nert Picopp Deni Jest 0.2
Konstanze Absolem Ronda Skilling 0.25806451612903225
Maxine Clousley Tarrah Caw 0.24
Port Prozescky Cacilie Nicklin 0.20689655172413793
Brana Cordeau Aurora Broadis 0.2962962962962963
Meryl Boncore Thedrick Dwine 0.37037037037037035
Riki Prujean Cece Brownstein 0.14814814814814814
Lynnelle Minkin Bar O'Hear 0.08
Carr Keemar Thedrick Dwine 0.24
Bobbe Wittering Maxine Francescone 0.24242424242424243

This is not a practical solution because all possible pairs quickly grows with the amount of nodes. There is also the issue of missing data and how to impute, a classic data science problem.

Topology

The most basic topological similarity is based on the idea that the more friends you share, the more likely you know each other. The more two nodes share the same nodes in their 1-hop neighborhood, the more they are similar. This is expressed in the Jaccard similarity and is easy to compute. We are only interested in the similarity between User nodes, so we pass the index as the second paramter. Note that the given tuples have nothing to do with edges, we are only specifying which couples are interesting:

G = nx.Graph(G) # Jaccard will not work with multi-graphs
import itertools
user_combinations = list(itertools.combinations(user_ids, 2)) # around 125K combinations
jaccard_all_sim = nx.jaccard_coefficient(G, user_combinations)
jaccard_user_sim = [(u,v,w) for (u,v,w) in jaccard_all_sim if w>0.9]
for (u,v,w) in jaccard_user_sim:
    user_u = G.nodes[u]["properties"]
    user_v = G.nodes[v]["properties"]
    print(user_u["firstName"] + " " + user_u["lastName"], user_v["firstName"] + " " + user_v["lastName"], w)

Shirlene Borres Forrester Borres 1.0
Ondrea Garnsey Nadine Garnsey 1.0
Wanda W Wanda Wroath 1.0

It’s important to stress here that the similarity is purely topological and does not depend on the properties of the nodes. That is, the method suggests that the nodes are similar because their neighborhood has some similarities.

In order to include the topology with the properties you need to step into graph machine learning and use node embeddings (node2vec).

Alternatively, you can use your heuristics together with the Jaccard coefficient to increase accuracy. Yet another road is to use classic machine learning and use the properties together with the topological similarity in your feature engineering.

• The Jaccard coefficient is available in Neo4j via GDS
• Neptune Analytics has Jaccard
• Memgraph calls it simply node similariy.

Feature Engineering

For data scientist this topic is obvious but experience with customers shows that it’s much less clear to most people. So, here’s a note on features and how extra info can be used.

No matter the ML technique you use (SVM, XGBoost, neural networks…), you have a table with columns containing the characteristics of stuff and a special column (the target) that you wish to predict. If you wish to train an ML model which learns the similarity of nodes you collect the relevant properties that each node carries and you add it as a column. Since you have two nodes, you have these columns twice. The target column is a boolean saying whether they are or they are not similar. There are plenty of variations on this theme but let’s keep it simple.

The more rows you give to the algorithm, the more it will learn the characteristics of ‘the same’ two things (nodes).

Feature engineering is in this respect all about using meaningful features, combining them and converting them in a format suitable for ML. The important bit is also that you can add whatever you think can contribute to the process of detecting patterns. With (property) graphs you can add the payload of the nodes and edges AND the topological information. Provided you can encode the topology in a numerical format.

The way a node sits in a graph, the immediate neighborhood (1-hop) or a bit more (N-hops) can be encoded in a vector via a technique called node embeddings. In essence, a vector captures the neighborhood (topology) and this augments the features with information that a ML algorithm can use.

Embeddings

The creation of vectors from nodes (typically abbreviated as node2vec) is a topic we’ll explain elsewhere but there are various easy to use libraries which hide the complexity of assembling these vectors. The best known one is Node2Vec (pip install node2vec) or a faster version fastnode2vec (pip install fastnode2vec) with linear memory usage.

To create an embedding you need to convert the graph to another type of graph (since fastnode2vec inherits from the Gensim framework), define the parameters and train things:

from fastnode2vec import Graph, Node2Vec
graph = Graph(list(G.edges()), directed=True, weighted=False)
n2v = Node2Vec(graph, dim=10, walk_length=2, window=10, p=2.0, q=0.5, workers=2)
n2v.train(epochs=100)

This happens in less than a second and you can fetch node vectors like so:

print(n2v.wv[user_ids[0]])

[ 0.00309572 -0.07105517  0.03536741 -0.00647308  0.0246973   0.01980644
 -0.10460257  0.01136639  0.05422755 -0.06904547]

With this in place we can start to use the vectors and the easiest thing is to use cosine similarity. Since the created vectors are Numpy arrays this is straightforward:

import numpy as np

def cosim(id_1, id_2):
    v1 = n2v.wv[id_1]
    v2 = n2v.wv[id_2]
    dot_product = np.dot(v1, v2)
    norm_a = np.linalg.norm(v1)
    norm_b = np.linalg.norm(v2)
    return dot_product / (norm_a * norm_b)

print(f"Cosine Similarity: {cosim(user_ids[1], user_ids[2]):.4f}")

Cosine Similarity: 0.4215

Cosine similarity is really a mathematical translation for how close two vectors are to one another. The closer the value is to 1.0 the more they are close. The vector embedding ensures that vectors are close if the their topologies are similar and so, if we find high cosine values it means the two nodes have similar neighborhoods.

Once again, we look at the user nodes:

user_combinations = list(itertools.combinations(user_ids, 2)) 
cosim_users = [(u,v, cosim(u,v)) for (u,v) in user_combinations if cosim(u,v) > 0.9 ]
print(len(cosim_users))

21

In decreasing similarity:

sorted_cosim_users = sorted(cosim_users, key=lambda x: x[2], reverse=True)

# Iterate through the sorted list
for (u, v, w) in sorted_cosim_users[:10]:
    user_u = G.nodes[u]["properties"]
    user_v = G.nodes[v]["properties"]
    print(user_u["firstName"] + " " + user_u["lastName"], user_v["firstName"] + " " + user_v["lastName"], w)

Wandis Blewett Kristian Twaits 0.96341246
Riki Prujean Huey Ferneyhough 0.95864064
Eal Grombridge Konstanze Absolem 0.9390747
Perceval Pilbury Berk Mauser 0.93898416
Morgan Mongenot Maurine Mesant 0.93397725
Tova Canton Christopher Cowmeadow 0.9336317
Robyn Haskett Christiana Goodey 0.9315399
Danyette Pinnigar Dalt Le Quesne 0.9289893
Gardiner Dudman Lexy Canton 0.92787033
Gerard Toy Loydie Regis 0.9258602

You might wonder what is the advantage over the Jaccard method above. Especially since both methods only take the topology into account and none of the properties.

The embedding is more general:

• it’s a ML trained model specifically on the graph
• the walk_length parameter reveals that underneath the hood there is a random walk collecting information about a node’s neighborhood. By increading the walk’s size you increase the size of the topology taken into account. The Jaccard method is 1-hop only. The additional paramters (p,q above) also are related to the random walk (how adventurous or exotic the walk is).
• the vector dimension allows one to be more sparse or dense in function of the graph and downstream tasks.

One could say that Jaccard is the simplistic measure and Node2Vec fits in the AI (neural network) space. Whether one is better than the other depends on your tech stack, team, amount of data and so on.