Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add hierarchical structure to the graph index #402

Open
wants to merge 127 commits into
base: main
Choose a base branch
from
Open

Add hierarchical structure to the graph index #402

wants to merge 127 commits into from
+2,649 −1,460

Conversation

marianotepper
Copy link
Collaborator

@marianotepper marianotepper commented Mar 13, 2025 •
edited
Loading

This PR introduces a hybrid HNSW-DiskANN graph. From HNSW, we take the idea of using multiple layers. This adds robustness to particularly hard datasets. Each layer is a Vamana graph. The upper layers reside in memory while the base layer resides on disk (in DiskANN style). It also enable using a single layer. In that case, it is a plain Vamana graph.

Jonathan Ellis and others added 30 commits February 7, 2025 17:21
degen dataset, tested in memory
8572202
extract IntMap, removing @deprecated put()
5bc9c85
SparseIntMap
2cf61be
move javadoc into IntMap
2907a1a
multilayer part 1
ae2fb7a
almost builds
b6b4bf0
r/m cache stuff so it builds
b1e864c
fixes
d174556
don't try to write to disk, it doesn't work yet
b17226a
fixes
98934c1
@marianotepper
Bypass the order check in NodeQueue.copyFrom when other.size == 1
2b8900a
@marianotepper
Search all levels when building the graph but only update the layers …
08f848f 
…where the node is present
tests build
b75000e
add support for copying from opposite-order queues
f2edbb5
need to update snapshot during construction to see other nodes that w…
1ef1010 
…ere added
fix copyFrom better
0114f5a
put back .parallel in build()
4ac9144
address TODOs in GIB. short edges computation is gone
7c167a7
fix TODOs in GraphIndex
6f54bd3
refactor getNodes and the underlying IntMap methods
d46d8e1
tests build
807a6c7
fix NodeArray overload of addNode to add layers if needed
ddf8ffa
need to reset visited between levels
7770ec1
copy the vector being searched for, this was a pre-existing bug in th…
7aafffb 
…e test
fix visited better and fix adding evicted notes from higher le
700bad1 
vels to candidates pool in layer 0
cleanup
7bc2072
r/m printlns
a5f8d6b
insertSorted ignores duplicates by design, not correct to assert that…
909b782 
… no dupes are submitted
acceptOrds should only be consulted in L0
66b807d
add multi-layer validateIndex
bdfbbcc
@marianotepper
Add javadocs to GraphIndex.java
57ebc52
@marianotepper
Update layerInfo to contain the correct size for L0
1db1448
@marianotepper
Add javadoc for GraphSearcher.usePruning
984cdf9
@marianotepper
Merge branch 'main' into hnsw-3
a617b29 
# Conflicts:
#	jvector-tests/src/test/java/io/github/jbellis/jvector/graph/Test2DThreshold.java
@marianotepper
Add javadoc for GraphSearcher.setView
485d8b1
@marianotepper
Update README with the new hierarchical structure.
55c98bd
@marianotepper
Update UPGRADING with the new hierarchical structure.
12e1dd5
@marianotepper
Add javadoc to NodeQueue.java
3ece48f
@marianotepper
Improve comments and javadoc in OnHeapGraphIndex.java
2f50810
@marianotepper
Switch IntStream.range by IntStream.rangeClosed in CommonHeader.fromG…
13b4d06 
…raph
@marianotepper
Add missing whitespace to OnDiskGraphIndex.getNodes
e70901e
@marianotepper
Switch IntStream.range by IntStream.rangeClosed in CommonHeader.fromG…
2cfa0f3 
…raph
@marianotepper
Fix degree check when writing L0 and throw an exception if a supplier…
71b8389 
… is null
@marianotepper
Remove blank line
940326e
@marianotepper
Remove duplicated line
7b495e6
@marianotepper
Uncomment test
491618b
@marianotepper
Fix bug with the search score provider in improveConnections
bec7ad4
@marianotepper
Update UPGRADING.md with the lack of support for varying degrees with…
ebcf32f 
… FusedADC
@marianotepper
Update comment in improveConnections
51d1f4d
@marianotepper
Correct the computation of the memory size in GraphIndexBuilder.remov…
0a37852 
…eDeletedNodes
@marianotepper
Read V4 and V3 graphs seamlessly
cda8ca6
@marianotepper
Remove TODO
9617d02
@marianotepper
Fix typo
7d0b8ab
@marianotepper
Remove duplicated line in TestADCGraphIndex.testFusedGraph
808a48a
@marianotepper
Test for graph equality in testSaveAndLoad
a30554a
@marianotepper
Uncomment saving and loading test in TestDeletions.testCleanup
0dbd5d5
@marianotepper
When loading a graph, we need to call markComplete after adding a nod…
fd85cf9 
…e in all the corresponding layers.
@marianotepper
In OnDiskGraphIndex, inMemoryNeighbors is now final. Replace snake by…
840ee98 
… camel casing
@marianotepper
Update poml.xml with the right revision id
703eef0
@marianotepper
Use other's parallelExecutor to copy the nodes in GraphIndexBuilder.r…
90bdc56 
…escore
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jkni jkni jkni requested changes

@tlwillke tlwillke Awaiting requested review from tlwillke

Requested changes must be addressed to merge this pull request.

Assignees

@marianotepper marianotepper

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@marianotepper @jkni