MB-65243: Merge 'facebookresearch/faiss@v1.10.0' into 'blevesearch/faiss@bleve' #46
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: Pull Request resolved: facebookresearch#3487 Sometimes it is not useful to serialize the storage index along with a HNSW index. This diff adds a flag that supports skipping the storage of the index. Searchign and adding to the index is not possible until a storage index is added back in. Reviewed By: junjieqi Differential Revision: D57911060 fbshipit-source-id: 5a4ceee4a8f53f6f746df59af3942b813a99c14f
…kresearch#3494) Summary: Pull Request resolved: facebookresearch#3494 This tasks focus on the refine index construction tutorial with different index refinement on fp16/sq8 quantization. The python version was added a while ago. Reviewed By: junjieqi Differential Revision: D58161983 fbshipit-source-id: 1c598fe612b5dee3952c5f7398e6802e117f141d
Summary: Adding build folder to gitignore, so that they don't show up in the commit tree while building from source Pull Request resolved: facebookresearch#3492 Reviewed By: junjieqi Differential Revision: D58171359 Pulled By: asadoughi fbshipit-source-id: b0efed348769328a3bdbcc13098dcb84cadb6c4f
Summary: Depends on facebookresearch#3084 Pull Request resolved: facebookresearch#3252 Reviewed By: junjieqi Differential Revision: D57971948 Pulled By: mdouze fbshipit-source-id: 4371f4d136eeceb59568593f98a6ae9163a768ba
Summary: Pull Request resolved: facebookresearch#3502 include probably added by vscode Reviewed By: mengdilin Differential Revision: D58411537 fbshipit-source-id: 3035f690d26decc937fb492c54ffa2f974ee2db8
Summary: Add support for `use_raft` in the torch version of `knn_gpu`. The numpy version already has this support, see https://github.com/facebookresearch/faiss/blob/main/faiss/python/gpu_wrappers.py#L59 Pull Request resolved: facebookresearch#3509 Reviewed By: mlomeli1, junjieqi Differential Revision: D58489851 Pulled By: algoriddle fbshipit-source-id: cfad722fefd4809b135b765d0d43587cfd782d0e
…rch#3512) Summary: Pull Request resolved: facebookresearch#3512 Pull Request resolved: facebookresearch#3510 GitHub hosted runners some with the build-essentials package pre-installed, self-hosted runners on AWS do not have this package. This made it all steps other than the `all targets` one fall back to the system executables which unintentially worked on GitHub hosted runners but not on the self-hosted ones. This diff fixes it by pulling the line that adds conda bin to path early in the cmake build action. Reviewed By: asadoughi Differential Revision: D58513853 fbshipit-source-id: 23e95459e0031c96bd142515db07d1b700d713cf
Summary: Pull Request resolved: facebookresearch#3515 Fix typo `test_io_no_storage` Reviewed By: kuarora, asadoughi Differential Revision: D58540190 fbshipit-source-id: b8b9cacd7ea6005c0edb94014de74188450318c1
…cebookresearch#3516) Summary: Pull Request resolved: facebookresearch#3516 This diff seeks to simplify the steps that install conda packages and environment configuration into a single step at the start of the cmake build action. Reviewed By: mnorris11 Differential Revision: D58560454 fbshipit-source-id: ee2c6b36865809f31eb335cfb3c2fffdccaa318d
Summary: Pull Request resolved: facebookresearch#3517 MSVC doesn't support unsigned int for OMP Reviewed By: kuarora, junjieqi, ramilbakhshyiev Differential Revision: D58591594 fbshipit-source-id: ac7d6b37a82f9543be3e0fe418f0f6b439751475
Summary: Pull Request resolved: facebookresearch#3519 Fixing the conda conflicts because of `_openmp_mutex` build versions. This change pins that version for RAFT conda package builds. Reviewed By: algoriddle Differential Revision: D58646659 fbshipit-source-id: 4c1eaa9f08bd354da016b9399a36698007a497d8
…h#3522) Summary: Pull Request resolved: facebookresearch#3522 Quick fix to unblock nightly Reviewed By: mlomeli1 Differential Revision: D58694193 fbshipit-source-id: ea323991cc2e2b958fc11ab614dcd6e09d4c072c
Summary: Pull Request resolved: facebookresearch#3524 Searches with the metric abs(dot(query, database)) This makes it possible to search vectors that are closest to a hyperplane * adds support for alternative metrics in faiss.knn in python * checks that it works with HNSW * simplifies the extra distances interface by removing the template on Reviewed By: asadoughi Differential Revision: D58695971 fbshipit-source-id: 2a0ff49c7f7ac2c005d85f141cc5de148081c9c4
Summary: Pull Request resolved: facebookresearch#3531 **In this diff** 1. I have add bench_fw to bento faiss kernel target 2. First part of notebook is changed to analyze sift1M results Reviewed By: algoriddle Differential Revision: D58823037 fbshipit-source-id: a67d4638af4368f0575bd289ce7aff8cf1fcd38b
…ookresearch#3527) Summary: Pull Request resolved: facebookresearch#3527 **Context** Design Doc: [Faiss Benchmarking](https://docs.google.com/document/d/1c7zziITa4RD6jZsbG9_yOgyRjWdyueldSPH6QdZzL98/edit) **In this diff** 1. Be able to reference codec and index from blobstore (bucket & path) outside the experiment 2. To support #1, naming is moved to descriptors. 3. Build index can be written as well. 4. You can run benchmark with train and then refer it in index built and then refer index built in knn search. Index serialization is optional. Although not yet exposed through index descriptor. 5. Benchmark can support index with different datasets sizes 6. Working with varying dataset now support multiple ground truth. There may be small fixes before we could use this. 7. Added targets for bench_fw_range, ivf, codecs and optimize. **Analysis of ivf result**: D58823037 Reviewed By: algoriddle Differential Revision: D57236543 fbshipit-source-id: ad03b28bae937a35f8c20f12e0a5b0a27c34ff3b
Summary: ### Description Add new signed 8 bit scalar quantizer, `QT_8bit_direct_signed` to ingest signed 8 bit vectors ([-128 to 127]). ### Issues Resolved facebookresearch#3488 Pull Request resolved: facebookresearch#3501 Reviewed By: mengdilin Differential Revision: D58639363 Pulled By: mdouze fbshipit-source-id: cf7f244fdbb7a34051d2b20c6f8086cd5628b4e0
Summary: The issue was that `uniform_int_distribution` generates numbers in the range `[0, ntotal]` and not `[0, ntotal)`, which was an oversight on my part. This PR also attempts to reduce the tolerance for `copyTo` tests as we have seen those fail intermittently. cc ramilbakhshyiev mdouze cjnolet Pull Request resolved: facebookresearch#3552 Reviewed By: junjieqi Differential Revision: D59097786 Pulled By: ramilbakhshyiev fbshipit-source-id: 9dac4367e25c6c219b116ed172089a2fa2a39c4f
…rch#3602) Summary: Start beta-testing AVX512 Build on self-hosted github runner with label faiss-aws-m7i.xlarge. This build is non-blocking on the PRs right now (via the parameter `continue-on-error`) Pull Request resolved: facebookresearch#3602 Reviewed By: ramilbakhshyiev Differential Revision: D59290555 Pulled By: mengdilin fbshipit-source-id: 6a82538bd44fecf4913885ac3b1999eb0fa52047
…earch#3609) Summary: Pull Request resolved: facebookresearch#3609 The current version of Faiss cannot be built due to missing #includes on a test file tests/test_ivf_index.cpp. This is better described in issue facebookresearch#3532. facebookresearch#3533 Reviewed By: ramilbakhshyiev Differential Revision: D59314273 fbshipit-source-id: 6ec8bfa973760d0a44bc94ae751b4fc55c4784ef
Summary: Pull Request resolved: facebookresearch#3607 Github issue: facebookresearch#3026 Reviewed By: mengdilin, junjieqi Differential Revision: D59284732 fbshipit-source-id: a1100c05e7eab40ae90e910364f9ee1db8b1bc55
Summary: After a reviewer labels the issue as "autoclose", issues will be labeled as "stale" after 7 days of no update and close after an additional 7 days. See https://docs.github.com/en/actions/managing-issues-and-pull-requests/closing-inactive-issues and https://github.com/marketplace/actions/close-stale-issues Pull Request resolved: facebookresearch#3614 Reviewed By: junjieqi Differential Revision: D59411424 Pulled By: asadoughi fbshipit-source-id: d79f173de900d5aec53d01c77e8ddc9a8312c12c
Summary: Pull Request resolved: facebookresearch#3618 Reviewed By: junjieqi Differential Revision: D59467510 Pulled By: asadoughi fbshipit-source-id: 52c4cc42e7a9362d78596a695ec4a1121bf455a2
Summary: Pull Request resolved: facebookresearch#3619 Resolves issue facebookresearch#3293 This just means that we wouldn't get verbose prints in train() before, and after we do. Reviewed By: junjieqi Differential Revision: D59474995 fbshipit-source-id: 72537643b661c553353e5f701cfcaf76d21f40d2
Summary: Add dispatcher function to avoid repeating dispatching code for distance computation and result handlers. Reviewed By: asadoughi Differential Revision: D59318865 fbshipit-source-id: 59046ede02f71a0da3b8061289fc70306bf875cb
Summary: Pull Request resolved: facebookresearch#3611 Using the new dispatcher functions, add search func to flat codes. To test it, make IndexLattice a subclass of FlatCodes and check the resonstruction there. Reviewed By: asadoughi Differential Revision: D59367989 fbshipit-source-id: 405dab4358fe34b2e38ac8bcc222b19f58643229
Summary: Pull Request resolved: facebookresearch#3608 This is a straightforward implementation of QINCo in CPU Faiss, with encoding and decoding capabilities (not training). For this, we translate a simplified version of some torch classes: - tensors, restricted to 2D and int32 + float32 - Linear and Embedding layer Then the QINCoStep and QINCo can just be defined as C++ objects that are copy-constructable. There is some plumbing required in the wrapping layers to support the integration. Pytroch tensors are converted to numpy for getting / setting them in C++. Reviewed By: asadoughi Differential Revision: D59132952 fbshipit-source-id: eea4856507a5b7c5f219efcf8d19fe56944df088
Summary: With the refactoring diff landed in [D59650573](https://www.internalfb.com/diff/D59650573), we take in the instance type string for aws and register it as the label for the runner. There was a typo in the previous registration (`m7i.xlarge` instead of `m7i.large`). The refactoring diff fixed the typo from the runner registration side but we need to fix it on the CI side as well so CI can find the runners Pull Request resolved: facebookresearch#3649 Test Plan: CI's AVX512 run on Github Action should succeed instead of stuck in pending Reviewed By: ramilbakhshyiev Differential Revision: D59916841 Pulled By: mengdilin fbshipit-source-id: 5403e1a449ed765f0bf1a00e4a0d81d6c97463b9
Summary: Pull Request resolved: facebookresearch#3635 Add a util function to return the version in the c api. Reviewed By: ramilbakhshyiev, fxdawnn Differential Revision: D59817407 fbshipit-source-id: ca805f8e04f554d0294ba9da8ec6dc7c31e91fe3
Summary: Pull Request resolved: facebookresearch#3679 T195237796 Claims we should be able to incldue nbits in the LSH factory string. Their example is: ``` index = faiss.index_factory(128, 'LSH16rt') Returns the following error. faiss/index_factory.cpp:880: could not parse index string LSHrt_16 ``` This is my first attempt at modifying the regex to accept an integer for nbits. Can an expert help me understand what the domain of accepted strings should be so I can modify the regex as necessary? Reviewed By: ramilbakhshyiev Differential Revision: D60054776 fbshipit-source-id: e47074eb9986b7c1c702832fc0bf758f60f45290
Summary: Pull Request resolved: facebookresearch#3687 We will write a warning if nbits is not specified while using index factory with LSH. The warning lets users know we will be using default d as nbits. Reviewed By: ramilbakhshyiev Differential Revision: D60187935 fbshipit-source-id: 0fa960eeed615d857add77fa131a4cfa1989809d
Summary: This is very convenient to have `range_seach()` in `IndexRefine`. Unlike the plain `search()` method, `range_search()` just reevaluates the computed distances from the baseline index. The labels are not re-sorted according to new distances, because this is not listed as a requirement in a method description https://github.com/facebookresearch/faiss/blob/adb188411a98c3af5b7295c7016e5f46fee9eb07/faiss/Index.h#L150-L161 https://github.com/facebookresearch/faiss/blob/adb188411a98c3af5b7295c7016e5f46fee9eb07/faiss/impl/AuxIndexStructures.h#L35 Pull Request resolved: facebookresearch#4022 Reviewed By: mnorris11 Differential Revision: D66116082 Pulled By: gtwang01 fbshipit-source-id: 915aca2570d5863c876c9497d4c885e270b9b220
Summary: Pull Request resolved: facebookresearch#4099 Expose interface of accumulate_to_mem. accumulate_to_mem calls pq4 fast scan accumulate with a dummy scaler and a StoreResultHandler. The function is not currently in use anywhere. We plan to use it in unicorn integration of PQ4 fast scan. Reviewed By: asadoughi Differential Revision: D67299378 fbshipit-source-id: d6b1b247c4f14b698058cb671dd418bf32d35ca0
Summary: `READSTRING` is not used and also the function's meaning is actually WRITESTRING, just remove it Pull Request resolved: facebookresearch#4122 Reviewed By: junjieqi, pankajsingh88 Differential Revision: D68043967 Pulled By: mnorris11 fbshipit-source-id: e2b5a7c4d7769315c6f91ba1bc0a921e7ce7e1a9
Summary: ```gh run``` seems to time out every now and then, crashing the retry build. I suspect this because whenever the retry build fails, it lacks any logging that would happen if ```gh run``` finishes. Changing to use ```gh view``` and sleeping for 10 minutes before viewing again may resolve this issue, as it isn't continuously watching. Pull Request resolved: facebookresearch#4124 Test Plan: https://github.com/gtwang01/faiss/actions/runs/12718383559/job/35456710369 {F1974306037} {F1974306103} Also, monitor nightlies to ensure it works properly Reviewed By: mengdilin Differential Revision: D68032708 Pulled By: gtwang01 fbshipit-source-id: 02f57239e6203a1d0accc09c48aa7071e553878f
…#4095) Summary: Resolves issue facebookresearch#4089 - IndexIVFPQFastScan crashes with certain nlist values The `reconstruct_from_offset` method in `IndexIVFFastScan` was incorrectly reconstructing vectors, causing crashes when the `nlist` parameter was not byte-aligned (e.g. 100 instead of 256). The root cause was that the `list_no` (Voronoi cell number) was not being properly encoded into the `code` vector before passing it to the `sa_decode` function. This resulted in invalid `list_no` values being read in `sa_decode`, triggering the assertion failure `'list_no >= 0 && list_no < nlist'` when `nlist` in some cases. This PR fixes the issue with the following changes to `reconstruct_from_offset`: 1. Encode the `list_no` into the beginning of the `code` vector using the existing `encode_listno` method 2. Start the `BitstringWriter` after the coarse code portion of `code` (shifted by `coarse_code_size()` bytes) 3. Remove the residual centroid addition logic, as it is already handled in `sa_decode` After these changes: - Crashes no longer occur for any `nlist` value - Reconstruction is now correct, matching the output of `IndexIVFPQ` Fixes facebookresearch#4089 Please review and let me know if any changes are needed. Thanks! Pull Request resolved: facebookresearch#4095 Reviewed By: asadoughi Differential Revision: D67937160 Pulled By: mdouze fbshipit-source-id: 4705106ba49c01c788b3c75c39c2260615f45764
Summary: Pull Request resolved: facebookresearch#4128 Fix reconstruction code for the fast-scan and IVF fast-scan indices. Reviewed By: asadoughi Differential Revision: D68159014 fbshipit-source-id: fb33416eed994196b34f0f6d3014f4d4859b6039
Summary: Hey there 👋 currently I am working on a project using faiss using [FaissMask](https://github.com/andyalm/faissmask) (.NET Bindings). As I am developing on a windows arm64 machine I needed a way to compile faiss for windows arm. I ran the tests in FaissMask and my little hack seemed valid. Here is a little script to compile it yourself: ```bash #!/bin/bash # Execute in git bash # and cmake (https://cmake.org/download/) BRANCH="feat/windows-arm64-support" GITHUB_ACCOUNT=Binozo git clone --depth=1 --recursive --branch $BRANCH https://github.com/$GITHUB_ACCOUNT/faiss.git cd faiss cmake -Wno-dev -DFAISS_ENABLE_GPU=OFF -DFAISS_ENABLE_PYTHON=OFF -DFAISS_ENABLE_C_API=ON -DBUILD_TESTING=OFF -DBUILD_SHARED_LIBS=ON -DCMAKE_WINDOWS_EXPORT_ALL_SYMBOLS=ON -DBLA_VENDOR=Intel10_64_dyn "-DMKL_LIBRARIES=C:\Users\<USER>\CLionProjects\OpenBLAS\cmake-build-release\lib\Release\openblas.lib" -B build -S . cmake --build build --config Release --target faiss faiss_c ``` (Make sure you compiled `OpenBLAS` for windows arm64 before and adjust `C:\Users\<USER>\CLionProjects\OpenBLAS\cmake-build-release\lib\Release\openblas.lib` if needed) Pull Request resolved: facebookresearch#4087 Reviewed By: asadoughi Differential Revision: D67297875 Pulled By: junjieqi fbshipit-source-id: c00f70e5f7fedd5f9fb86fee9258e5ef53b59ce7
) Summary: This pull request introduces a new default argument, `ngpu=-1`, to the `knn_ground_truth` function in the `faiss.contrib`. ## Purpose of Change ### Bug Fix In the current implementation, running tests under the tests directory (CPU tests) in an environment with faiss-gpu installed would inadvertently use the GPU and cause unintended behavior. This pull request prevents the GPU from being used during CPU-only tests by explicitly controlling GPU allocation via the ngpu parameter. ### API Consistency Other functions that call `faiss.get_num_gpus` in `faiss.contrib`, such as `range_search_max_results` and `range_ground_truth`, already include the `ngpu` argument. Adding this parameter to `knn_ground_truth` will ensure consistency across the API, reduce potential confusion, and improve ease of use. Pull Request resolved: facebookresearch#4123 Reviewed By: asadoughi Differential Revision: D68199506 Pulled By: junjieqi fbshipit-source-id: cb50e206d8a1a982c21b0ccb42825ea45873f3ef
Summary: Pull Request resolved: facebookresearch#4130 same as title Reviewed By: asadoughi Differential Revision: D68388863 fbshipit-source-id: 4ebb38d8454bf95733c918950d7d8d3b22e00d5d
…esearch#4139) Summary: Pull Request resolved: facebookresearch#4139 Reviewed By: junjieqi Differential Revision: D68630805 fbshipit-source-id: ff2446e8179cefcf588b58673239ef8549d9fd1d
Summary: Pull Request resolved: facebookresearch#4140 Auto retry build should have been made more reliable in D68032708, so reenable it. Reviewed By: mengdilin Differential Revision: D68636531 fbshipit-source-id: 00fe52bbd2a711d3a74b6febaac9763d384af0bf
Summary: Pull Request resolved: facebookresearch#4126 Good resource on overriding channels to make sure we aren't using `defaults`:https://stackoverflow.com/questions/67695893/how-do-i-completely-purge-and-disable-the-default-channel-in-anaconda-and-switch Explanation of changes: - - changed to miniforge from miniconda: this ensures we only pull in from conda-defaults when creating the environment - architecture: ARM64 and aarch64 are the same thing. But there is no miniforge package for ARM64, so we need to make it check for aarch64 instead. However, mac breaks this rule, and does have macOS-arm64! So there is a conditional for mac to use arm64. https://github.com/conda-forge/miniforge/releases/ - action.yml mkl 2022.2.1 change: conda-forge and defaults have completely different dependencies. Defaults required intel-openmp, but now on conda-forge, mkl 2023.1 or higher requires llvm-openmp >=14.0.6, but this is incompatible with the pytorch build <2.5 which requires llvm-openmp<14.0. We would need to upgrade Python to 3.12 first, upgrade Pytorch build, then upgrade this mkl. (The meta.yaml changes are the ones that narrow it to 2022.2.1 during `conda build faiss`.) So, this has just been changed to 2022.2.1. - mkl now requires _openmp_mutex of type "llvm" instead of "gnu": prior non-cuVS builds all used gnu, because intel-openmp from anaconda defaults channel does not require llvm-openmp. Now we need to remove the gnu one which is automatically pulled in during miniconda setup, and only keep the llvm version of _openmp_mutex. - liblief: The above changes tried to pull in liblief 0.15. This results in an error like `AttributeError: module 'lief._lief.ELF' has no attribute 'ELF_CLASS'`. When I checked passing PR builds on defaults, they use lief 0.12, so I pinned that one for Python 3.9 3.10 3.11. For Python 3.12, we need lief 0.14 or higher. - gcc_linux-64 =11.2 for faiss-gpu on cudatoolkit-11.2: GPU builds kept trying to reference 11.2 when 14.2 was installed. I couldn't figure out why, or how to point it to the 14.2 installed on the host. Current nightly builds still reference 11.2, so I gave up and pinned 11.2 to keep it the same. Moving to 14.2 will take some more investigation. - meta.yaml mkl 2023.0 vs 2023.1 with python versions: 3.9, 3.10, and 3.11 pass with 2023.0, but python 3.12 needs mkl 2023.1 or higher. Otherwise we get: ``` INTEL MKL ERROR: $PREFIX/lib/python3.12/site-packages/faiss/../../.././libmkl_def.so.2: undefined symbol: mkl_sparse_optimize_bsr_trsm_i8. Intel MKL FATAL ERROR: Cannot load libmkl_def.so.2. ``` so the solution was to put a bunch of conditions in in faiss/meta.yaml. We should be able to use Jinja macros to reduce duplication but it requires some investigation. It was failing: https://github.com/facebookresearch/faiss/actions/runs/12915187334/job/36016477707?pr=4126 (paste of logs here: P1716887936). This can be a future BE task. Macro example (the `-` signs remove whitespace lines before and after) ``` {% macro inclmkldevel() %} {%- if PY_VER == '3.9' or PY_VER == '3.10' or PY_VER == '3.11' -%} - mkl-devel =2023.0 # [x86_64] - liblief =0.12.3 # [not win] - python_abi <3.12 {%- elif PY_VER == '3.12' %} - mkl-devel >=2023.2.0 # [x86_64] - liblief =0.15.1 # [not win] - python_abi =3.12 {% endif -%} {% endmacro %} ``` The python_abi was required to be pinned inside these conditions because otherwise several builds got this error: ``` File "/Users/runner/miniconda3/lib/python3.12/site-packages/conda_build/utils.py", line 1919, in insert_variant_versions matches = [regex.match(pkg) for pkg in reqs] ^^^^^^^^^^^^^^^^ TypeError: expected string or bytes-like object, got 'list' ``` Unit test notes: - - test_gpu_basics.py: GPU residual quantizer: Debugged extensively with Matthijs. The problem is in the C++ -> Python conversion. The C++ side prints the right values, but when getting it back to Python, it is filled with junk data. It is only reproducible on CUDA 11.4.4 after switching channels. It is likely a compiler problem. We discussed, and resolved to create a C++ side unit test (so this diff creates TestGpuResidualQuantizer) to verify the functionality and disable the Python unit test, but leave it in the codebase with a comment. Matthijs made extensive notes in https://docs.google.com/document/d/1MjMdOpPgx-MArdrYJZCaQlRqlrhSj5Y1Z9lTyiab8jc/edit?usp=sharing . - test_contrib.py: this now hangs forever and times out the runner for Windows on Python 3.12. I have it skipping now. - test_mem_leak.cpp seems flaky. It sometimes fails, then passes with rerun. Unfixed issues: - - I noticed sometimes downloads will fail with the text like below. It passes on re-run. ``` libgomp-14.2.0-h77fa898_1.conda extraction failed Warning: error libmamba Error when extracting package: Could not chdir info/recipe/parent/patches/0005-Hardcode-HAVE_ALIGNED_ALLOC-1-in-libstdc-v3-configur.patch error libmamba Error when extracting package: Could not chdir info/recipe/parent/patches/0005-Hardcode-HAVE_ALIGNED_ALLOC-1-in-libstdc-v3-configur.patch Warning: Found incorrect download: libgomp. Aborting Found incorrect download: libgomp. Aborting Warning: ``` Green build and tests for both build pull request and nightlies: https://github.com/facebookresearch/faiss/actions/runs/12956402963/job/36148818361 Reviewed By: asadoughi Differential Revision: D68043874 fbshipit-source-id: b105a1e3e6272763ad9daab7fc6f05a79f01c9e2
…rch#4117) Summary: - Small updates to FAISS + cuVS docs. - Fixes IVF-Flat and IVF-PQ benchmarking scripts. - Re-enable cuVS backend in gpu python tests Pull Request resolved: facebookresearch#4117 Reviewed By: mnorris11 Differential Revision: D68792968 Pulled By: asadoughi fbshipit-source-id: 36cd0bbd51635bc559144e9d5733dc29e6f7a229
Summary: Pull Request resolved: facebookresearch#4149 This pins the cudart version which is possibly causing the issue. After the fix, the cudart is all 12.6.77 which matches the passing build, rather than a mix of 12.4 and 12.8 in the failing builds. Reviewed By: gtwang01 Differential Revision: D68781021 fbshipit-source-id: 7fa1a956e313994c9965502ea09b123c4c64acb4
Summary: Pull Request resolved: facebookresearch#4147 Reviewed By: mnorris11 Differential Revision: D68790116 Pulled By: gtwang01 fbshipit-source-id: b0213913963991db6123e7f94e957f27d55ffa60
Summary: For the release, we want to standardize on Faiss instead of FAISS. Changed everything except the CHANGELOG.md which I assume should not be changed after it lands. This doesn't aim to fix any existing lints / errors. Those can be handled at another time. Reviewed By: junjieqi Differential Revision: D68842649 fbshipit-source-id: c0b60d5baa0e1f710db3638ffcc6f223fb3408ad
Summary: Pull Request resolved: facebookresearch#4153 The nightly failed again, but this time it was 12.4.0. The last diff fixed 11.8.0: D68781021 When I checked the last passing nightly, the 12.4.0 CUDA used 12.4.0 cuda-rt. So the solution is to keep cudart 12.6 for cuda 11.8.0, and cudart 12.4 for cuda 12.4.0 Reviewed By: junjieqi Differential Revision: D68837631 fbshipit-source-id: 5aeccd117e1de574f77e29c8c1ede1e83e0e396f
Summary: Pull Request resolved: facebookresearch#4148 Reviewed By: asadoughi Differential Revision: D68779274 Pulled By: mnorris11 fbshipit-source-id: 66db7ebc990e79205fe316c386a4402b7ce2bd80
Summary: Pull Request resolved: facebookresearch#4162 https://docs.github.com/en/actions/sharing-automations/reusing-workflows#using-inputs-and-secrets-in-a-reusable-workflow Seems like we need `secrets` in the workflow and in the caller. Reviewed By: asadoughi Differential Revision: D68965899 fbshipit-source-id: 25580471c727d5ae86c4e7dd4c6ea72b388b1acf
…iss@bleve' * Merging faiss's [v1.10.0](https://github.com/facebookresearch/faiss/releases/tag/v1.10.0) release into bleve branch on fork. * AD: fixed type mismatches for std::vector in unit test over hamming distance: true_ids, ids_gen, ids_ham_knn * Brings in: ``` |\ | * 189a9d4 Michael Norris | Fix build-release workflow (facebookresearch#4162) | * 8581b86 mnorris11 | Increment next release, v1.10.0 (facebookresearch#4148) | * 7856053 Michael Norris | Fix cuVS 12.4.0 nightly failure (facebookresearch#4153) | * 1334d16 Michael Norris | Correct capitalization of FAISS to Faiss | * 9e03ef0 Jesper Stemann Andersen | Added support for building without MKL (facebookresearch#4147) | * dd4cec2 Michael Norris | Fix nightly cuVS 11.8.0 failure (facebookresearch#4149) | * cd797c7 Tarang Jain | Benchmarking Scripts for cuVS Index, more docs updates (facebookresearch#4117) | * 32beb16 Michael Norris | Migration off defaults to conda-forge channel (facebookresearch#4126) | * 3c0133f George Wang | Reenable auto retry workflow (facebookresearch#4140) | * dc8d29a Amir Sadoughi | Update code comment to reflect the range of IF from [1, k] (facebookresearch#4139) | * 3f3d18d Junjie Qi | add test to cover GPU (facebookresearch#4130) | * 905963f Di-Is | Add `ngpu` default argument to `knn_ground_truth` (facebookresearch#4123) | * 4c315a9 Binozo | Windows Arm64 support (facebookresearch#4087) | * 89e93e2 Matthijs Douze | more fast-scan reconstruction (facebookresearch#4128) | * 86fa0db Ali Safaya | Fix IndexIVFFastScan reconstruct_from_offset method (facebookresearch#4095) | * b9fe1dc gtwang01 | Disable retry build (facebookresearch#4124) | * 88676b8 chasingegg | Remove unused(and wrong) io macro (facebookresearch#4122) | * 3d0ac32 Michelle Ma | Expose accumulate_to_mem from faiss interface (facebookresearch#4099) | * 162e6ce Alexandr Guzhva | add range_search() to IndexRefine (facebookresearch#4022) | * 9590ad2 Maria Lomeli | PQ with pytorch (facebookresearch#4116) | * 0cbc2a8 Mulugeta Mammo | Use _mm512_popcnt_epi64 to speedup hamming distance evaluation. (facebookresearch#4020) | * ab479a1 Junjie Qi | enable quiet mode for conda install (facebookresearch#4112) | * 3c8dc41 Satyendra Mishra | Set KnnDescriptor.desc_name in the Benchmarking core framework in FAISS like other descriptors (facebookresearch#4109) | * ab8cb9c Tarang Jain | Link cuVS Docs (facebookresearch#4084) | * 3beb07b Mulugeta Mammo | Add a new architecture mode: 'avx512_spr'. (facebookresearch#4025) | * 3a1ad46 Artem Labazov | Conditionally compile extras like benchmarks and demos (facebookresearch#4094) | * 5637bb8 Tarang Jain | Upgrade CUVS to 24.12 (facebookresearch#4021) | * 0a0af00 Amir Sadoughi | Update action.yml (facebookresearch#4100) | * 3ff8a24 Satyendra Mishra | Test sa_decode methd on IndexIVFFlat (facebookresearch#4098) | * db3409a Satyendra Mishra | Update benchmarking library code to work for IdMap index as well (facebookresearch#4093) | * 66c989a Junjie Qi | Pin Nightlies with testing on PR (facebookresearch#4088) | * bfdda6a Tom Jackson | Exhaustive IVF probing in scalar quantizer tests (facebookresearch#4075) | * 87a760e George Wang | Add testing for utils/hamming.cpp (facebookresearch#4079) | * 5f0a99a Amir Sadoughi | Update __init__.py (facebookresearch#4086) | * f5cd7d0 Tarang Jain | Resolve Packaging Issues (facebookresearch#4044) | * ddb7e0d Junjie Qi | Change github action workflows name (facebookresearch#4083) | * d997ea0 Junjie Qi | fix nightily build (facebookresearch#4080) | * 96bc9c7 Michael Norris | Add more unit tests for index_read and index_write (facebookresearch#4068) | * 90e4c4d George Wang | Fix SCD Table test flakiness (facebookresearch#4069) | * 750381d Michael Norris | Add more unit testing for HNSW [4/n] (facebookresearch#4061) | * b5a5846 Michael Norris | Add more unit testing for HNSW [3/n] (facebookresearch#4059) | * 713f379 Mengdi Lin | improve naming due to codemod (facebookresearch#4072) | * 2483fea Mengdi Lin | improve naming due to codemod (facebookresearch#4071) | * 616c503 Mengdi Lin | improve naming due to codemod (facebookresearch#4067) | * 0f86f33 Mengdi Lin | Improve naming due to codemod (facebookresearch#4070) | * 0d568bc Junjie Qi | separare the github build into two conditions (facebookresearch#4066) | * 9bb6948 Mengdi Lin | Improve naming due to codemod (facebookresearch#4065) | * 0868ffc Michael Norris | Add more unit testing for IndexHNSW [2/n] (facebookresearch#4056) | * 8939f48 Mengdi Lin | Improve naming due to codemod (facebookresearch#4064) | * 661557e Michael Norris | Add more unit testing for IndexHNSW [1/n] (facebookresearch#4054) | * 1af8951 Mengdi Lin | Improve naming due to codemod (facebookresearch#4063) | * 1ac6f37 Junjie Qi | Pin conda build (facebookresearch#4062) | * eb49d8c Junjie Qi | pin arm64 dependency (facebookresearch#4060) | * 82f8e64 Junjie Qi | add cuda-toolkit for GPU (facebookresearch#4057) | * 80a8ff5 George Wang | Unit tests for distances_simd.cpp | * eed3860 Michael Norris | stop dealloc of coarse quantizer when it is deleted (facebookresearch#4045) | * f8ae5f4 Pankaj Singh | remove inconsistent oom exception test (facebookresearch#4052) | * cc5ff1f Junjie Qi | pin the dependecies version for x86_64 (facebookresearch#4046) | * 697b6dd Amir Sadoughi | Fix unused variable compilation error (facebookresearch#4041) | * 37f52dc Michael Pittard | ROCm support for bfloat16 (facebookresearch#4039) | * d1ae64e Tarang Jain | Updates to faiss-gpu-cuvs nightly pkg (facebookresearch#4032) | * f31f06a sadoughi | Constrain conda version for Windows build (facebookresearch#4040) | * eaab46c Jeff Johnson | Faiss GPU: bfloat16 brute-force kNN support (facebookresearch#4018) | * 844e3ce Satyendra Mishra | (1/n) - Preload datasets in manifold so that subsequent stages of training, indexing and search can use those instead of each trainer or indexer downloading data. (facebookresearch#4034) | * 98d335b Shengqi Chen | Some chore fixes (facebookresearch#4010) | * 3c25a68 Junjie Qi | fix linter (facebookresearch#4035) | * 94b640c Amir Sadoughi | Pin to numpy<2 (facebookresearch#4033) | * eb5e734 Nicholas Ormrod | facebook-unused-include-check in fbcode/faiss (facebookresearch#4029) | * 1349220 Tarang Jain | Migrate from RAFT to CUVS (facebookresearch#3549) | * 0fb56d9 Richard Barnes | Remove unused-variable in dumbo/backup/dumbo/service/tests/ChainReplicatorTests.cpp +3 (facebookresearch#4024) | * adb1884 Richard Barnes | Fix shadowed variable in faiss/IndexAdditiveQuantizer.cpp (facebookresearch#4011) | * 1f2b7ce Mengdi Lin | introduce data splits in dataset descriptor (facebookresearch#4012) | * cfd4804 Mengdi Lin | write distributed_kmeans centroids and assignments to hive tables (facebookresearch#4017) | * a11c1db Amir Sadoughi | Added IndexLSH to the demo (facebookresearch#4009) | * 2c961cc Michael Kosten | Add VectorTransform read from filename to the C API (facebookresearch#3970) | * 9766d64 Michael Norris | Add index binary to telemetry (facebookresearch#4001) | * ecb4b80 Amir Sadoughi | Update autoclose.yml (facebookresearch#4000) | * c5fbefe Tom Jackson | Fix reverse_index_factory formatting of ScalarQuantizers (facebookresearch#4003) | * acaa01f Amir Sadoughi | Moved add_sa_codes, sa_code_size to Index, IndexBinary base classes (facebookresearch#3989) | * d657dfb Pankaj Singh | fix merge_flat_ondisk stress run failures (facebookresearch#3999) | * 5539039 Pankaj Singh | add validity check AlignedTableTightAlloc clear method (facebookresearch#3997) | * efa18da Michael Norris | add all wrapped indexes to the index_read | * 1e7ef59 Amir Sadoughi | demo: IndexPQ: separate codes from codebook (facebookresearch#3987) | * 9c3cd77 Michael Norris | Re-add example of how to build, link, and test an external SWIG module"" (facebookresearch#3981) | * ebe5a69 Amir Sadoughi | Update RAFT CI with pytorch 2.4.1 (facebookresearch#3980) | * eff0898 Michael Norris | Enable linting: lint config changes plus arc lint command (facebookresearch#3966) | * e7d153d Amir Sadoughi | Adjust nightly build (facebookresearch#3978) | * da20af4 Michael Norris | fix some more nvidia licenses that get erased (facebookresearch#3977) | * 56a383f Michael Norris | Resolve "duplicate-license-header": Find and replace duplicate license headers (facebookresearch#3967) | * adcde10 Matthijs Douze | Allow to replace graph structure for NSG graphs (facebookresearch#3975) | * fd6d784 Michael Norris | Resolve "incorrect-portions-license" errors: add no license lint to top of GPU files with both licenses (facebookresearch#3965) | * f9a01c6 Michael Pittard | Removing Manual Hipify Build Step (facebookresearch#3962) | * 6617b13 Kyle Edwards | Consolidate set_target_properties() calls in faiss/CMakeLists.txt (facebookresearch#3973) | * 1a799d0 Richard Barnes | Fix shadowed variable in faiss/utils/NeuralNet.cpp (facebookresearch#3952) | * d492753 Richard Barnes | Fix shadowed variable in faiss/impl/simd_result_handlers.h (facebookresearch#3960) | * c93d1fd Richard Barnes | Fix shadowed variable in faiss/impl/HNSW.cpp (facebookresearch#3961) | * 68f66bc Richard Barnes | Fix shadowed variable in faiss/IndexIVFAdditiveQuantizer.cpp (facebookresearch#3958) | * b8ae854 Maria Lomeli | Cache device major version value to avoid multiple calls of getCudaDeviceProperties (facebookresearch#3950) | * 7a51922 Richard Barnes | Fix shadowed variable in faiss/IndexPQ.cpp (facebookresearch#3959) | * cb1a512 Michael Norris | Back out "Add example of how to build, link, and test an external SWIG module" (facebookresearch#3954) | * dce7c09 vorj | Add some SVE implementations (facebookresearch#3933) | * 1ab7e5c Michael Norris | add copyright header (facebookresearch#3948) | * e017c35 vorj | Fix INSTALL.md due to failure of conflict resolving (facebookresearch#3915) | * 4d78137 Alexandr Guzhva | Place a useful cmake function 'link_to_faiss_lib' into a separate file (facebookresearch#3939) | * 3daf48c George Wang | Fix total_rows (facebookresearch#3942) | * 6441b56 George Wang | PQFS into Index trainer (facebookresearch#3941) | * 847cde8 Michael Norris | Add example of how to build, link, and test an external SWIG module (facebookresearch#3922) | * a99dbcd Matthijs Douze | implement ST_norm_from_LUT for the ResidualQuantizer (facebookresearch#3917) | * 07a345c Kumar Saurabh Arora | Add desc_name to dataset descriptor (facebookresearch#3935) | * 61eaf19 Kumar Saurabh Arora | Move train, build and search to their respective operators (facebookresearch#3934) | * d243e62 Amir Sadoughi | Updated conda CI label from staging to main, INSTALL.md (facebookresearch#3929) | * af70c5b George Wang | 3893 - Fix index factory order of idmap and refinement (facebookresearch#3928) | * c5aed7c mengdilin | FIX SVE CI latency regression (facebookresearch#3918) | * 2e6551f Matthijs Douze | Support search_preassigned in torch (facebookresearch#3916) | * be4fc8e Mengdi Lin | fix hnsw unit test in opt mode (facebookresearch#3919) | * 092e2cd Amir Sadoughi | Fix package upload for main releases (facebookresearch#3910) | * ab59374 Tarang Jain | Update Pinned RAFT version (facebookresearch#3921) | * d2692b8 Kumar Saurabh Arora | Small nits and formatting (facebookresearch#3907) | * caa3b34 Amir Sadoughi | Increment next release, v1.9.0 (facebookresearch#3887) | * 0df5d24 Mengdi Lin | clean up hnsw benchmark (facebookresearch#3901) | * 09bffd2 lranon1 | Update INSTALL.md to ensure AVX2 and AV512 support for python package (facebookresearch#3892) | * 1ee7561 Jaap Aarts | Disable the NEON optimisations on gcc <= 8 (facebookresearch#3869) | * d8aec60 Kumar Saurabh Arora | Changing dependency for bench_fw to *_cpu instead of *_gpu (facebookresearch#3889) | * c8d1474 Facebook Community Bot | Re-sync with internal repository (facebookresearch#3885) | * 149c1f4 mengdilin | Add performance regression tests (facebookresearch#3793) | * 0660b23 Michael Norris | Add reverse factory string util, add StringIOReader, add centralized JK (facebookresearch#3879) | * 1f42e81 mengdilin | Fix CI 2.0: Compile SQ for avx512 (facebookresearch#3880) | * 03f1d2a Mengdi Lin | fix open source CI (facebookresearch#3878) | * 4eecd91 Mulugeta Mammo | Add AVX-512 implementation for the distance and scalar quantizer functions. (facebookresearch#3853) | * 838612c Matthijs Douze | torch.distributed kmeans (facebookresearch#3876) | * 6baebe2 Matthijs Douze | begin torch_contrib (facebookresearch#3872) | * 0d7817e Matthijs Douze | rewrite python kmeans without scipy (facebookresearch#3873) | * 4e30901 Alexandr Guzhva | Introduce QuantizerTemplateScaling for SQ (facebookresearch#3870) | * 317bd00 Alexandr Guzhva | FIx a bug for a non-simdlib code of ResidualQuantizer (facebookresearch#3868) | * fa60ae6 Michael Norris | more telemetry classes (facebookresearch#3854) | * 73a41f1 Mengdi Lin | simplify and refactor create_servicelab_experiment utility (facebookresearch#3867) | * 83d0599 Mengdi Lin | RCQ search microbenchmark (facebookresearch#3863) | * b6024e9 Michael Pittard | Adding Documentation for ROCm (facebookresearch#3856) | * 6da9952 Jaap Aarts | Add explicit instanciations of `search_dispatch_implem` (facebookresearch#3860) | * f67f518 Mengdi Lin | add benchmarking for hnsw flat based on efSearch (facebookresearch#3858) | * 8a9e600 Shengqi Chen | Fix several typos in code detected by lintian (facebookresearch#3861) | * dc55e11 Mengdi Lin | add hnsw flat benchmark (facebookresearch#3857) | * d104275 Alexandr Guzhva | Add a dockerfile for development (facebookresearch#3851) | * dbdd63b Mengdi Lin | assign_index should default to null (facebookresearch#3855) | * 52ce3f5 Mengdi Lin | add hnsw unit test for PR 3840 (facebookresearch#3849) | * a166e13 Kumar Saurabh Arora | Adding bucket/path (blobstore) in dataset descriptor (facebookresearch#3848) | * d85fda7 Matthijs Douze | Allow k and M suffixes in IVF indexes (facebookresearch#3812) | * 6fe4640 Kumar Saurabh Arora | Fixing headers as per OSS requirement (facebookresearch#3847) | * 21dfdba Alexandr Guzhva | Fix an incorrectly counted the number of computed distances for HNSW (facebookresearch#3840) | * 3f41161 Ramil Bakhshyiev | Re-enable Query_L2_MMCodeDistance and Query_IP_MMCodeDistance tests for ROCm (facebookresearch#3838) | * 753833c Ramil Bakhshyiev | Upgrade to ROCm 6.2 (facebookresearch#3839) | * 736cd4d Ramil Bakhshyiev | Do not unnecessarily install CUDA for ROCm | * 18bc38a Ramil Bakhshyiev | Quiet down apt-get on ROCm builds (facebookresearch#3836) | * e261725 Alexandr Guzhva | faster hnsw CPU index training (facebookresearch#3822) | * 52cf9af Matthijs Douze | group SWIG tests into one file (facebookresearch#3807) | * 202a204 Kumar Saurabh Arora | Allow search Index without Gt (facebookresearch#3827) | * a4ebcb1 Michael Norris | Add error for overflowing nbits during PQ construction (facebookresearch#3833) | * 1cafc71 Mengdi Lin | remove compile options label from gbench (facebookresearch#3834) | * d296b2c Bhavik Sheth | Prevent reordering of imports by auto formatter to avoid crashes (facebookresearch#3826) | * 501a8be Mengdi Lin | more refactor and add encode/decode steps to benchmark (facebookresearch#3825) | * 4683cc1 Mengdi Lin | create perf_tests directory and onboard all scalar quantizer types to benchmark (facebookresearch#3824) | * 5e61450 mengdilin | Build SVE CI with openblas that was compiled with USE_OPENMP=1 (facebookresearch#3776) | * ca1ab78 Mengdi Lin | always upload pytest results (facebookresearch#3808) | * c418b30 David Tolnay | Fix deprecated use of 0/NULL (facebookresearch#3817) | * 383b5d9 Sergii Dymchenko | Use weights_only for load (facebookresearch#3796) | * 95e0a66 Pankaj Singh | Nightly failure fix - ignore searched vectors with identical distances (facebookresearch#3809) | * 97e6f48 Alexandr Guzhva | Some small improvements. (facebookresearch#3692) | * a5ad714 Ramil Bakhshyiev | Re-order imports in Python tests to avoid crashing (facebookresearch#3806) | * 145e93d Kumar Saurabh Arora | Fix bench_fw_codec | * 4283e5b Alexandr Guzhva | Add standalone Link-Time Optimization option to CMake (facebookresearch#2943) | * 37f6b76 Kumar Saurabh Arora | Adding support for index builder (facebookresearch#3800) | * 084496a Sergii Dymchenko | Fix parameter names in docstrings (facebookresearch#3795) | * 3614cc7 Mengdi Lin | avx512 compilation option (facebookresearch#3798) | * 4ca6734 Mengdi Lin | add AMD_ROCM as part of get_compile_options (facebookresearch#3790) | * 58a673d Ramil Bakhshyiev | Enable most of C++ tests on ROCm (facebookresearch#3786) | * 6053348 Mengdi Lin | fix get_compile_options bug (facebookresearch#3785) | * 5c87f13 Kumar Saurabh Arora | Add sampling fields to dataset descriptor (facebookresearch#3782) | * a43afd6 George Wang | Specify to retry only on failed jobs (facebookresearch#3772) | * a10b883 Michael Norris | Move static functions to header file (facebookresearch#3757) | * f3c05bd Pankaj Singh | add reconstruct support to additive quantizers (facebookresearch#3752) | * 6e6685b Mengdi Lin | delete circle CI config (facebookresearch#3732) | * c0b32d2 Mengdi Lin | fix ARM64 SVE CI due to openblas version bump (facebookresearch#3777) | * 924c24d Ramil Bakhshyiev | Enable Python tests for ROCm (facebookresearch#3763) | * 772d860 Ramil Bakhshyiev | Reorder imports in torch_test_contrib_gpu (facebookresearch#3761) | * d40adca Ramil Bakhshyiev | Add midding hipStream SWIG typedef to fix ROCm memleak in Python (facebookresearch#3760) | * afe9c40 Alexandr Guzhva | introduce options for reducing the overhead for a clustering procedure (facebookresearch#3731) | * b10f001 Pankaj Singh | minor refactor to avoid else block in IVFPQ reconstruct_from_offset. (facebookresearch#3753) | * 2968ab1 Emy Sun | Add hnsw search params for bounded queue option (facebookresearch#3748) | * 80a2462 Kumar Saurabh Arora | Fixing initialization of dictionary in dataclass (facebookresearch#3749) | * a56ee81 Ramil Bakhshyiev | Containerize ROCm build and move it to AMD GPU runners (facebookresearch#3747) | * 290464f Kumar Saurabh Arora | Adding embedding column to dataset descriptor (facebookresearch#3736) | * ac18577 Ramil Bakhshyiev | Install gpg for ROCm builds (facebookresearch#3744) | * 9f9b53b Ramil Bakhshyiev | Use $HOME variable to find Conda binaries instead of hard-coded path (facebookresearch#3743) | * 11c15af Ramil Bakhshyiev | Append ROCm flag to compile definitions for Python instead of overwriting (facebookresearch#3742) | * 13371c7 George Wang | Auto-retry failed CI builds once (facebookresearch#3740) | * 01317af Ramil Bakhshyiev | Add human readable names to PR build steps (facebookresearch#3739) | * c669357 George Wang | Introduce retry-build workflow (facebookresearch#3718) | * acc3a97 Ramil Bakhshyiev | Add labels to test-results file and include ROCm flag (facebookresearch#3738) | * 9f79bcf Ramil Bakhshyiev | Move nightly builds to 11pm PT (facebookresearch#3737) | * 90b1da4 Amir Sadoughi | Update code comment regarding PQ's search metrics (facebookresearch#3733) | * e5f4647 gtwang01 | Unpin gxx_linux-64 requirement (facebookresearch#3655) | * 85df0e0 Ramil Bakhshyiev | CMake step to symlink system libraries for RAFT and ROCm (facebookresearch#3725) | * 0b2328f Mengdi Lin | Turn on blocking build for AVX512 and SVE on GHA (facebookresearch#3717) | * a757309 Ramil Bakhshyiev | ROCm CMake configuration cleanup (facebookresearch#3716) | * e59d8c3 Ramil Bakhshyiev | ROCm linter and shellcheck warning cleanup (facebookresearch#3715) | * 4e2aead Ramil Bakhshyiev | Enable ROCm in build-only mode (facebookresearch#3713) | * b670cb1 iotamudelta | ROCm support (facebookresearch#3462) | * 677e73f vorj | suppress warning (facebookresearch#3708) | * 2883129 Charles Coulombe | Include libfaiss_python_callbacks.so in python installation. (facebookresearch#2062) | * 8013fbd generatedunixname2443911735787003 | fbcode//faiss/tests (facebookresearch#3707) | * 96d0781 Mengdi Lin | turn on SVE opt mode in CI (facebookresearch#3703) | * 28f7d3a Ramil Bakhshyiev | Merge cmake command in the cmake build action (facebookresearch#3702) | * fecabc2 Ramil Bakhshyiev | Gate ARM SVE build behind base Linux build (facebookresearch#3701) | * 4eeaa42 vorj | Add sve targets (facebookresearch#2886) | * 7178bf8 Matthijs Douze | Fix radius search with HSNW and IP (facebookresearch#3698) | * 34bbe5e mengdilin | Add ARM64 build to build_cmake actions for SVE (facebookresearch#3653) | * 4cfa638 Bhavik Sheth | add get_version() for c_api. (facebookresearch#3688) | * 6e1f23f Amir Sadoughi | Moved statements to faiss.ai (facebookresearch#3694) | * f2361a4 Bhavik Sheth | Back out "Add warning on adding nbits to LSH index factory" (facebookresearch#3690) | * 0363934 Bhavik Sheth | Add warning on adding nbits to LSH index factory (facebookresearch#3687) | * aed7b0e Bhavik Sheth | First attempt at LSH matching with nbits (facebookresearch#3679) | * 8b5895f Bhavik Sheth | 1720 - expose FAISS version field to c_api (facebookresearch#3635) | * 749163e mengdilin | Fix CI for AVX512 (facebookresearch#3649) | * dd72e41 Matthijs Douze | QINCo implementation in CPU Faiss (facebookresearch#3608) | * ab109c2 Matthijs Douze | Add search functionality to FlatCodes (facebookresearch#3611) | * 261edde Matthijs Douze | add dispatcher for VectorDistance and ResultHandlers | * 444614b Michael Norris | Set verbosoe before train (facebookresearch#3619) | * 036a7e3 Amir Sadoughi | Rename autoclose to autoclose.yml (facebookresearch#3618) | * e5ab701 Amir Sadoughi | Create autoclose GHA workflow (facebookresearch#3614) | * 3fe0b93 Michael Norris | Fix typo in matrix mult (facebookresearch#3607) | * db251e2 Junjie Qi | Adding missing includes which are necessary for building (facebookresearch#3609) | * f821704 mengdilin | Non-Blocking AVX512 Build on self-hosted github runner (facebookresearch#3602) | * 079fd55 divyegala | Fix seg faults in CAGRA C++ unit tests (facebookresearch#3552) | * 33c0ba5 Naveen Tatikonda | Add SQ8bit signed quantization (facebookresearch#3501) | * da75d03 Kumar Saurabh Arora | Refactor bench_fw to support train, build & search in parallel (facebookresearch#3527) | * 3a7c718 Kumar Saurabh Arora | Adding faiss bench_fw to bento faiss kernel (facebookresearch#3531) | * e758973 Matthijs Douze | Add ABS_INNER_PRODUCT metric (facebookresearch#3524) | * e188eb3 Ramil Bakhshyiev | Bump libraft to 24.06 to unblock nightly RAFT builds (facebookresearch#3522) | * 849557a Ramil Bakhshyiev | Unbreak RAFT conda builds (facebookresearch#3519) | * e65a910 Gergely Szilvasy | fix Windows build - signed int OMP for MSVC (facebookresearch#3517) | * 44d21ee Ramil Bakhshyiev | Consolidate build environment configuration steps in cmake builds (facebookresearch#3516) | * 34feae4 Gergely Szilvasy | typo in test_io_no_storage (facebookresearch#3515) | * d45f78b Ramil Bakhshyiev | Add conda bin to path early in the cmake GitHub action (facebookresearch#3512) | * 3d32330 Gergely Szilvasy | add use_raft to knn_gpu (torch) (facebookresearch#3509) | * f71d5b9 Matthijs Douze | fix spurious include to land the cagra diff (facebookresearch#3502) | * df0dea6 divyegala | Interop between CAGRA and HNSW (facebookresearch#3252) | * ec67ac1 Abhiram Vadlapatla | Update .gitignore (facebookresearch#3492) | * a900cfa Xiao Fu | Add cpp tutorial for index factory refine index construction (facebookresearch#3494) | * bf73e38 Matthijs Douze | add skip_storage flag to HNSW (facebookresearch#3487) | * 2230434 Kumar Saurabh Arora | Adding buck target for experiment bench_fw_ivf (facebookresearch#3423) | * 0beecb4 Kumar Saurabh Arora | sys.big_endian to sys.byteorder (facebookresearch#3422) | * 6e7d9e0 simshi | fix algorithm of spreading vectors over shards (facebookresearch#3374) | * db6ff2e Jim Borden | Workaround for missing intrinsic on gcc < 9 (facebookresearch#3481) | * 6e423cc Xiao Fu | Add python tutorial on different indexs refinement and respect accuracy measurement (facebookresearch#3480) | * 6580156 Tarang Jain | Delete Raft Handle (facebookresearch#3435) | * eb28481 Alexandr Guzhva | Remove duplicate NegativeDistanceComputer instances (facebookresearch#3450) | * 729a66f Richard Barnes | Remove extra semi colon from deprecated/libmccpp/ThreadSafeClientPool.h (facebookresearch#3479) | * eec4cba Ramil Bakhshyiev | Disable CircleCI builds (facebookresearch#3477) | * 93bc9b6 Ramil Bakhshyiev | Gate all PR builds behind linux-x86_64-cmake in GitHub Actions (facebookresearch#3476) | * ee7ce21 Ramil Bakhshyiev | Add display names to all PR build jobs on GitHub Actions (facebookresearch#3475) | * 6a94c67 Alexandr Guzhva | QT_bf16 for scalar quantizer for bfloat16 (facebookresearch#3444) | * 414fd1e Xiao Fu | Add tutorial for FastScan with refinement for cpp (facebookresearch#3474) | * b39dd4d Ramil Bakhshyiev | Fix CUDA 11.4.4 nightly in GitHub Actions (facebookresearch#3473) | * f352168 Ramil Bakhshyiev | Fix cron schedule for nightlies via GitHub Actions (facebookresearch#3470) | * 7d7fef0 Xiao Fu | Add FastScan refinement tutorial for python (facebookresearch#3469) | * f38e52c Xiao Fu | Add tutorial on PQFastScan for cpp (facebookresearch#3468) | * 59e3ee1 Saarth Deshpande | Missed printing 'D' (facebookresearch#3433) | * 4489773 Xiao Fu | Add tutorial for FastScan (facebookresearch#3465) | * c1528b5 Ramil Bakhshyiev | Enable nightly builds via GitHub Actions (facebookresearch#3467) | * a60a9e5 Ramil Bakhshyiev | Fix CUDA 11.4.4 builds under CircleCI (facebookresearch#3466) | * 0698ac7 Ramil Bakhshyiev | Properly pass the label for conda upload steps (facebookresearch#3464) | * 8c95c69 Ramil Bakhshyiev | Fix linter warnings in faiss-gpu Conda build script (facebookresearch#3463) | * 7fc8184 Ramil Bakhshyiev | Relax version requirements for action steps (facebookresearch#3461) | * 86bf74d Ramil Bakhshyiev | Enable linux-x86_64-GPU-packages-CUDA-11-4-4 build via GitHub Actions (facebookresearch#3460) | * 0c983f3 Ramil Bakhshyiev | Workaround for CUDA 11.4.4 build in Conda on Ubuntu 22 / v6 kernel (facebookresearch#3459) | * 5e452ed Xiao Fu | Cleaning up more unnecessary print (facebookresearch#3455) | * e822a8c Ramil Bakhshyiev | GitHub Actions files cleanup (facebookresearch#3454) | * bf8bd6b Xiao Fu | Delete all remaining print (facebookresearch#3452) | * 4972abd Xiao Fu | Improve testing code step 1 (facebookresearch#3451) | * 1876925 Amir Sadoughi | Implement METRIC.NaNEuclidean (facebookresearch#3414) | * 72571c7 Ramil Bakhshyiev | Enable both RAFT package builds and CUDA 12.1.1 GPU package build (facebookresearch#3441) | * 745bca8 Mengdi Lin | stabilize formatting for bench_cppcontrib_sa_decode.cpp (facebookresearch#3443) | * 2050a03 Ramil Bakhshyiev | Add cuda-toolkit package dependency to faiss-gpu and faiss-gpu-raft conda build recipes (facebookresearch#3440) | * b8e4489 Richard Barnes | Remove unused variables in faiss/IndexIVFFastScan.cpp (facebookresearch#3439) | * 558a7c3 Mengdi Lin | interrupt for NNDescent (facebookresearch#3432) | * 509f4c1 Matthijs Douze | fix install instructions (facebookresearch#3442) | * 83df64c Alexandr Guzhva | Get rid of redundant instructions in ScalarQuantizer (facebookresearch#3430) | * 4d06d70 Ramil Bakhshyiev | Add disabled linux-x86_64-AVX512-cmake build on GitHub Actions (facebookresearch#3428) | * 2e04533 Ramil Bakhshyiev | Enable linux-x86_64-GPU-cmake build on GitHub Actions (facebookresearch#3427) | * b487c62 Ramil Bakhshyiev | Update system dependencies to enable CUDA builds on v6 kernel and newer libc (facebookresearch#3426) | * e1e4ad0 Carl Love | PowerPC, improve code generation for function fvec_L2sqr (facebookresearch#3416) | * 34fa2ae Ramil Bakhshyiev | Enable linux-x86_64-GPU-w-RAFT-cmake build via GitHub Actions (facebookresearch#3418) | * b3e3c2d Amir Sadoughi | TimeoutCallback C++ and Python (facebookresearch#3417) | * 0cc0e19 Ramil Bakhshyiev | Enable osx-arm64-packages build via GitHub Actions (facebookresearch#3411) | * 1b1a403 Ramil Bakhshyiev | Change linux-arm64-packages build to use 2-core-ubuntu-arm for better availability (facebookresearch#3410) | * 7b8b981 Ramil Bakhshyiev | Enable packages builds on main for windows, linux-arm64, linux-x86_64 via GitHub Actions (facebookresearch#3409) | * 96b88ac Ramil Bakhshyiev | Enable linux-arm64-conda check via GitHub Actions (facebookresearch#3407) | * 74562b2 Ramil Bakhshyiev | Enable windows-x86_64-conda build via GitHub Actions (facebookresearch#3406) | * 5fd8b81 Ramil Bakhshyiev | Enable linux-x86_64-conda build via GitHub Actions (facebookresearch#3405) | * c92b480 Ramil Bakhshyiev | Add format check | * 3121fc6 Jayjeet Chakraborty | Fix facebookresearch#3379: Add tutorial for HNSW index (facebookresearch#3381) | * 825cbac Ramil Bakhshyiev | Add linux-x86_64-AVX2-cmake build | * 7e1d2b1 Ramil Bakhshyiev | Initial config and linux-x86_64-cmake build job only | * c5599a0 Richard Barnes | Fix deprecated use of 0/NULL in faiss/python/python_callbacks.cpp + 1 | * a233bc9 Matthijs Douze | Demo on how to address mulitple index contents | * 5cbff67 Gergely Szilvasy | fix raft log spew | * bd22c93 Junjie Qi | Fix swig osx (facebookresearch#3357) | * 03750f5 Amir Sadoughi | Fix IndexBinary.assign Python method | * 2379b45 Kumar Saurabh Arora | Few fixes in bench_fw to enable IndexFromCodec (facebookresearch#3383) | * 783e044 Matthijs Douze | support big-endian machines (facebookresearch#3361) | * 67574aa Aditya Vidyadhar Kamath | Fix the endianness issue in AIX while running the benchmark. (facebookresearch#3345) | * b2e91f6 Carl Love | Unroll loop in lookup_2_lanes (facebookresearch#3364) | * 5893ab7 Junjie Qi | remove unused code (facebookresearch#3371) | * 3677ab5 Junjie Qi | Switch clang-format-11 to clang-format-18 (facebookresearch#3372) | * 0169f29 iotamudelta | Update required cmake version to 3.24. (facebookresearch#3305) | * ab2b7f5 Andres Suarez | Apply clang-format 18 | * a35eb0a Richard Barnes | Remove unused variables in faiss/IndexIVF.cpp | * acd06d6 Junjie Qi | Switch sprintf to snprintf (facebookresearch#3363) | * 40e8643 Matthijs Douze | selector parameter for FastScan (facebookresearch#3362) | * 17fbeb8 Alexandr Guzhva | Improve filtering & search parameters propagation (facebookresearch#3304) | * 252ae16 Kumar Saurabh Arora | Support for Remove ids from IVFPQFastScan index (facebookresearch#3354) | * 366a814 Gufan Yin | Revert D55723390: Support for Remove ids from IVFPQFastScan index | * 7657e81 Ramil Bakhshyiev | Change index_cpu_to_gpu to throw for indices not implemented on GPU (facebookresearch#3336) | * f34588a Kumar Saurabh Arora | Support for Remove ids from IVFPQFastScan index (facebookresearch#3349) | * cfc7fe5 Junjie Qi | Implement reconstruct_n for GPU IVFFlat indexes (facebookresearch#3338) | * da9f292 Kumar Saurabh Arora | Support of skip_ids in merge_from_multiple function of OnDiskInvertedLists (facebookresearch#3327) | * c9c86f0 Warmchay | Fix missing overload variable in Rocksdb ivf demo (facebookresearch#3326) | * 77e2e79 Ramil Bakhshyiev | Throw when attempting to move IndexPQ to GPU (facebookresearch#3328) | * 4e6b6f8 Aalekh Patel | Add the ability to clone and read binary indexes to the C API. (facebookresearch#3318) | * d99f07e Alexandr Guzhva | AVX512 for PQFastScan (facebookresearch#3276) | * d685413 Junjie Qi | Fix faiss swig build with version > 4.2.x (facebookresearch#3315) | * 03db694 Chip-Kerchner | Fix problems when using 64-bit integers. (facebookresearch#3322) | * 55dc880 Junjie Qi | Change cmake to build googletest from source (facebookresearch#3319) | * 14b8af6 Junjie Qi | Fix IVFPQFastScan decode function (facebookresearch#3312) | * 0c96b0d Gergely Szilvasy | enable rapidsai-nightly channel for libraft (facebookresearch#3317) | * af5793c Kumar Saurabh Arora | Adding test for IndexBinaryFlat.reconstruct_n() (facebookresearch#3310) | * 798427c Kumar Saurabh Arora | Handling FaissException in few destructors of ResultHandler.h (facebookresearch#3311) | * fa1f39e Matthijs Douze | Fix HNSW stats (facebookresearch#3309) | * b77061f Gergely Szilvasy | move to raft 24.04 (facebookresearch#3302) | * 8274c38 Ramil Bakhshyiev | Remove TypedStorage usage when working with torch_utils (facebookresearch#3301) | * 9c79e3d divyegala | RAFT 24.04 API changes (facebookresearch#3282) | * 5483f21 Yuri Victorovich | Use cmake's find_package to link to GTest (facebookresearch#3278) | * 6f3843e Junjie Qi | Back out "Remove swig version and always rely on the latest version" (facebookresearch#3297) | * 0e06a28 Jason Sylka | Revert D54973709: Remove unused fallthrough | * cf364ec Junjie Qi | Remove unused fallthrough (facebookresearch#3296) | * f7fe62e Junjie Qi | Remove swig version and always rely on the latest version (facebookresearch#3295) | * 7d21c92 Maria | Dim reduction support in OIVFBBS (facebookresearch#3290) | * d5e4c79 Maria | Removed index_shard_and_quantize OIVFBBS (facebookresearch#3291) | * e99ad12 ranjitsastra | AIX compilation fix for io classes (facebookresearch#3275) | * dafdff1 Junjie Qi | Change intall.md to reflect faiss 1.8.0 | * 12b92e9 John Mazanec | Skip HNSWPQ sdc init with new io flag (facebookresearch#3250) ```
abhinavdangeti
requested review from
ceejatec,
deepkaran,
Thejas-bhat,
metonymic-smokey,
CascadingRadium and
Likith101
deepkaran
approved these changes
Feb 10, 2025
ceejatec
approved these changes
Feb 18, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Merging faiss's v1.10.0 release into bleve branch on fork.
AD: fixed type mismatches within unit test:
TEST(TestHamming, test_hamming_knn)in file:
tests/test_hamming.cpp: true_ids, ids_gen, ids_ham_knnBrings in: