Skip to content

feat(core): integrate turbo quantizers with flat indexes - #716

Open
richyreachy wants to merge 61 commits into
alibaba:mainfrom
richyreachy:refactor/flat_on_turbo
Open

feat(core): integrate turbo quantizers with flat indexes#716
richyreachy wants to merge 61 commits into
alibaba:mainfrom
richyreachy:refactor/flat_on_turbo

Conversation

@richyreachy

Copy link
Copy Markdown
Collaborator

This PR integrates turbo quantizers into the flat index build, search, and streaming paths while adding optimized AVX2 and AVX-512 distance kernels.

It also makes CRC32C and FP16 hardware acceleration safe on baseline x86 systems by using runtime CPU dispatch and per-function target attributes.

@iaojnh

iaojnh commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

目前该PR不支持旧索引格式,需要考虑兼容性问题

@richyreachy

Copy link
Copy Markdown
Collaborator Author

目前该PR不支持旧索引格式,需要考虑兼容性问题

已经更新:

  • 旧格式 INT8 文件 → 检测到持久化 meta 的 quantizer_name 为空 → 回退 legacy 管线 → 打开、检索、fetch、继续写入都正常(CosineLegacyLayoutReopenFallsBack / L2LegacyLayoutReopenFallsBack 已验证)
  • 新 turbo 格式文件 → meta 里有 "Int8Quantizer" → 保持 turbo 路径重开(CosineAddSearchReopenFetch 验证)
  • schema 与文件不一致(如用 rotate schema 开 turbo 布局文件)→ 仍被 entity 打开时的 meta 校验安全拒绝,不会静默算错

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants