zoekt/testdata/shards/ at main · boltless.me/zoekt · Tangled

boltless.me / zoekt

0

fork of https://github.com/sourcegraph/zoekt

0

zoekt / testdata / shards /

at main 5 files

John Mason Add support for indexing and searching custom fields for repositories (#962) 1y ago

ctagsrepo_v16.00000.zoekt

Add support for indexing and searching custom fields for repositories (#962) At GitLab, we encountered limitations when searching within large namespaces containing thousands of repositories. Specifically, we cannot pass a complete list of RepoIDs due to size constraints. This change introduces support for indexing and searching on custom repository metadata by extending Repository to include an additional Metadata field. All fields within Repository.Metadata are searchable using a regular expression evaluator. This enables more scalable filtering by allowing clients to express regular expression prefix queries on metadata fields, such as: traversal_ids:123-456-.* Or any field really: haystack:nee.*le

1 year ago

ctagsrepo_v17.00000.zoekt

all: compound shard support (#95) This commit adds support for compound shards. A shard now has multiple repositories associated with it, rather than always one. Most of zoekt is document based, so minimal changes are required in the core search evaluation codepath. The only change here is the addition of a mapping from document to repo and storing the subrepo paths per repo. The other change is the addition of tombstones. A tombstoned repository is hidden from List and Search results. This was added so we can index a new version of a repository in a compound shard without the need of recomputing the whole shard. This commit is mostly focussed on the read path. It ensures everything keeps working correctly once compound shards are introduced. However, the write path is mostly missing. We add a merge command for manual merging. However, zoekt-sourcegraph-indexserver is mostly unaware of compound shards and has no way to mutate them. This will be follow-up work. To support compound shards we had to bump the indexed format version. This is since the repoMetaData field is not backwards compatible. However, we know we plan on making other changes to the index format. So we introduced NextIndexFormatVersion. With this change we will continue to use v16, unless a common opts into v17. This will allow us to effectively feature flag the new format while we work on it. Co-authored-by: Stefan Hengl <stefan@sourcegraph.com>

4 years ago

repo17_v17.00000.zoekt

ranking: incorporate file signals into BM25F (#922) This PR reworks the way we incorporate file signals into BM25. Previously, we were applying them as a tie-breaker. But in dogfooding, we found that these rarely impact results, because it's so rare to have a tie in BM25 scores. Now, we take the file signal into account when computing BM25F. The interpretation is that this data lives in a separate 'field' that is half the priority of regular content.

1 year ago

repo2_v16.00000.zoekt

Add support for indexing and searching custom fields for repositories (#962) At GitLab, we encountered limitations when searching within large namespaces containing thousands of repositories. Specifically, we cannot pass a complete list of RepoIDs due to size constraints. This change introduces support for indexing and searching on custom repository metadata by extending Repository to include an additional Metadata field. All fields within Repository.Metadata are searchable using a regular expression evaluator. This enables more scalable filtering by allowing clients to express regular expression prefix queries on metadata fields, such as: traversal_ids:123-456-.* Or any field really: haystack:nee.*le

1 year ago

repo_v16.00000.zoekt

Add support for indexing and searching custom fields for repositories (#962) At GitLab, we encountered limitations when searching within large namespaces containing thousands of repositories. Specifically, we cannot pass a complete list of RepoIDs due to size constraints. This change introduces support for indexing and searching on custom repository metadata by extending Repository to include an additional Metadata field. All fields within Repository.Metadata are searchable using a regular expression evaluator. This enables more scalable filtering by allowing clients to express regular expression prefix queries on metadata fields, such as: traversal_ids:123-456-.* Or any field really: haystack:nee.*le

1 year ago