zoekt/index/ at bc8f4ffaa611b638fc821bbcd553d6f99c8fa3ab · boltless.me/zoekt · Tangled

boltless.me / zoekt

0

fork of https://github.com/sourcegraph/zoekt

0

zoekt / index /

at bc8f4ffaa611b638fc821bbcd553d6f99c8fa3ab 39 files

Keegan Carruthers-Smith index: use ID based shard names only on multi-tenant instances (#956) 1y ago

index: use ID based shard names only on multi-tenant instances (#956) We only want to use ID based shard names on multi-tenant instances. Before this change we decided this based on if tenant enforcement was turned on. However, we would like to always have tenant enforcment on at Sourcegraph but not migrate all shards to the new naming scheme (just yet). This change is quite simple. However, it works since we never actually read the filenames to extract tenant ID or repository ID. We only ever use the data that is persisted inside of the shard to do the enforcement. The environment variable we read (WORKSPACES_API_URL) is the same one we use in the Sourcegraph codebase to determine multi-tenant specific code. Test Plan: Added a unit test. Will do a much larger manual E2E test with the Sourcegraph project.

1 year ago

Fix compilation on 32 bit architectures (#936) This PR fixes a bug where Zoekt would not compile on 32-bit architectures. It also takes the opportunity to start using the `math` library everywhere instead of our own constants like `maxUInt32` to help prevent this sort of issue in the future by encouraging devs to select the most accurate "max" type for their specific situation. Closes https://github.com/sourcegraph/zoekt/issues/935

1 year ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

index: use ID based shard names only on multi-tenant instances (#956) We only want to use ID based shard names on multi-tenant instances. Before this change we decided this based on if tenant enforcement was turned on. However, we would like to always have tenant enforcment on at Sourcegraph but not migrate all shards to the new naming scheme (just yet). This change is quite simple. However, it works since we never actually read the filenames to extract tenant ID or repository ID. We only ever use the data that is persisted inside of the shard to do the enforcement. The environment variable we read (WORKSPACES_API_URL) is the same one we use in the Sourcegraph codebase to determine multi-tenant specific code. Test Plan: Added a unit test. Will do a much larger manual E2E test with the Sourcegraph project.

1 year ago

builder_test.go

index: use ID based shard names only on multi-tenant instances (#956) We only want to use ID based shard names on multi-tenant instances. Before this change we decided this based on if tenant enforcement was turned on. However, we would like to always have tenant enforcment on at Sourcegraph but not migrate all shards to the new naming scheme (just yet). This change is quite simple. However, it works since we never actually read the filenames to extract tenant ID or repository ID. We only ever use the data that is persisted inside of the shard to do the enforcement. The environment variable we read (WORKSPACES_API_URL) is the same one we use in the Sourcegraph codebase to determine multi-tenant specific code. Test Plan: Added a unit test. Will do a much larger manual E2E test with the Sourcegraph project.

1 year ago

contentprovider.go

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

contentprovider_test.go

Move root-level index code to index package (#902) In the repo root, we have a bunch of low level logic around index building and searching. So we end up exposing internal logic through the main public `zoekt` package, for example `zoekt.Merge(...)`. This PR moves it into the `build` package, so all code related to index building lives together. It then renames `build` to `index` to reflect the broader focus on indexing and searching the index.

1 year ago

Move root-level index code to index package (#902) In the repo root, we have a bunch of low level logic around index building and searching. So we end up exposing internal logic through the main public `zoekt` package, for example `zoekt.Merge(...)`. This PR moves it into the `build` package, so all code related to index building lives together. It then renames `build` to `index` to reflect the broader focus on indexing and searching the index.

1 year ago

Move root-level index code to index package (#902) In the repo root, we have a bunch of low level logic around index building and searching. So we end up exposing internal logic through the main public `zoekt` package, for example `zoekt.Merge(...)`. This PR moves it into the `build` package, so all code related to index building lives together. It then renames `build` to `index` to reflect the broader focus on indexing and searching the index.

1 year ago

ranking: downweight binary files (#924) In testing, I noticed another problem with BM25: sometimes a binary file is ranked highly because of a match on its filename. In classic Zoekt scoring, these are ranked low because they are skipped, and we always sort skipped docs to the end of the index. This PR ensures they're also ranked low for BM25 by adding a 'binary' category, and marking it low priority. Adding this category required updating `SkipReason` to track the reason a document was skipped. This is necessary because we set the content of skipped docs to `nil`, and `SkipReason` is the only lasting indication that it was binary.

1 year ago

fix empty branch names getting returned by using uint64 id (#926) The gatherBranches() function shifts a mask of type uint64 until it is 0, but uses an id for accessing a map of type uint32. This results in empty branch names getting returned from this function. It can be fixed by changing the type of id to also be uint64.

1 year ago

refactor(all): goimports -w -local github.com/sourcegraph/zoekt (#948)

1 year ago

file_category.go

ranking: downweight binary files (#924) In testing, I noticed another problem with BM25: sometimes a binary file is ranked highly because of a match on its filename. In classic Zoekt scoring, these are ranked low because they are skipped, and we always sort skipped docs to the end of the index. This PR ensures they're also ranked low for BM25 by adding a 'binary' category, and marking it low priority. Adding this category required updating `SkipReason` to track the reason a document was skipped. This is necessary because we set the content of skipped docs to `nil`, and `SkipReason` is the only lasting indication that it was binary.

1 year ago

file_category_test.go

ranking: incorporate file signals into BM25F (#922) This PR reworks the way we incorporate file signals into BM25. Previously, we were applying them as a tie-breaker. But in dogfooding, we found that these rarely impact results, because it's so rare to have a tie in BM25 scores. Now, we take the file signal into account when computing BM25F. The interpretation is that this data lives in a separate 'field' that is half the priority of regular content.

1 year ago

Fix compilation on 32 bit architectures (#936) This PR fixes a bug where Zoekt would not compile on 32-bit architectures. It also takes the opportunity to start using the `math` library everywhere instead of our own constants like `maxUInt32` to help prevent this sort of issue in the future by encouraging devs to select the most accurate "max" type for their specific situation. Closes https://github.com/sourcegraph/zoekt/issues/935

1 year ago

hititer_test.go

refactor(all): goimports -w -local github.com/sourcegraph/zoekt (#948)

1 year ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

Fix compilation on 32 bit architectures (#936) This PR fixes a bug where Zoekt would not compile on 32-bit architectures. It also takes the opportunity to start using the `math` library everywhere instead of our own constants like `maxUInt32` to help prevent this sort of issue in the future by encouraging devs to select the most accurate "max" type for their specific situation. Closes https://github.com/sourcegraph/zoekt/issues/935

1 year ago

indexdata_test.go

Move root-level index code to index package (#902) In the repo root, we have a bunch of low level logic around index building and searching. So we end up exposing internal logic through the main public `zoekt` package, for example `zoekt.Merge(...)`. This PR moves it into the `build` package, so all code related to index building lives together. It then renames `build` to `index` to reflect the broader focus on indexing and searching the index.

1 year ago

Fix compilation on 32 bit architectures (#936) This PR fixes a bug where Zoekt would not compile on 32-bit architectures. It also takes the opportunity to start using the `math` library everywhere instead of our own constants like `maxUInt32` to help prevent this sort of issue in the future by encouraging devs to select the most accurate "max" type for their specific situation. Closes https://github.com/sourcegraph/zoekt/issues/935

1 year ago

Move root-level index code to index package (#902) In the repo root, we have a bunch of low level logic around index building and searching. So we end up exposing internal logic through the main public `zoekt` package, for example `zoekt.Merge(...)`. This PR moves it into the `build` package, so all code related to index building lives together. It then renames `build` to `index` to reflect the broader focus on indexing and searching the index.

1 year ago

refactor(all): goimports -w -local github.com/sourcegraph/zoekt (#948)

1 year ago

Fix compilation on 32 bit architectures (#936) This PR fixes a bug where Zoekt would not compile on 32-bit architectures. It also takes the opportunity to start using the `math` library everywhere instead of our own constants like `maxUInt32` to help prevent this sort of issue in the future by encouraging devs to select the most accurate "max" type for their specific situation. Closes https://github.com/sourcegraph/zoekt/issues/935

1 year ago

matchiter_test.go

Move root-level index code to index package (#902) In the repo root, we have a bunch of low level logic around index building and searching. So we end up exposing internal logic through the main public `zoekt` package, for example `zoekt.Merge(...)`. This PR moves it into the `build` package, so all code related to index building lives together. It then renames `build` to `index` to reflect the broader focus on indexing and searching the index.

1 year ago

refactor(all): goimports -w -local github.com/sourcegraph/zoekt (#948)

1 year ago

matchtree_test.go

refactor(all): goimports -w -local github.com/sourcegraph/zoekt (#948)

1 year ago

index: remove Options.{TenantID,RepoID} (#954) These are duplicated with RepositoryDescription. There is exactly one place we set these values, and in that case we set it in the repository description as well. Note: we update some the calls in builder.go for o.IndexOptions.RepoID to be just o.RepoID. This is to make it consistent with how we access TenantID. IndexOptions is embedded in that struct (indexArgs) which is why we can do the shortcut. Test Plan: go test is sufficient for this change. I will do a larger round of manual tests soon with all my changes stacked.

1 year ago

refactor(all): goimports -w -local github.com/sourcegraph/zoekt (#948)

1 year ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

index: decide between tenant and non-tenant shard name in one place (#953) We have two places with duplicated logic around how it decides the layout of shards on disk. This now moves that decision into one place. Additionally we can now unexport index.ShardName. It was only used in one place outside the package, and that was easy to replace with a hardcoded string since it is just a test. Test Plan: Just CI. This has no actual change in functionality, just refactoring.

1 year ago

refactor(all): goimports -w -local github.com/sourcegraph/zoekt (#948)

1 year ago

Move root-level index code to index package (#902) In the repo root, we have a bunch of low level logic around index building and searching. So we end up exposing internal logic through the main public `zoekt` package, for example `zoekt.Merge(...)`. This PR moves it into the `build` package, so all code related to index building lives together. It then renames `build` to `index` to reflect the broader focus on indexing and searching the index.

1 year ago

shard_builder.go

index: decide between tenant and non-tenant shard name in one place (#953) We have two places with duplicated logic around how it decides the layout of shards on disk. This now moves that decision into one place. Additionally we can now unexport index.ShardName. It was only used in one place outside the package, and that was easy to replace with a hardcoded string since it is just a test. Test Plan: Just CI. This has no actual change in functionality, just refactoring.

1 year ago

shard_builder_test.go

index: decide between tenant and non-tenant shard name in one place (#953) We have two places with duplicated logic around how it decides the layout of shards on disk. This now moves that decision into one place. Additionally we can now unexport index.ShardName. It was only used in one place outside the package, and that was easy to replace with a hardcoded string since it is just a test. Test Plan: Just CI. This has no actual change in functionality, just refactoring.

1 year ago

ranking: incorporate file signals into BM25F (#922) This PR reworks the way we incorporate file signals into BM25. Previously, we were applying them as a tie-breaker. But in dogfooding, we found that these rarely impact results, because it's so rare to have a tie in BM25 scores. Now, we take the file signal into account when computing BM25F. The interpretation is that this data lives in a separate 'field' that is half the priority of regular content.

1 year ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

tombstones_test.go

Move root-level index code to index package (#902) In the repo root, we have a bunch of low level logic around index building and searching. So we end up exposing internal logic through the main public `zoekt` package, for example `zoekt.Merge(...)`. This PR moves it into the `build` package, so all code related to index building lives together. It then renames `build` to `index` to reflect the broader focus on indexing and searching the index.

1 year ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago