zoekt/cmd/zoekt-sourcegraph-indexserver/ at 75879a727dd21041fd8485106bf93e75be5ef028 · boltless.me/zoekt · Tangled

boltless.me / zoekt

0

fork of https://github.com/sourcegraph/zoekt

0

zoekt / cmd / zoekt-sourcegraph-indexserver /

at 75879a727dd21041fd8485106bf93e75be5ef028 2 folders 23 files

Julie Tibshirani Add README for proto directories (#943) 1y ago

Add README for proto directories (#943) Somehow I just found out about `gen-proto.sh` recently, after many months of working on Zoekt! This PR adds brief `README` files in the proto directories to guide people (and LLMs) to it.

1 year ago

zoekt-sourcegraph-indexserver: add gRPC support to sourcegraph client (#551) Co-authored-by: Camden Cheek <camden@ccheek.com>

3 years ago

Add README for proto directories (#943) Somehow I just found out about `gen-proto.sh` recently, after many months of working on Zoekt! This PR adds brief `README` files in the proto directories to guide people (and LLMs) to it.

1 year ago

backoff_test.go

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

Move root-level index code to index package (#902) In the repo root, we have a bunch of low level logic around index building and searching. So we end up exposing internal logic through the main public `zoekt` package, for example `zoekt.Merge(...)`. This PR moves it into the `build` package, so all code related to index building lives together. It then renames `build` to `index` to reflect the broader focus on indexing and searching the index.

1 year ago

cleanup_test.go

Rename IndexBuilder -> ShardBuilder (#908) When navigating the code, I've often forgotten the difference between `NewBuilder` and `NewIndexBuilder`. This rename clarifies that one of these indexes a whole repo, while the other builds individual shards. Also `index.NewShardBuilder` sounds better.

1 year ago

sourcegraph-indexserver: add GRPC methods for Index and Delete (#933) Closes SPLF-913 This only affects Sourcegraph. This adds Index and Delete methods to sourcegraph-indexserver's GRPC server. The methods aren't called yet by Sourcegraph. Notable differences to our current indexing loop: - IndexOptions are pushed by the client instead of pulled by Zoekt - A semaphore restricts concurrency - New ENV `SRC_STOP_INDEXING` stops the "competing" indexing loop. This is just for testing and will eventually be removed - The Index method tries to recover a possibly older index from `indexDir/.trash` first before starting to index Review: I recommend reviewing the `.proto` file first, followed by `main.go` Test plan: - I did a bunch of manual testing to test the trash recovery and the semaphore logic. I successfuly triggered index and delete calls to the GRPC server. I will put the Insomnia collection I used for testing in the ticket. - New unit tests

1 year ago

default_grpc_service_configuration.json

zoekt-sourcegraph-indexserver: add gRPC support to sourcegraph client (#551) Co-authored-by: Camden Cheek <camden@ccheek.com>

3 years ago

Fix compilation on 32 bit architectures (#936) This PR fixes a bug where Zoekt would not compile on 32-bit architectures. It also takes the opportunity to start using the `math` library everywhere instead of our own constants like `maxUInt32` to help prevent this sort of issue in the future by encouraging devs to select the most accurate "max" type for their specific situation. Closes https://github.com/sourcegraph/zoekt/issues/935

1 year ago

indexserver: only log if someone else is running (#395) We inverted the condition around alreadyRunning. Luckily the only side-effect was extra debug logs. I noticed this while testing with debug logs on. Test Plan: Run with debug logs and notice the already running messages disappear. Add in a sleep for 2 minutes in indexing then check already running messages pop up. SRC_LOG_LEVEL=dbug go run ./cmd/zoekt-sourcegraph-indexserver \ -listen 127.0.0.1:6072 \ -index_concurrency 4 \ -sourcegraph_url ~/src \ -index /tmp/zoekt-test plz-review-url: https://plz.review/review/6899

4 years ago

sourcegraph-indexserver: add GRPC methods for Index and Delete (#933) Closes SPLF-913 This only affects Sourcegraph. This adds Index and Delete methods to sourcegraph-indexserver's GRPC server. The methods aren't called yet by Sourcegraph. Notable differences to our current indexing loop: - IndexOptions are pushed by the client instead of pulled by Zoekt - A semaphore restricts concurrency - New ENV `SRC_STOP_INDEXING` stops the "competing" indexing loop. This is just for testing and will eventually be removed - The Index method tries to recover a possibly older index from `indexDir/.trash` first before starting to index Review: I recommend reviewing the `.proto` file first, followed by `main.go` Test plan: - I did a bunch of manual testing to test the trash recovery and the semaphore logic. I successfuly triggered index and delete calls to the GRPC server. I will put the Insomnia collection I used for testing in the ticket. - New unit tests

1 year ago

sourcegraph-indexserver: add GRPC methods for Index and Delete (#933) Closes SPLF-913 This only affects Sourcegraph. This adds Index and Delete methods to sourcegraph-indexserver's GRPC server. The methods aren't called yet by Sourcegraph. Notable differences to our current indexing loop: - IndexOptions are pushed by the client instead of pulled by Zoekt - A semaphore restricts concurrency - New ENV `SRC_STOP_INDEXING` stops the "competing" indexing loop. This is just for testing and will eventually be removed - The Index method tries to recover a possibly older index from `indexDir/.trash` first before starting to index Review: I recommend reviewing the `.proto` file first, followed by `main.go` Test plan: - I did a bunch of manual testing to test the trash recovery and the semaphore logic. I successfuly triggered index and delete calls to the GRPC server. I will put the Insomnia collection I used for testing in the ticket. - New unit tests

1 year ago

sourcegraph-indexserver: add GRPC methods for Index and Delete (#933) Closes SPLF-913 This only affects Sourcegraph. This adds Index and Delete methods to sourcegraph-indexserver's GRPC server. The methods aren't called yet by Sourcegraph. Notable differences to our current indexing loop: - IndexOptions are pushed by the client instead of pulled by Zoekt - A semaphore restricts concurrency - New ENV `SRC_STOP_INDEXING` stops the "competing" indexing loop. This is just for testing and will eventually be removed - The Index method tries to recover a possibly older index from `indexDir/.trash` first before starting to index Review: I recommend reviewing the `.proto` file first, followed by `main.go` Test plan: - I did a bunch of manual testing to test the trash recovery and the semaphore logic. I successfuly triggered index and delete calls to the GRPC server. I will put the Insomnia collection I used for testing in the ticket. - New unit tests

1 year ago

sourcegraph-indexserver: GRPC, implement DeleteAllData (#923) Relates to #920 Relates to SPLF-874 This implements DeleteAllData. We hold the global lock while deleting all simple shards belonging to a tenant. We also handle compound shards by disassembling them first. Note that this "only" deletes persisted data. Updating the queue, for example, seems fragile because it might immediately get updated by Sourcegraph. This implies that Sourcegraph first has to delete the tenant in the Sourcegraph DB first and then call this new endpoint. Even if the queue still has a reference to a deleted tenant, indexserver won't be able to retrieve index options or clone the repo from gitserver. Test plan: - new unit tests - manual testing: I ran this together with Sourcegraph and triggered a delete by calling DeleteAllData directly. I confirmed that all shards, including compound shards are deleted.

1 year ago

sourcegraph-indexserver: GRPC, implement DeleteAllData (#923) Relates to #920 Relates to SPLF-874 This implements DeleteAllData. We hold the global lock while deleting all simple shards belonging to a tenant. We also handle compound shards by disassembling them first. Note that this "only" deletes persisted data. Updating the queue, for example, seems fragile because it might immediately get updated by Sourcegraph. This implies that Sourcegraph first has to delete the tenant in the Sourcegraph DB first and then call this new endpoint. Even if the queue still has a reference to a deleted tenant, indexserver won't be able to retrieve index options or clone the repo from gitserver. Test plan: - new unit tests - manual testing: I ran this together with Sourcegraph and triggered a delete by calling DeleteAllData directly. I confirmed that all shards, including compound shards are deleted.

1 year ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

Move root-level index code to index package (#902) In the repo root, we have a bunch of low level logic around index building and searching. So we end up exposing internal logic through the main public `zoekt` package, for example `zoekt.Merge(...)`. This PR moves it into the `build` package, so all code related to index building lives together. It then renames `build` to `index` to reflect the broader focus on indexing and searching the index.

1 year ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

all: gofumpt -l -w . gofumpt is a stricter gofmt. I took a look at the changes and in general they are nice. I don't think we need to enforce the use of gofumpt, but I like the idea of running it every once in a while. Test Plan: go test ./...

2 years ago

sourcegraph-indexserver: GRPC, implement DeleteAllData (#923) Relates to #920 Relates to SPLF-874 This implements DeleteAllData. We hold the global lock while deleting all simple shards belonging to a tenant. We also handle compound shards by disassembling them first. Note that this "only" deletes persisted data. Updating the queue, for example, seems fragile because it might immediately get updated by Sourcegraph. This implies that Sourcegraph first has to delete the tenant in the Sourcegraph DB first and then call this new endpoint. Even if the queue still has a reference to a deleted tenant, indexserver won't be able to retrieve index options or clone the repo from gitserver. Test plan: - new unit tests - manual testing: I ran this together with Sourcegraph and triggered a delete by calling DeleteAllData directly. I confirmed that all shards, including compound shards are deleted.

1 year ago

sourcegraph-indexserver: GRPC, implement DeleteAllData (#923) Relates to #920 Relates to SPLF-874 This implements DeleteAllData. We hold the global lock while deleting all simple shards belonging to a tenant. We also handle compound shards by disassembling them first. Note that this "only" deletes persisted data. Updating the queue, for example, seems fragile because it might immediately get updated by Sourcegraph. This implies that Sourcegraph first has to delete the tenant in the Sourcegraph DB first and then call this new endpoint. Even if the queue still has a reference to a deleted tenant, indexserver won't be able to retrieve index options or clone the repo from gitserver. Test plan: - new unit tests - manual testing: I ran this together with Sourcegraph and triggered a delete by calling DeleteAllData directly. I confirmed that all shards, including compound shards are deleted.

1 year ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

sourcegraph-indexserver: add GRPC methods for Index and Delete (#933) Closes SPLF-913 This only affects Sourcegraph. This adds Index and Delete methods to sourcegraph-indexserver's GRPC server. The methods aren't called yet by Sourcegraph. Notable differences to our current indexing loop: - IndexOptions are pushed by the client instead of pulled by Zoekt - A semaphore restricts concurrency - New ENV `SRC_STOP_INDEXING` stops the "competing" indexing loop. This is just for testing and will eventually be removed - The Index method tries to recover a possibly older index from `indexDir/.trash` first before starting to index Review: I recommend reviewing the `.proto` file first, followed by `main.go` Test plan: - I did a bunch of manual testing to test the trash recovery and the semaphore logic. I successfuly triggered index and delete calls to the GRPC server. I will put the Insomnia collection I used for testing in the ticket. - New unit tests

1 year ago

ranking: removing document ranks (#853) Document ranks was an experimental feature of Sourcegraph. We have already removed the code in Sourcegraph in the last release. This is the corresponding cleanup for Zoekt. Test plan: updated tests

2 years ago