zoekt/cmd/zoekt-sourcegraph-indexserver/ at 5baf7e20587e1b6ad0195742e439494b91aa1cb7 · boltless.me/zoekt · Tangled

boltless.me / zoekt

0

fork of https://github.com/sourcegraph/zoekt

0

zoekt / cmd / zoekt-sourcegraph-indexserver /

at 5baf7e20587e1b6ad0195742e439494b91aa1cb7 2 folders 24 files

Keegan Carruthers-Smith indexserver: integration test for sourcegraph (#1033) 2mo ago

sourcegraph: remove GRPC index methods (#977) This was part of an effort to move the queue from Zoekt to Sourcegraph. However, we are not going to pursue this for now and we can remove the corresponging grpc methods. Sourcegraph never had callers of Index and Delete, so both are save to remove. `DeleteAllData` is still being called from Sourcegraph. Test plan: CI

10 months ago

zoekt-sourcegraph-indexserver: add gRPC support to sourcegraph client (#551) Co-authored-by: Camden Cheek <camden@ccheek.com>

3 years ago

indexserver: integration test for sourcegraph (#1033)

2 months ago

backoff_test.go

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

refactor(all): goimports -w -local github.com/sourcegraph/zoekt (#948)

1 year ago

cleanup_test.go

index: decide between tenant and non-tenant shard name in one place (#953) We have two places with duplicated logic around how it decides the layout of shards on disk. This now moves that decision into one place. Additionally we can now unexport index.ShardName. It was only used in one place outside the package, and that was easy to replace with a hardcoded string since it is just a test. Test Plan: Just CI. This has no actual change in functionality, just refactoring.

1 year ago

sourcegraph-indexserver: add GRPC methods for Index and Delete (#933) Closes SPLF-913 This only affects Sourcegraph. This adds Index and Delete methods to sourcegraph-indexserver's GRPC server. The methods aren't called yet by Sourcegraph. Notable differences to our current indexing loop: - IndexOptions are pushed by the client instead of pulled by Zoekt - A semaphore restricts concurrency - New ENV `SRC_STOP_INDEXING` stops the "competing" indexing loop. This is just for testing and will eventually be removed - The Index method tries to recover a possibly older index from `indexDir/.trash` first before starting to index Review: I recommend reviewing the `.proto` file first, followed by `main.go` Test plan: - I did a bunch of manual testing to test the trash recovery and the semaphore logic. I successfuly triggered index and delete calls to the GRPC server. I will put the Insomnia collection I used for testing in the ticket. - New unit tests

1 year ago

default_grpc_service_configuration.json

zoekt-sourcegraph-indexserver: add gRPC support to sourcegraph client (#551) Co-authored-by: Camden Cheek <camden@ccheek.com>

3 years ago

indexserver: set blob:limit based on FileLimit (#1027) Previously we hardcoded this value to 1mb (which aligns with what is set in Sourcegraph). But we might as well set it based on the limit that is passed in. Additionally we set the special HTTP headers sourcegraph needs in the config so that when git double checks if it needs to hydrate an object it can do it successfully.

3 months ago

index_integration_test.go

indexserver: integration test for sourcegraph (#1033)

2 months ago

indexserver: only log if someone else is running (#395) We inverted the condition around alreadyRunning. Luckily the only side-effect was extra debug logs. I noticed this while testing with debug logs on. Test Plan: Run with debug logs and notice the already running messages disappear. Add in a sleep for 2 minutes in indexing then check already running messages pop up. SRC_LOG_LEVEL=dbug go run ./cmd/zoekt-sourcegraph-indexserver \ -listen 127.0.0.1:6072 \ -index_concurrency 4 \ -sourcegraph_url ~/src \ -index /tmp/zoekt-test plz-review-url: https://plz.review/review/6899

4 years ago

indexserver: set blob:limit based on FileLimit (#1027) Previously we hardcoded this value to 1mb (which aligns with what is set in Sourcegraph). But we might as well set it based on the limit that is passed in. Additionally we set the special HTTP headers sourcegraph needs in the config so that when git double checks if it needs to hydrate an object it can do it successfully.

3 months ago

indexserver: set blob:limit based on FileLimit (#1027) Previously we hardcoded this value to 1mb (which aligns with what is set in Sourcegraph). But we might as well set it based on the limit that is passed in. Additionally we set the special HTTP headers sourcegraph needs in the config so that when git double checks if it needs to hydrate an object it can do it successfully.

3 months ago

indexserver: set blob:limit based on FileLimit (#1027) Previously we hardcoded this value to 1mb (which aligns with what is set in Sourcegraph). But we might as well set it based on the limit that is passed in. Additionally we set the special HTTP headers sourcegraph needs in the config so that when git double checks if it needs to hydrate an object it can do it successfully.

3 months ago

refactor(all): goimports -w -local github.com/sourcegraph/zoekt (#948)

1 year ago

sourcegraph-indexserver: GRPC, implement DeleteAllData (#923) Relates to #920 Relates to SPLF-874 This implements DeleteAllData. We hold the global lock while deleting all simple shards belonging to a tenant. We also handle compound shards by disassembling them first. Note that this "only" deletes persisted data. Updating the queue, for example, seems fragile because it might immediately get updated by Sourcegraph. This implies that Sourcegraph first has to delete the tenant in the Sourcegraph DB first and then call this new endpoint. Even if the queue still has a reference to a deleted tenant, indexserver won't be able to retrieve index options or clone the repo from gitserver. Test plan: - new unit tests - manual testing: I ran this together with Sourcegraph and triggered a delete by calling DeleteAllData directly. I confirmed that all shards, including compound shards are deleted.

1 year ago

refactor(all): goimports -w -local github.com/sourcegraph/zoekt (#948)

1 year ago

refactor(all): goimports -w -local github.com/sourcegraph/zoekt (#948)

1 year ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

all: gofumpt -l -w . gofumpt is a stricter gofmt. I took a look at the changes and in general they are nice. I don't think we need to enforce the use of gofumpt, but I like the idea of running it every once in a while. Test Plan: go test ./...

2 years ago

sourcegraph-indexserver: GRPC, implement DeleteAllData (#923) Relates to #920 Relates to SPLF-874 This implements DeleteAllData. We hold the global lock while deleting all simple shards belonging to a tenant. We also handle compound shards by disassembling them first. Note that this "only" deletes persisted data. Updating the queue, for example, seems fragile because it might immediately get updated by Sourcegraph. This implies that Sourcegraph first has to delete the tenant in the Sourcegraph DB first and then call this new endpoint. Even if the queue still has a reference to a deleted tenant, indexserver won't be able to retrieve index options or clone the repo from gitserver. Test plan: - new unit tests - manual testing: I ran this together with Sourcegraph and triggered a delete by calling DeleteAllData directly. I confirmed that all shards, including compound shards are deleted.

1 year ago

refactor(all): goimports -w -local github.com/sourcegraph/zoekt (#948)

1 year ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

ci: fix fmt string errors (#994) I think these snuck in as we changed go versions

7 months ago

sourcegraph-indexserver: add GRPC methods for Index and Delete (#933) Closes SPLF-913 This only affects Sourcegraph. This adds Index and Delete methods to sourcegraph-indexserver's GRPC server. The methods aren't called yet by Sourcegraph. Notable differences to our current indexing loop: - IndexOptions are pushed by the client instead of pulled by Zoekt - A semaphore restricts concurrency - New ENV `SRC_STOP_INDEXING` stops the "competing" indexing loop. This is just for testing and will eventually be removed - The Index method tries to recover a possibly older index from `indexDir/.trash` first before starting to index Review: I recommend reviewing the `.proto` file first, followed by `main.go` Test plan: - I did a bunch of manual testing to test the trash recovery and the semaphore logic. I successfuly triggered index and delete calls to the GRPC server. I will put the Insomnia collection I used for testing in the ticket. - New unit tests

1 year ago

ranking: removing document ranks (#853) Document ranks was an experimental feature of Sourcegraph. We have already removed the code in Sourcegraph in the last release. This is the corresponding cleanup for Zoekt. Test plan: updated tests

2 years ago