zoekt/cmd/zoekt-sourcegraph-indexserver/ at bc8f4ffaa611b638fc821bbcd553d6f99c8fa3ab · boltless.me/zoekt · Tangled

boltless.me / zoekt

0

fork of https://github.com/sourcegraph/zoekt

0

zoekt / cmd / zoekt-sourcegraph-indexserver /

at bc8f4ffaa611b638fc821bbcd553d6f99c8fa3ab 2 folders 23 files

Keegan Carruthers-Smith index: remove Options.{TenantID,RepoID} (#954) 1y ago

Add README for proto directories (#943) Somehow I just found out about `gen-proto.sh` recently, after many months of working on Zoekt! This PR adds brief `README` files in the proto directories to guide people (and LLMs) to it.

1 year ago

zoekt-sourcegraph-indexserver: add gRPC support to sourcegraph client (#551) Co-authored-by: Camden Cheek <camden@ccheek.com>

3 years ago

index: remove Options.{TenantID,RepoID} (#954) These are duplicated with RepositoryDescription. There is exactly one place we set these values, and in that case we set it in the repository description as well. Note: we update some the calls in builder.go for o.IndexOptions.RepoID to be just o.RepoID. This is to make it consistent with how we access TenantID. IndexOptions is embedded in that struct (indexArgs) which is why we can do the shortcut. Test Plan: go test is sufficient for this change. I will do a larger round of manual tests soon with all my changes stacked.

1 year ago

backoff_test.go

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

refactor(all): goimports -w -local github.com/sourcegraph/zoekt (#948)

1 year ago

cleanup_test.go

index: decide between tenant and non-tenant shard name in one place (#953) We have two places with duplicated logic around how it decides the layout of shards on disk. This now moves that decision into one place. Additionally we can now unexport index.ShardName. It was only used in one place outside the package, and that was easy to replace with a hardcoded string since it is just a test. Test Plan: Just CI. This has no actual change in functionality, just refactoring.

1 year ago

sourcegraph-indexserver: add GRPC methods for Index and Delete (#933) Closes SPLF-913 This only affects Sourcegraph. This adds Index and Delete methods to sourcegraph-indexserver's GRPC server. The methods aren't called yet by Sourcegraph. Notable differences to our current indexing loop: - IndexOptions are pushed by the client instead of pulled by Zoekt - A semaphore restricts concurrency - New ENV `SRC_STOP_INDEXING` stops the "competing" indexing loop. This is just for testing and will eventually be removed - The Index method tries to recover a possibly older index from `indexDir/.trash` first before starting to index Review: I recommend reviewing the `.proto` file first, followed by `main.go` Test plan: - I did a bunch of manual testing to test the trash recovery and the semaphore logic. I successfuly triggered index and delete calls to the GRPC server. I will put the Insomnia collection I used for testing in the ticket. - New unit tests

1 year ago

default_grpc_service_configuration.json

zoekt-sourcegraph-indexserver: add gRPC support to sourcegraph client (#551) Co-authored-by: Camden Cheek <camden@ccheek.com>

3 years ago

index: remove Options.{TenantID,RepoID} (#954) These are duplicated with RepositoryDescription. There is exactly one place we set these values, and in that case we set it in the repository description as well. Note: we update some the calls in builder.go for o.IndexOptions.RepoID to be just o.RepoID. This is to make it consistent with how we access TenantID. IndexOptions is embedded in that struct (indexArgs) which is why we can do the shortcut. Test Plan: go test is sufficient for this change. I will do a larger round of manual tests soon with all my changes stacked.

1 year ago

indexserver: only log if someone else is running (#395) We inverted the condition around alreadyRunning. Luckily the only side-effect was extra debug logs. I noticed this while testing with debug logs on. Test Plan: Run with debug logs and notice the already running messages disappear. Add in a sleep for 2 minutes in indexing then check already running messages pop up. SRC_LOG_LEVEL=dbug go run ./cmd/zoekt-sourcegraph-indexserver \ -listen 127.0.0.1:6072 \ -index_concurrency 4 \ -sourcegraph_url ~/src \ -index /tmp/zoekt-test plz-review-url: https://plz.review/review/6899

4 years ago

index: remove shard_prefix option (#947) We instead rely on the environment which is inheritted to make this decision. The logic will become a little different in the next PR which will instead decide layout based on if we are in "workspaces" mode vs just wanting to enforce tenants. Note: this is Sourcegraph specific. We want to move to always enforcing tenant in our environments, even outside of our multitenant environments. Test Plan: go test

1 year ago

sourcegraph-indexserver: add GRPC methods for Index and Delete (#933) Closes SPLF-913 This only affects Sourcegraph. This adds Index and Delete methods to sourcegraph-indexserver's GRPC server. The methods aren't called yet by Sourcegraph. Notable differences to our current indexing loop: - IndexOptions are pushed by the client instead of pulled by Zoekt - A semaphore restricts concurrency - New ENV `SRC_STOP_INDEXING` stops the "competing" indexing loop. This is just for testing and will eventually be removed - The Index method tries to recover a possibly older index from `indexDir/.trash` first before starting to index Review: I recommend reviewing the `.proto` file first, followed by `main.go` Test plan: - I did a bunch of manual testing to test the trash recovery and the semaphore logic. I successfuly triggered index and delete calls to the GRPC server. I will put the Insomnia collection I used for testing in the ticket. - New unit tests

1 year ago

refactor(all): goimports -w -local github.com/sourcegraph/zoekt (#948)

1 year ago

refactor(all): goimports -w -local github.com/sourcegraph/zoekt (#948)

1 year ago

sourcegraph-indexserver: GRPC, implement DeleteAllData (#923) Relates to #920 Relates to SPLF-874 This implements DeleteAllData. We hold the global lock while deleting all simple shards belonging to a tenant. We also handle compound shards by disassembling them first. Note that this "only" deletes persisted data. Updating the queue, for example, seems fragile because it might immediately get updated by Sourcegraph. This implies that Sourcegraph first has to delete the tenant in the Sourcegraph DB first and then call this new endpoint. Even if the queue still has a reference to a deleted tenant, indexserver won't be able to retrieve index options or clone the repo from gitserver. Test plan: - new unit tests - manual testing: I ran this together with Sourcegraph and triggered a delete by calling DeleteAllData directly. I confirmed that all shards, including compound shards are deleted.

1 year ago

refactor(all): goimports -w -local github.com/sourcegraph/zoekt (#948)

1 year ago

refactor(all): goimports -w -local github.com/sourcegraph/zoekt (#948)

1 year ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

all: gofumpt -l -w . gofumpt is a stricter gofmt. I took a look at the changes and in general they are nice. I don't think we need to enforce the use of gofumpt, but I like the idea of running it every once in a while. Test Plan: go test ./...

2 years ago

sourcegraph-indexserver: GRPC, implement DeleteAllData (#923) Relates to #920 Relates to SPLF-874 This implements DeleteAllData. We hold the global lock while deleting all simple shards belonging to a tenant. We also handle compound shards by disassembling them first. Note that this "only" deletes persisted data. Updating the queue, for example, seems fragile because it might immediately get updated by Sourcegraph. This implies that Sourcegraph first has to delete the tenant in the Sourcegraph DB first and then call this new endpoint. Even if the queue still has a reference to a deleted tenant, indexserver won't be able to retrieve index options or clone the repo from gitserver. Test plan: - new unit tests - manual testing: I ran this together with Sourcegraph and triggered a delete by calling DeleteAllData directly. I confirmed that all shards, including compound shards are deleted.

1 year ago

refactor(all): goimports -w -local github.com/sourcegraph/zoekt (#948)

1 year ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

refactor(all): goimports -w -local github.com/sourcegraph/zoekt (#948)

1 year ago

sourcegraph-indexserver: add GRPC methods for Index and Delete (#933) Closes SPLF-913 This only affects Sourcegraph. This adds Index and Delete methods to sourcegraph-indexserver's GRPC server. The methods aren't called yet by Sourcegraph. Notable differences to our current indexing loop: - IndexOptions are pushed by the client instead of pulled by Zoekt - A semaphore restricts concurrency - New ENV `SRC_STOP_INDEXING` stops the "competing" indexing loop. This is just for testing and will eventually be removed - The Index method tries to recover a possibly older index from `indexDir/.trash` first before starting to index Review: I recommend reviewing the `.proto` file first, followed by `main.go` Test plan: - I did a bunch of manual testing to test the trash recovery and the semaphore logic. I successfuly triggered index and delete calls to the GRPC server. I will put the Insomnia collection I used for testing in the ticket. - New unit tests

1 year ago

ranking: removing document ranks (#853) Document ranks was an experimental feature of Sourcegraph. We have already removed the code in Sourcegraph in the last release. This is the corresponding cleanup for Zoekt. Test plan: updated tests

2 years ago