zoekt/cmd/zoekt-sourcegraph-indexserver/ at 47287adecfbd6b068cf7ad5fb67c9548a7062156 · boltless.me/zoekt · Tangled

boltless.me / zoekt

0

fork of https://github.com/sourcegraph/zoekt

0

zoekt / cmd / zoekt-sourcegraph-indexserver /

at 47287adecfbd6b068cf7ad5fb67c9548a7062156 2 folders 23 files

Keegan Carruthers-Smith all: run modernize across codebase (#919) 1y ago

sourcegraph-indexserver: add grpc server (#920) Relates to SPLF-874 This adds a grpc server to sourcegraph-indexserver. For now it supports just one method. The diff is quite big, so I left comments to mark the most important bits. I used the opportunity to clean up a bit (=> hence the big diff): - Reuse grpc logic from webserver and move those bits to "/gprc/..." - Move "protos" inside the new "grpc" directory (=> requires changes of import statements in Sourcegraph) - Refactor import aliases for grpc packages across the codebase Test plan: I tested this locally by calling the new grpc endpoint directly.

1 year ago

zoekt-sourcegraph-indexserver: add gRPC support to sourcegraph client (#551) Co-authored-by: Camden Cheek <camden@ccheek.com>

3 years ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

backoff_test.go

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

Move root-level index code to index package (#902) In the repo root, we have a bunch of low level logic around index building and searching. So we end up exposing internal logic through the main public `zoekt` package, for example `zoekt.Merge(...)`. This PR moves it into the `build` package, so all code related to index building lives together. It then renames `build` to `index` to reflect the broader focus on indexing and searching the index.

1 year ago

cleanup_test.go

Rename IndexBuilder -> ShardBuilder (#908) When navigating the code, I've often forgotten the difference between `NewBuilder` and `NewIndexBuilder`. This rename clarifies that one of these indexes a whole repo, while the other builds individual shards. Also `index.NewShardBuilder` sounds better.

1 year ago

all: set severity hint in most log output (#871) I was inspecting logs in GKE and it incorrectly categorized the severity of nearly all logs from zoekt-webserver and zoekt-indexserver. This is a hack to make it work better without putting in the bigger work of migrating us to structured logging. Test Plan: go test

2 years ago

default_grpc_service_configuration.json

zoekt-sourcegraph-indexserver: add gRPC support to sourcegraph client (#551) Co-authored-by: Camden Cheek <camden@ccheek.com>

3 years ago

Move root-level index code to index package (#902) In the repo root, we have a bunch of low level logic around index building and searching. So we end up exposing internal logic through the main public `zoekt` package, for example `zoekt.Merge(...)`. This PR moves it into the `build` package, so all code related to index building lives together. It then renames `build` to `index` to reflect the broader focus on indexing and searching the index.

1 year ago

indexserver: only log if someone else is running (#395) We inverted the condition around alreadyRunning. Luckily the only side-effect was extra debug logs. I noticed this while testing with debug logs on. Test Plan: Run with debug logs and notice the already running messages disappear. Add in a sleep for 2 minutes in indexing then check already running messages pop up. SRC_LOG_LEVEL=dbug go run ./cmd/zoekt-sourcegraph-indexserver \ -listen 127.0.0.1:6072 \ -index_concurrency 4 \ -sourcegraph_url ~/src \ -index /tmp/zoekt-test plz-review-url: https://plz.review/review/6899

4 years ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

sourcegraph-indexserver: GRPC, implement DeleteAllData (#923) Relates to #920 Relates to SPLF-874 This implements DeleteAllData. We hold the global lock while deleting all simple shards belonging to a tenant. We also handle compound shards by disassembling them first. Note that this "only" deletes persisted data. Updating the queue, for example, seems fragile because it might immediately get updated by Sourcegraph. This implies that Sourcegraph first has to delete the tenant in the Sourcegraph DB first and then call this new endpoint. Even if the queue still has a reference to a deleted tenant, indexserver won't be able to retrieve index options or clone the repo from gitserver. Test plan: - new unit tests - manual testing: I ran this together with Sourcegraph and triggered a delete by calling DeleteAllData directly. I confirmed that all shards, including compound shards are deleted.

1 year ago

sourcegraph-indexserver: GRPC, implement DeleteAllData (#923) Relates to #920 Relates to SPLF-874 This implements DeleteAllData. We hold the global lock while deleting all simple shards belonging to a tenant. We also handle compound shards by disassembling them first. Note that this "only" deletes persisted data. Updating the queue, for example, seems fragile because it might immediately get updated by Sourcegraph. This implies that Sourcegraph first has to delete the tenant in the Sourcegraph DB first and then call this new endpoint. Even if the queue still has a reference to a deleted tenant, indexserver won't be able to retrieve index options or clone the repo from gitserver. Test plan: - new unit tests - manual testing: I ran this together with Sourcegraph and triggered a delete by calling DeleteAllData directly. I confirmed that all shards, including compound shards are deleted.

1 year ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

Move root-level index code to index package (#902) In the repo root, we have a bunch of low level logic around index building and searching. So we end up exposing internal logic through the main public `zoekt` package, for example `zoekt.Merge(...)`. This PR moves it into the `build` package, so all code related to index building lives together. It then renames `build` to `index` to reflect the broader focus on indexing and searching the index.

1 year ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

all: gofumpt -l -w . gofumpt is a stricter gofmt. I took a look at the changes and in general they are nice. I don't think we need to enforce the use of gofumpt, but I like the idea of running it every once in a while. Test Plan: go test ./...

2 years ago

sourcegraph-indexserver: GRPC, implement DeleteAllData (#923) Relates to #920 Relates to SPLF-874 This implements DeleteAllData. We hold the global lock while deleting all simple shards belonging to a tenant. We also handle compound shards by disassembling them first. Note that this "only" deletes persisted data. Updating the queue, for example, seems fragile because it might immediately get updated by Sourcegraph. This implies that Sourcegraph first has to delete the tenant in the Sourcegraph DB first and then call this new endpoint. Even if the queue still has a reference to a deleted tenant, indexserver won't be able to retrieve index options or clone the repo from gitserver. Test plan: - new unit tests - manual testing: I ran this together with Sourcegraph and triggered a delete by calling DeleteAllData directly. I confirmed that all shards, including compound shards are deleted.

1 year ago

sourcegraph-indexserver: GRPC, implement DeleteAllData (#923) Relates to #920 Relates to SPLF-874 This implements DeleteAllData. We hold the global lock while deleting all simple shards belonging to a tenant. We also handle compound shards by disassembling them first. Note that this "only" deletes persisted data. Updating the queue, for example, seems fragile because it might immediately get updated by Sourcegraph. This implies that Sourcegraph first has to delete the tenant in the Sourcegraph DB first and then call this new endpoint. Even if the queue still has a reference to a deleted tenant, indexserver won't be able to retrieve index options or clone the repo from gitserver. Test plan: - new unit tests - manual testing: I ran this together with Sourcegraph and triggered a delete by calling DeleteAllData directly. I confirmed that all shards, including compound shards are deleted.

1 year ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

all: run modernize across codebase (#919) The latest release of gopls has a feature called modernize which will update your code where it can to use modern go features/pkgs. https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0 Generated with: go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./... Test Plan: CI

1 year ago

sourcegraph-indexserver: add grpc server (#920) Relates to SPLF-874 This adds a grpc server to sourcegraph-indexserver. For now it supports just one method. The diff is quite big, so I left comments to mark the most important bits. I used the opportunity to clean up a bit (=> hence the big diff): - Reuse grpc logic from webserver and move those bits to "/gprc/..." - Move "protos" inside the new "grpc" directory (=> requires changes of import statements in Sourcegraph) - Refactor import aliases for grpc packages across the codebase Test plan: I tested this locally by calling the new grpc endpoint directly.

1 year ago

ranking: removing document ranks (#853) Document ranks was an experimental feature of Sourcegraph. We have already removed the code in Sourcegraph in the last release. This is the corresponding cleanup for Zoekt. Test plan: updated tests

2 years ago