alpha
Login
or
Join now
boltless.me
/
zoekt
Star
0
Fork
0
Atom
Configure Feed
Issues
Pull Requests
Commits
Tags
Feed URL
Select the types of activity you want to include in your feed.
fork of https://github.com/sourcegraph/zoekt
Star
0
Fork
0
Atom
Configure Feed
Issues
Pull Requests
Commits
Tags
Feed URL
Select the types of activity you want to include in your feed.
Overview
Issues
Pulls
Pipelines
zoekt
/
cmd
/
zoekt-sourcegraph-indexserver
/
at
b51a2335d51b865e1ffe84aa549e85570da61463
2 folders
23 files
Stefan Hengl
refactor shard name logic (#878)
2y ago
dc1b23bb
json_schemas
zoekt-sourcegraph-indexserver: add gRPC support to sourcegraph client (#551) Co-authored-by: Camden Cheek <camden@ccheek.com>
3 years ago
protos
sourcegraph: multi-tenant Zoekt (#859) This updates webserver and sourcegraph-indexserver to support multi-tenancy. The change is behind an ENV feature-flag. Key changes: - tenant ID is now part of the index (repo metadata) - GRPC: IndexOption and Repository have a new field TenantId - If multi-tenancy is enabled, webserver checks if tenant in context matches the tenant id in the shard - zoekt-git-index has a new parameter "-shard_prefix ". If set, the value will be used instead of repository name as prefix for the name of the shard. For Sourcegraph we use "<tenant id>_<repository id>" as prefix if multi-tenancy is enabled Assumption: All calls to Sourcegraph are privileged Test plan: - New tests - Ran this together with Sourcegraph (with and without MT enabled)
2 years ago
backoff.go
refactor shard name logic (#878) We had 2 copies of this logic in the code and this bit me when I introduced id-based shards indexbuilder.go seems an "ok" place to put. We cannot put it in the shards package because of circular dependencies. Test plan: refactor, so relying on CI
2 years ago
backoff_test.go
Add metric for indexing delay (#804) Our dashboards currently report indexing delay using the `index_queue_age_seconds` metric, which tracks the duration indexing jobs wait in the queue before being "popped" for indexing. For dot com, we regularly see the median for this metric at 2 days. However, this metric may be misleading because it includes "noop" indexing jobs where the index is already up to date. These jobs can stay in the queue for a long time since they are not high priority. This PR adds a new metric `index_indexing_delay_seconds`, which reports the duration from when an indexing job enters the queue, to when it completes. It uses 'state' as a label, so by setting `state='success'` we can (roughly) see the delay from when Zoekt learns about a new version, to when that version is available to search.
2 years ago
cleanup.go
all: set severity hint in most log output (#871) I was inspecting logs in GKE and it incorrectly categorized the severity of nearly all logs from zoekt-webserver and zoekt-indexserver. This is a hack to make it work better without putting in the bigger work of migrating us to structured logging. Test Plan: go test
2 years ago
cleanup_test.go
refactor shard name logic (#878) We had 2 copies of this logic in the code and this bit me when I introduced id-based shards indexbuilder.go seems an "ok" place to put. We cannot put it in the shards package because of circular dependencies. Test plan: refactor, so relying on CI
2 years ago
debug.go
all: set severity hint in most log output (#871) I was inspecting logs in GKE and it incorrectly categorized the severity of nearly all logs from zoekt-webserver and zoekt-indexserver. This is a hack to make it work better without putting in the bigger work of migrating us to structured logging. Test Plan: go test
2 years ago
default_grpc_service_configuration.json
zoekt-sourcegraph-indexserver: add gRPC support to sourcegraph client (#551) Co-authored-by: Camden Cheek <camden@ccheek.com>
3 years ago
index.go
all: set severity hint in most log output (#871) I was inspecting logs in GKE and it incorrectly categorized the severity of nearly all logs from zoekt-webserver and zoekt-indexserver. This is a hack to make it work better without putting in the bigger work of migrating us to structured logging. Test Plan: go test
2 years ago
index_mutex.go
indexserver: only log if someone else is running (#395) We inverted the condition around alreadyRunning. Luckily the only side-effect was extra debug logs. I noticed this while testing with debug logs on. Test Plan: Run with debug logs and notice the already running messages disappear. Add in a sleep for 2 minutes in indexing then check already running messages pop up. SRC_LOG_LEVEL=dbug go run ./cmd/zoekt-sourcegraph-indexserver \ -listen 127.0.0.1:6072 \ -index_concurrency 4 \ -sourcegraph_url ~/src \ -index /tmp/zoekt-test plz-review-url: https://plz.review/review/6899
4 years ago
index_test.go
sourcegraph: multi-tenant Zoekt (#859) This updates webserver and sourcegraph-indexserver to support multi-tenancy. The change is behind an ENV feature-flag. Key changes: - tenant ID is now part of the index (repo metadata) - GRPC: IndexOption and Repository have a new field TenantId - If multi-tenancy is enabled, webserver checks if tenant in context matches the tenant id in the shard - zoekt-git-index has a new parameter "-shard_prefix ". If set, the value will be used instead of repository name as prefix for the name of the shard. For Sourcegraph we use "<tenant id>_<repository id>" as prefix if multi-tenancy is enabled Assumption: All calls to Sourcegraph are privileged Test plan: - New tests - Ran this together with Sourcegraph (with and without MT enabled)
2 years ago
main.go
all: set severity hint in most log output (#871) I was inspecting logs in GKE and it incorrectly categorized the severity of nearly all logs from zoekt-webserver and zoekt-indexserver. This is a hack to make it work better without putting in the bigger work of migrating us to structured logging. Test Plan: go test
2 years ago
main_test.go
all: set severity hint in most log output (#871) I was inspecting logs in GKE and it incorrectly categorized the severity of nearly all logs from zoekt-webserver and zoekt-indexserver. This is a hack to make it work better without putting in the bigger work of migrating us to structured logging. Test Plan: go test
2 years ago
merge.go
all: set severity hint in most log output (#871) I was inspecting logs in GKE and it incorrectly categorized the severity of nearly all logs from zoekt-webserver and zoekt-indexserver. This is a hack to make it work better without putting in the bigger work of migrating us to structured logging. Test Plan: go test
2 years ago
merge_test.go
all: gofumpt -l -w . gofumpt is a stricter gofmt. I took a look at the changes and in general they are nice. I don't think we need to enforce the use of gofumpt, but I like the idea of running it every once in a while. Test Plan: go test ./...
2 years ago
meta.go
peterguy/cross platform build (#535) * Replace unix/syscall `mmap` with `mmap-go` `mmap-go` is a cross-platform memory-mapping package. It uses `Mmap` on *Nix systems and `CreateFileMapping` + `MapViewOfFile` on Windows. `mmap-go` benchmarks the same as `unix.Mmap` (see [https://github.com/peterguy/benchmark-mmap](https://github.com/peterguy/benchmark-mmap)). BONUS! `indexfile_other.go` used straight file reads (`os.File::ReadAt`) to support the same kind of behavior as mmap on non-*Nix platforms; `mmap-go` uses Windows APIs to do real memory mapping on Windows, which is faster by several magnitudes. Also change the rounding/padding of the memory buffer being allocated. It was hard-coded to round up to the nearest 4096 bytes in order to page align for mmap, but not all operating systems use 4k pages. Instead of the hard coded value, use `os.Getpagesize()`. * Build-constrain usages of `unix.Umask` * build-constrain zoekt webserver for cross-platform * Fix buffer size in Windows and refacter to a method that calculates the correct buffer size for the platform * use gopsutil v3 for disk space metrics instead of unix/syscall Statfs * Build-constrain the Prometheus memory map metrics Windows does not have the concept of a count of mapped memory areas. Non-linux operating systems do not use the /sys pseudo-filesystem. Constrain `mustRegisterMemoryMapMetrics` to linux-only builds. Other platforms use a no-op function. * cleanup - go get -u -t ./.. - remove unused import * Update cmd/zoekt-webserver/main.go Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com> --------- Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>
3 years ago
meta_test.go
all: rename module to github.com/sourcegraph/zoekt We are a hard enough for now that we might as well have our own module name. This will also make it possible to more directly use go tooling.
3 years ago
meta_unix.go
peterguy/cross platform build (#535) * Replace unix/syscall `mmap` with `mmap-go` `mmap-go` is a cross-platform memory-mapping package. It uses `Mmap` on *Nix systems and `CreateFileMapping` + `MapViewOfFile` on Windows. `mmap-go` benchmarks the same as `unix.Mmap` (see [https://github.com/peterguy/benchmark-mmap](https://github.com/peterguy/benchmark-mmap)). BONUS! `indexfile_other.go` used straight file reads (`os.File::ReadAt`) to support the same kind of behavior as mmap on non-*Nix platforms; `mmap-go` uses Windows APIs to do real memory mapping on Windows, which is faster by several magnitudes. Also change the rounding/padding of the memory buffer being allocated. It was hard-coded to round up to the nearest 4096 bytes in order to page align for mmap, but not all operating systems use 4k pages. Instead of the hard coded value, use `os.Getpagesize()`. * Build-constrain usages of `unix.Umask` * build-constrain zoekt webserver for cross-platform * Fix buffer size in Windows and refacter to a method that calculates the correct buffer size for the platform * use gopsutil v3 for disk space metrics instead of unix/syscall Statfs * Build-constrain the Prometheus memory map metrics Windows does not have the concept of a count of mapped memory areas. Non-linux operating systems do not use the /sys pseudo-filesystem. Constrain `mustRegisterMemoryMapMetrics` to linux-only builds. Other platforms use a no-op function. * cleanup - go get -u -t ./.. - remove unused import * Update cmd/zoekt-webserver/main.go Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com> --------- Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>
3 years ago
meta_windows.go
peterguy/cross platform build (#535) * Replace unix/syscall `mmap` with `mmap-go` `mmap-go` is a cross-platform memory-mapping package. It uses `Mmap` on *Nix systems and `CreateFileMapping` + `MapViewOfFile` on Windows. `mmap-go` benchmarks the same as `unix.Mmap` (see [https://github.com/peterguy/benchmark-mmap](https://github.com/peterguy/benchmark-mmap)). BONUS! `indexfile_other.go` used straight file reads (`os.File::ReadAt`) to support the same kind of behavior as mmap on non-*Nix platforms; `mmap-go` uses Windows APIs to do real memory mapping on Windows, which is faster by several magnitudes. Also change the rounding/padding of the memory buffer being allocated. It was hard-coded to round up to the nearest 4096 bytes in order to page align for mmap, but not all operating systems use 4k pages. Instead of the hard coded value, use `os.Getpagesize()`. * Build-constrain usages of `unix.Umask` * build-constrain zoekt webserver for cross-platform * Fix buffer size in Windows and refacter to a method that calculates the correct buffer size for the platform * use gopsutil v3 for disk space metrics instead of unix/syscall Statfs * Build-constrain the Prometheus memory map metrics Windows does not have the concept of a count of mapped memory areas. Non-linux operating systems do not use the /sys pseudo-filesystem. Constrain `mustRegisterMemoryMapMetrics` to linux-only builds. Other platforms use a no-op function. * cleanup - go get -u -t ./.. - remove unused import * Update cmd/zoekt-webserver/main.go Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com> --------- Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>
3 years ago
owner.go
all: set severity hint in most log output (#871) I was inspecting logs in GKE and it incorrectly categorized the severity of nearly all logs from zoekt-webserver and zoekt-indexserver. This is a hack to make it work better without putting in the bigger work of migrating us to structured logging. Test Plan: go test
2 years ago
owner_test.go
all: gofumpt -l -w . gofumpt is a stricter gofmt. I took a look at the changes and in general they are nice. I don't think we need to enforce the use of gofumpt, but I like the idea of running it every once in a while. Test Plan: go test ./...
2 years ago
queue.go
all: set severity hint in most log output (#871) I was inspecting logs in GKE and it incorrectly categorized the severity of nearly all logs from zoekt-webserver and zoekt-indexserver. This is a hack to make it work better without putting in the bigger work of migrating us to structured logging. Test Plan: go test
2 years ago
queue_test.go
Add metric for indexing delay (#804) Our dashboards currently report indexing delay using the `index_queue_age_seconds` metric, which tracks the duration indexing jobs wait in the queue before being "popped" for indexing. For dot com, we regularly see the median for this metric at 2 days. However, this metric may be misleading because it includes "noop" indexing jobs where the index is already up to date. These jobs can stay in the queue for a long time since they are not high priority. This PR adds a new metric `index_indexing_delay_seconds`, which reports the duration from when an indexing job enters the queue, to when it completes. It uses 'state' as a label, so by setting `state='success'` we can (roughly) see the delay from when Zoekt learns about a new version, to when that version is available to search.
2 years ago
sg.go
sourcegraph: multi-tenant Zoekt (#859) This updates webserver and sourcegraph-indexserver to support multi-tenancy. The change is behind an ENV feature-flag. Key changes: - tenant ID is now part of the index (repo metadata) - GRPC: IndexOption and Repository have a new field TenantId - If multi-tenancy is enabled, webserver checks if tenant in context matches the tenant id in the shard - zoekt-git-index has a new parameter "-shard_prefix ". If set, the value will be used instead of repository name as prefix for the name of the shard. For Sourcegraph we use "<tenant id>_<repository id>" as prefix if multi-tenancy is enabled Assumption: All calls to Sourcegraph are privileged Test plan: - New tests - Ran this together with Sourcegraph (with and without MT enabled)
2 years ago
sg_test.go
ranking: removing document ranks (#853) Document ranks was an experimental feature of Sourcegraph. We have already removed the code in Sourcegraph in the last release. This is the corresponding cleanup for Zoekt. Test plan: updated tests
2 years ago