alpha
Login
or
Join now
boltless.me
/
zoekt
Star
0
Fork
0
Atom
Configure Feed
Issues
Pull Requests
Commits
Tags
Feed URL
Select the types of activity you want to include in your feed.
fork of https://github.com/sourcegraph/zoekt
Star
0
Fork
0
Atom
Configure Feed
Issues
Pull Requests
Commits
Tags
Feed URL
Select the types of activity you want to include in your feed.
Overview
Issues
Pulls
Pipelines
zoekt
/
cmd
/
zoekt-sourcegraph-indexserver
/
at
87f1948d79751f7728f4f7ec0a13697f721907f0
2 folders
23 files
Julie Tibshirani
Remove block profile rate config (#818)
2y ago
44eab0c5
json_schemas
zoekt-sourcegraph-indexserver: add gRPC support to sourcegraph client (#551) Co-authored-by: Camden Cheek <camden@ccheek.com>
3 years ago
protos
Indexing: configure CPU count through index options (#704) This change allows the shard concurrency to be set through index options. This is much more convenient than the current way to limit CPU through the server flag `-cpu_fraction`. In a follow-up we'll expose shard concurrency through Sourcegraph site config, and pass it through these index options. Maybe we can deprecate and move away from `-cpu_fraction` in favor of this approach.
2 years ago
backoff.go
Remove block profile rate config (#818) Tiny clean-up, as we rarely use Datadog profiling (and I've never heard of us setting this env var).
2 years ago
backoff_test.go
Add metric for indexing delay (#804) Our dashboards currently report indexing delay using the `index_queue_age_seconds` metric, which tracks the duration indexing jobs wait in the queue before being "popped" for indexing. For dot com, we regularly see the median for this metric at 2 days. However, this metric may be misleading because it includes "noop" indexing jobs where the index is already up to date. These jobs can stay in the queue for a long time since they are not high priority. This PR adds a new metric `index_indexing_delay_seconds`, which reports the duration from when an indexing job enters the queue, to when it completes. It uses 'state' as a label, so by setting `state='success'` we can (roughly) see the delay from when Zoekt learns about a new version, to when that version is available to search.
2 years ago
cleanup.go
all: gofumpt -l -w . gofumpt is a stricter gofmt. I took a look at the changes and in general they are nice. I don't think we need to enforce the use of gofumpt, but I like the idea of running it every once in a while. Test Plan: go test ./...
2 years ago
cleanup_test.go
all: gofumpt -l -w . gofumpt is a stricter gofmt. I took a look at the changes and in general they are nice. I don't think we need to enforce the use of gofumpt, but I like the idea of running it every once in a while. Test Plan: go test ./...
2 years ago
debug.go
all: remove curl (#401) We only have curl for debugging. We can just use wget which comes with busybox. Additionally bump alpine patch version to latest. Test Plan: ran wget manually to ensure it works. Also built both images locally and had clean scans from snyk. docker build -t zoekt . docker scan zoekt docker build -t zoekt-indexserver . -f Dockerfile.indexserver docker scan zoekt-indexserver docker build -t zoekt-webserver . -f Dockerfile.webserver docker scan zoekt-webserver
4 years ago
default_grpc_service_configuration.json
zoekt-sourcegraph-indexserver: add gRPC support to sourcegraph client (#551) Co-authored-by: Camden Cheek <camden@ccheek.com>
3 years ago
index.go
Avoid fetching files over size limit (#814) We never index files over 1MB, unless the "LargeFiles" allowlist is set. So in most cases, we can avoid fetching them at all. This PR updates the `git fetch` to filter out files over 1MB when possible, and exclude tags. It also refactors the very long `gitIndex` method.
2 years ago
index_mutex.go
indexserver: only log if someone else is running (#395) We inverted the condition around alreadyRunning. Luckily the only side-effect was extra debug logs. I noticed this while testing with debug logs on. Test Plan: Run with debug logs and notice the already running messages disappear. Add in a sleep for 2 minutes in indexing then check already running messages pop up. SRC_LOG_LEVEL=dbug go run ./cmd/zoekt-sourcegraph-indexserver \ -listen 127.0.0.1:6072 \ -index_concurrency 4 \ -sourcegraph_url ~/src \ -index /tmp/zoekt-test plz-review-url: https://plz.review/review/6899
4 years ago
index_test.go
Avoid fetching files over size limit (#814) We never index files over 1MB, unless the "LargeFiles" allowlist is set. So in most cases, we can avoid fetching them at all. This PR updates the `git fetch` to filter out files over 1MB when possible, and exclude tags. It also refactors the very long `gitIndex` method.
2 years ago
main.go
Remove block profile rate config (#818) Tiny clean-up, as we rarely use Datadog profiling (and I've never heard of us setting this env var).
2 years ago
main_test.go
sourcegraph: Remove HTTP APIs (#738) We no longer serve these old HTTP APIs starting with Sourcegraph 5.3, so this code is no longer required. I hope this makes your lives a little easier by not having to maintain two code paths. If this change is annoying (because you cannot backport fixes to older versions as easily), feel free to either keep it on hold or close it. ## Test plan CI still passes, anything else I should be trying?
2 years ago
merge.go
index: enable shard merging by default (#798) This enables shard merging by default for zoekt-sourcegraph-indexserver. Sourcegraph has been using shard merging in production for several years. We have recently confirmed significant performance improvements for queries which are bound by matchTree construction. I also remove -merge_max_priority because we have stopped using it. Use SRC_DISABLE_SHARD_MERGING to disable shard merging. Test plan: mostly CI, I did some manual testing to confirm that shard merging is enabled by default for zoekt-sourcegraph-indexserver.
2 years ago
merge_test.go
all: gofumpt -l -w . gofumpt is a stricter gofmt. I took a look at the changes and in general they are nice. I don't think we need to enforce the use of gofumpt, but I like the idea of running it every once in a while. Test Plan: go test ./...
2 years ago
meta.go
peterguy/cross platform build (#535) * Replace unix/syscall `mmap` with `mmap-go` `mmap-go` is a cross-platform memory-mapping package. It uses `Mmap` on *Nix systems and `CreateFileMapping` + `MapViewOfFile` on Windows. `mmap-go` benchmarks the same as `unix.Mmap` (see [https://github.com/peterguy/benchmark-mmap](https://github.com/peterguy/benchmark-mmap)). BONUS! `indexfile_other.go` used straight file reads (`os.File::ReadAt`) to support the same kind of behavior as mmap on non-*Nix platforms; `mmap-go` uses Windows APIs to do real memory mapping on Windows, which is faster by several magnitudes. Also change the rounding/padding of the memory buffer being allocated. It was hard-coded to round up to the nearest 4096 bytes in order to page align for mmap, but not all operating systems use 4k pages. Instead of the hard coded value, use `os.Getpagesize()`. * Build-constrain usages of `unix.Umask` * build-constrain zoekt webserver for cross-platform * Fix buffer size in Windows and refacter to a method that calculates the correct buffer size for the platform * use gopsutil v3 for disk space metrics instead of unix/syscall Statfs * Build-constrain the Prometheus memory map metrics Windows does not have the concept of a count of mapped memory areas. Non-linux operating systems do not use the /sys pseudo-filesystem. Constrain `mustRegisterMemoryMapMetrics` to linux-only builds. Other platforms use a no-op function. * cleanup - go get -u -t ./.. - remove unused import * Update cmd/zoekt-webserver/main.go Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com> --------- Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>
3 years ago
meta_test.go
all: rename module to github.com/sourcegraph/zoekt We are a hard enough for now that we might as well have our own module name. This will also make it possible to more directly use go tooling.
3 years ago
meta_unix.go
peterguy/cross platform build (#535) * Replace unix/syscall `mmap` with `mmap-go` `mmap-go` is a cross-platform memory-mapping package. It uses `Mmap` on *Nix systems and `CreateFileMapping` + `MapViewOfFile` on Windows. `mmap-go` benchmarks the same as `unix.Mmap` (see [https://github.com/peterguy/benchmark-mmap](https://github.com/peterguy/benchmark-mmap)). BONUS! `indexfile_other.go` used straight file reads (`os.File::ReadAt`) to support the same kind of behavior as mmap on non-*Nix platforms; `mmap-go` uses Windows APIs to do real memory mapping on Windows, which is faster by several magnitudes. Also change the rounding/padding of the memory buffer being allocated. It was hard-coded to round up to the nearest 4096 bytes in order to page align for mmap, but not all operating systems use 4k pages. Instead of the hard coded value, use `os.Getpagesize()`. * Build-constrain usages of `unix.Umask` * build-constrain zoekt webserver for cross-platform * Fix buffer size in Windows and refacter to a method that calculates the correct buffer size for the platform * use gopsutil v3 for disk space metrics instead of unix/syscall Statfs * Build-constrain the Prometheus memory map metrics Windows does not have the concept of a count of mapped memory areas. Non-linux operating systems do not use the /sys pseudo-filesystem. Constrain `mustRegisterMemoryMapMetrics` to linux-only builds. Other platforms use a no-op function. * cleanup - go get -u -t ./.. - remove unused import * Update cmd/zoekt-webserver/main.go Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com> --------- Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>
3 years ago
meta_windows.go
peterguy/cross platform build (#535) * Replace unix/syscall `mmap` with `mmap-go` `mmap-go` is a cross-platform memory-mapping package. It uses `Mmap` on *Nix systems and `CreateFileMapping` + `MapViewOfFile` on Windows. `mmap-go` benchmarks the same as `unix.Mmap` (see [https://github.com/peterguy/benchmark-mmap](https://github.com/peterguy/benchmark-mmap)). BONUS! `indexfile_other.go` used straight file reads (`os.File::ReadAt`) to support the same kind of behavior as mmap on non-*Nix platforms; `mmap-go` uses Windows APIs to do real memory mapping on Windows, which is faster by several magnitudes. Also change the rounding/padding of the memory buffer being allocated. It was hard-coded to round up to the nearest 4096 bytes in order to page align for mmap, but not all operating systems use 4k pages. Instead of the hard coded value, use `os.Getpagesize()`. * Build-constrain usages of `unix.Umask` * build-constrain zoekt webserver for cross-platform * Fix buffer size in Windows and refacter to a method that calculates the correct buffer size for the platform * use gopsutil v3 for disk space metrics instead of unix/syscall Statfs * Build-constrain the Prometheus memory map metrics Windows does not have the concept of a count of mapped memory areas. Non-linux operating systems do not use the /sys pseudo-filesystem. Constrain `mustRegisterMemoryMapMetrics` to linux-only builds. Other platforms use a no-op function. * cleanup - go get -u -t ./.. - remove unused import * Update cmd/zoekt-webserver/main.go Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com> --------- Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>
3 years ago
owner.go
all: gofumpt -l -w . gofumpt is a stricter gofmt. I took a look at the changes and in general they are nice. I don't think we need to enforce the use of gofumpt, but I like the idea of running it every once in a while. Test Plan: go test ./...
2 years ago
owner_test.go
all: gofumpt -l -w . gofumpt is a stricter gofmt. I took a look at the changes and in general they are nice. I don't think we need to enforce the use of gofumpt, but I like the idea of running it every once in a while. Test Plan: go test ./...
2 years ago
queue.go
Remove index queue age metric (#810) The new metric `index_indexing_delay_seconds` is looking good on dot com and producing more interpretable results. This PR removes the old metric `index_queue_age_seconds`. And it increases the bucket range for the new metric, since we sometimes exceed the current 2 day limit.
2 years ago
queue_test.go
Add metric for indexing delay (#804) Our dashboards currently report indexing delay using the `index_queue_age_seconds` metric, which tracks the duration indexing jobs wait in the queue before being "popped" for indexing. For dot com, we regularly see the median for this metric at 2 days. However, this metric may be misleading because it includes "noop" indexing jobs where the index is already up to date. These jobs can stay in the queue for a long time since they are not high priority. This PR adds a new metric `index_indexing_delay_seconds`, which reports the duration from when an indexing job enters the queue, to when it completes. It uses 'state' as a label, so by setting `state='success'` we can (roughly) see the delay from when Zoekt learns about a new version, to when that version is available to search.
2 years ago
sg.go
sourcegraph: Remove HTTP APIs (#738) We no longer serve these old HTTP APIs starting with Sourcegraph 5.3, so this code is no longer required. I hope this makes your lives a little easier by not having to maintain two code paths. If this change is annoying (because you cannot backport fixes to older versions as easily), feel free to either keep it on hold or close it. ## Test plan CI still passes, anything else I should be trying?
2 years ago
sg_test.go
zoekt-sourcegraph-indexserver: add gRPC support to sourcegraph client (#551) Co-authored-by: Camden Cheek <camden@ccheek.com>
3 years ago