alpha
Login
or
Join now
boltless.me
/
zoekt
Star
0
Fork
0
Atom
Configure Feed
Issues
Pull Requests
Commits
Tags
Feed URL
Select the types of activity you want to include in your feed.
fork of https://github.com/sourcegraph/zoekt
Star
0
Fork
0
Atom
Configure Feed
Issues
Pull Requests
Commits
Tags
Feed URL
Select the types of activity you want to include in your feed.
Overview
Issues
Pulls
Pipelines
zoekt
/
cmd
/
zoekt-sourcegraph-indexserver
/
at
344e33ae3c44c220a48b27e1d2e70e30dcd09a85
2 folders
23 files
Keegan Carruthers-Smith
all: remove detailed debug file logging (#707)
2y ago
51fc9db1
json_schemas
zoekt-sourcegraph-indexserver: add gRPC support to sourcegraph client (#551) Co-authored-by: Camden Cheek <camden@ccheek.com>
3 years ago
protos
Indexing: configure CPU count through index options (#704) This change allows the shard concurrency to be set through index options. This is much more convenient than the current way to limit CPU through the server flag `-cpu_fraction`. In a follow-up we'll expose shard concurrency through Sourcegraph site config, and pass it through these index options. Maybe we can deprecate and move away from `-cpu_fraction` in favor of this approach.
2 years ago
backoff.go
all: remove detailed debug file logging (#707) We have noticed profiles where lumberjack is responsible for 1GB of memory. This is surprising and we haven't used the log output in a long time. Rather than debugging, we are going to remove it for now and when we need detailed logs again we can add something back. Test Plan: go test
2 years ago
backoff_test.go
indexserver: backoff repo if failing repeatedly (#442) Introduce backoff strategy for failed indexing attempts. The strategy's goal is to backoff subsequent indexing attempts for a given repository for a period of time because we've identified that the preceding indexing attempt has failed. The backoff period of time is finite and increases linearly with consecutive indexing failures. Test Plan: Add queue tests and backoff duration tests
3 years ago
cleanup.go
all: remove detailed debug file logging (#707) We have noticed profiles where lumberjack is responsible for 1GB of memory. This is surprising and we haven't used the log output in a long time. Rather than debugging, we are going to remove it for now and when we need detailed logs again we can add something back. Test Plan: go test
2 years ago
cleanup_test.go
merging: delete tmp code that removed duplicates (#445) @mpimenov added this code a while ago to remove duplicate indexes in compound shards on .com. The code path was active for a short period of time and fulfilled its purpose. Now we can remove it. Background: For a while we saw a handfull of duplicate indexes per shard on .com. The root cause turned out to be a missing lock on index dir during merging. This has since been fixed. However, the duplicates still lingered which is why we added this cleanup code. I have checked several times over the last couple of weeks and can confirm that no new duplicates materialized. As a reference, this is how I checked for duplicates ``` cd /data/index/ && apk add jq && for i in compound-*.zoekt; do zoekt-sourcegraph-indexserver debug meta $i | tail -n1 | jq -r '.[] | select(.Tombstone==false) .Name' >> sh_repos; done && cat sh_repos | sort | uniq -d && rm sh_repos ``` I believe the issue is resolved.
3 years ago
debug.go
all: remove curl (#401) We only have curl for debugging. We can just use wget which comes with busybox. Additionally bump alpine patch version to latest. Test Plan: ran wget manually to ensure it works. Also built both images locally and had clean scans from snyk. docker build -t zoekt . docker scan zoekt docker build -t zoekt-indexserver . -f Dockerfile.indexserver docker scan zoekt-indexserver docker build -t zoekt-webserver . -f Dockerfile.webserver docker scan zoekt-webserver
4 years ago
default_grpc_service_configuration.json
zoekt-sourcegraph-indexserver: add gRPC support to sourcegraph client (#551) Co-authored-by: Camden Cheek <camden@ccheek.com>
3 years ago
index.go
Indexing: configure CPU count through index options (#704) This change allows the shard concurrency to be set through index options. This is much more convenient than the current way to limit CPU through the server flag `-cpu_fraction`. In a follow-up we'll expose shard concurrency through Sourcegraph site config, and pass it through these index options. Maybe we can deprecate and move away from `-cpu_fraction` in favor of this approach.
2 years ago
index_mutex.go
indexserver: only log if someone else is running (#395) We inverted the condition around alreadyRunning. Luckily the only side-effect was extra debug logs. I noticed this while testing with debug logs on. Test Plan: Run with debug logs and notice the already running messages disappear. Add in a sleep for 2 minutes in indexing then check already running messages pop up. SRC_LOG_LEVEL=dbug go run ./cmd/zoekt-sourcegraph-indexserver \ -listen 127.0.0.1:6072 \ -index_concurrency 4 \ -sourcegraph_url ~/src \ -index /tmp/zoekt-test plz-review-url: https://plz.review/review/6899
4 years ago
index_test.go
grpc: add prometheus server and client prometheus metrics (#642)
2 years ago
main.go
Indexing: configure CPU count through index options (#704) This change allows the shard concurrency to be set through index options. This is much more convenient than the current way to limit CPU through the server flag `-cpu_fraction`. In a follow-up we'll expose shard concurrency through Sourcegraph site config, and pass it through these index options. Maybe we can deprecate and move away from `-cpu_fraction` in favor of this approach.
2 years ago
main_test.go
Indexing: configure CPU count through index options (#704) This change allows the shard concurrency to be set through index options. This is much more convenient than the current way to limit CPU through the server flag `-cpu_fraction`. In a follow-up we'll expose shard concurrency through Sourcegraph site config, and pass it through these index options. Maybe we can deprecate and move away from `-cpu_fraction` in favor of this approach.
2 years ago
merge.go
all: remove detailed debug file logging (#707) We have noticed profiles where lumberjack is responsible for 1GB of memory. This is surprising and we haven't used the log output in a long time. Rather than debugging, we are going to remove it for now and when we need detailed logs again we can add something back. Test Plan: go test
2 years ago
merge_test.go
indexserver: expose all merge options (#527) With this change we expose "the maximum allowed priority" and "the minimum age of the latest commit" as command line flags and ENVs. At the same time we clean up the Server struct a bit and bundle all fields related to shard merging in a dedicated options struct. The motiviation is to make shard merging fully configurable before rolling it out to all customers.
3 years ago
meta.go
peterguy/cross platform build (#535) * Replace unix/syscall `mmap` with `mmap-go` `mmap-go` is a cross-platform memory-mapping package. It uses `Mmap` on *Nix systems and `CreateFileMapping` + `MapViewOfFile` on Windows. `mmap-go` benchmarks the same as `unix.Mmap` (see [https://github.com/peterguy/benchmark-mmap](https://github.com/peterguy/benchmark-mmap)). BONUS! `indexfile_other.go` used straight file reads (`os.File::ReadAt`) to support the same kind of behavior as mmap on non-*Nix platforms; `mmap-go` uses Windows APIs to do real memory mapping on Windows, which is faster by several magnitudes. Also change the rounding/padding of the memory buffer being allocated. It was hard-coded to round up to the nearest 4096 bytes in order to page align for mmap, but not all operating systems use 4k pages. Instead of the hard coded value, use `os.Getpagesize()`. * Build-constrain usages of `unix.Umask` * build-constrain zoekt webserver for cross-platform * Fix buffer size in Windows and refacter to a method that calculates the correct buffer size for the platform * use gopsutil v3 for disk space metrics instead of unix/syscall Statfs * Build-constrain the Prometheus memory map metrics Windows does not have the concept of a count of mapped memory areas. Non-linux operating systems do not use the /sys pseudo-filesystem. Constrain `mustRegisterMemoryMapMetrics` to linux-only builds. Other platforms use a no-op function. * cleanup - go get -u -t ./.. - remove unused import * Update cmd/zoekt-webserver/main.go Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com> --------- Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>
3 years ago
meta_test.go
all: rename module to github.com/sourcegraph/zoekt We are a hard enough for now that we might as well have our own module name. This will also make it possible to more directly use go tooling.
3 years ago
meta_unix.go
peterguy/cross platform build (#535) * Replace unix/syscall `mmap` with `mmap-go` `mmap-go` is a cross-platform memory-mapping package. It uses `Mmap` on *Nix systems and `CreateFileMapping` + `MapViewOfFile` on Windows. `mmap-go` benchmarks the same as `unix.Mmap` (see [https://github.com/peterguy/benchmark-mmap](https://github.com/peterguy/benchmark-mmap)). BONUS! `indexfile_other.go` used straight file reads (`os.File::ReadAt`) to support the same kind of behavior as mmap on non-*Nix platforms; `mmap-go` uses Windows APIs to do real memory mapping on Windows, which is faster by several magnitudes. Also change the rounding/padding of the memory buffer being allocated. It was hard-coded to round up to the nearest 4096 bytes in order to page align for mmap, but not all operating systems use 4k pages. Instead of the hard coded value, use `os.Getpagesize()`. * Build-constrain usages of `unix.Umask` * build-constrain zoekt webserver for cross-platform * Fix buffer size in Windows and refacter to a method that calculates the correct buffer size for the platform * use gopsutil v3 for disk space metrics instead of unix/syscall Statfs * Build-constrain the Prometheus memory map metrics Windows does not have the concept of a count of mapped memory areas. Non-linux operating systems do not use the /sys pseudo-filesystem. Constrain `mustRegisterMemoryMapMetrics` to linux-only builds. Other platforms use a no-op function. * cleanup - go get -u -t ./.. - remove unused import * Update cmd/zoekt-webserver/main.go Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com> --------- Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>
3 years ago
meta_windows.go
peterguy/cross platform build (#535) * Replace unix/syscall `mmap` with `mmap-go` `mmap-go` is a cross-platform memory-mapping package. It uses `Mmap` on *Nix systems and `CreateFileMapping` + `MapViewOfFile` on Windows. `mmap-go` benchmarks the same as `unix.Mmap` (see [https://github.com/peterguy/benchmark-mmap](https://github.com/peterguy/benchmark-mmap)). BONUS! `indexfile_other.go` used straight file reads (`os.File::ReadAt`) to support the same kind of behavior as mmap on non-*Nix platforms; `mmap-go` uses Windows APIs to do real memory mapping on Windows, which is faster by several magnitudes. Also change the rounding/padding of the memory buffer being allocated. It was hard-coded to round up to the nearest 4096 bytes in order to page align for mmap, but not all operating systems use 4k pages. Instead of the hard coded value, use `os.Getpagesize()`. * Build-constrain usages of `unix.Umask` * build-constrain zoekt webserver for cross-platform * Fix buffer size in Windows and refacter to a method that calculates the correct buffer size for the platform * use gopsutil v3 for disk space metrics instead of unix/syscall Statfs * Build-constrain the Prometheus memory map metrics Windows does not have the concept of a count of mapped memory areas. Non-linux operating systems do not use the /sys pseudo-filesystem. Constrain `mustRegisterMemoryMapMetrics` to linux-only builds. Other platforms use a no-op function. * cleanup - go get -u -t ./.. - remove unused import * Update cmd/zoekt-webserver/main.go Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com> --------- Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>
3 years ago
owner.go
indexserver: assert and check for ownership on index directory (#347) We had a customer misconfigure a zoekt setup such that there were multiple replicas owning the same directory. This lead to hard to debug bugs. Instead we will now write a file (owner.txt) which contains the identity of the owner and proactively look for changes in ownership. Test Plan: go test and the following manual test plan $ go run ./cmd/zoekt-sourcegraph-indexserver -sourcegraph_url ~/src -debug # hit ctrl-c # now I expect it to have my computers hostname "habitat" $ cat ~/.zoekt/owner.txt DO NOT EDIT! generated by zoekt-sourcegraph-indexserver. This file records the identity of the owner of this zoekt index directory. If it changes zoekt-sourcegraph-indexserver will exit with a non-zero exit code. This is to prevent multiple owners/writers. hostname=habitat # Now I will run with an overrided hostname, I expect a warning and a new hostname $ go run ./cmd/zoekt-sourcegraph-indexserver -hostname=foobarbaz -sourcegraph_url ~/src -debug 2022/05/19 16:43:10 WARN: detected a change in ownership at startup. You can ignore this if you only have one zoekt replica: detected a change of ownership of /home/keegan/.zoekt/owner.txt. In multiple replica setups this can lead to un-needed rebalancing or bugs if there are multiple writers: owner="habitat" current="foobarbaz" # Now I mutate ~/.zoekt/owner.txt to have a different hostname and I expect the program to crash 2022/05/19 16:47:47 detected a change of ownership of /home/keegan/.zoekt/owner.txt. In multiple replica setups this can lead to un-needed rebalancing or bugs if there are multiple writers: owner="boom" current="foobarbaz" exit status 1
4 years ago
owner_test.go
indexserver: assert and check for ownership on index directory (#347) We had a customer misconfigure a zoekt setup such that there were multiple replicas owning the same directory. This lead to hard to debug bugs. Instead we will now write a file (owner.txt) which contains the identity of the owner and proactively look for changes in ownership. Test Plan: go test and the following manual test plan $ go run ./cmd/zoekt-sourcegraph-indexserver -sourcegraph_url ~/src -debug # hit ctrl-c # now I expect it to have my computers hostname "habitat" $ cat ~/.zoekt/owner.txt DO NOT EDIT! generated by zoekt-sourcegraph-indexserver. This file records the identity of the owner of this zoekt index directory. If it changes zoekt-sourcegraph-indexserver will exit with a non-zero exit code. This is to prevent multiple owners/writers. hostname=habitat # Now I will run with an overrided hostname, I expect a warning and a new hostname $ go run ./cmd/zoekt-sourcegraph-indexserver -hostname=foobarbaz -sourcegraph_url ~/src -debug 2022/05/19 16:43:10 WARN: detected a change in ownership at startup. You can ignore this if you only have one zoekt replica: detected a change of ownership of /home/keegan/.zoekt/owner.txt. In multiple replica setups this can lead to un-needed rebalancing or bugs if there are multiple writers: owner="habitat" current="foobarbaz" # Now I mutate ~/.zoekt/owner.txt to have a different hostname and I expect the program to crash 2022/05/19 16:47:47 detected a change of ownership of /home/keegan/.zoekt/owner.txt. In multiple replica setups this can lead to un-needed rebalancing or bugs if there are multiple writers: owner="boom" current="foobarbaz" exit status 1
4 years ago
queue.go
logger: remove description param (#657) * temporarily use log branch * logging: remove description param in Scoped * use latest log and mountinfo versions
2 years ago
queue_test.go
indexserver: backoff repo if failing repeatedly (#442) Introduce backoff strategy for failed indexing attempts. The strategy's goal is to backoff subsequent indexing attempts for a given repository for a period of time because we've identified that the preceding indexing attempt has failed. The backoff period of time is finite and increases linearly with consecutive indexing failures. Test Plan: Add queue tests and backoff duration tests
3 years ago
sg.go
Indexing: configure CPU count through index options (#704) This change allows the shard concurrency to be set through index options. This is much more convenient than the current way to limit CPU through the server flag `-cpu_fraction`. In a follow-up we'll expose shard concurrency through Sourcegraph site config, and pass it through these index options. Maybe we can deprecate and move away from `-cpu_fraction` in favor of this approach.
2 years ago
sg_test.go
zoekt-sourcegraph-indexserver: add gRPC support to sourcegraph client (#551) Co-authored-by: Camden Cheek <camden@ccheek.com>
3 years ago