zoekt/cmd/zoekt-sourcegraph-indexserver/ at 1d71fd02269063322b1636affdbe8d9d45ffce68 · boltless.me/zoekt · Tangled

boltless.me / zoekt

0

fork of https://github.com/sourcegraph/zoekt

0

zoekt / cmd / zoekt-sourcegraph-indexserver /

at 1d71fd02269063322b1636affdbe8d9d45ffce68 2 folders 24 files

Keegan Carruthers-Smith indexserver: remove unused GetRepoRank (#604) 3y ago

zoekt-sourcegraph-indexserver: add gRPC support to sourcegraph client (#551) Co-authored-by: Camden Cheek <camden@ccheek.com>

3 years ago

indexserver: remove unused GetRepoRank (#604) I think this is our older ranking stuff. Test Plan: go test

3 years ago

indexserver: remove unused GetRepoRank (#604) I think this is our older ranking stuff. Test Plan: go test

3 years ago

indexserver: backoff repo if failing repeatedly (#442) Introduce backoff strategy for failed indexing attempts. The strategy's goal is to backoff subsequent indexing attempts for a given repository for a period of time because we've identified that the preceding indexing attempt has failed. The backoff period of time is finite and increases linearly with consecutive indexing failures. Test Plan: Add queue tests and backoff duration tests

3 years ago

backoff_test.go

indexserver: backoff repo if failing repeatedly (#442) Introduce backoff strategy for failed indexing attempts. The strategy's goal is to backoff subsequent indexing attempts for a given repository for a period of time because we've identified that the preceding indexing attempt has failed. The backoff period of time is finite and increases linearly with consecutive indexing failures. Test Plan: Add queue tests and backoff duration tests

3 years ago

indexserver: expose all merge options (#527) With this change we expose "the maximum allowed priority" and "the minimum age of the latest commit" as command line flags and ENVs. At the same time we clean up the Server struct a bit and bundle all fields related to shard merging in a dedicated options struct. The motiviation is to make shard merging fully configurable before rolling it out to all customers.

3 years ago

cleanup_test.go

merging: delete tmp code that removed duplicates (#445) @mpimenov added this code a while ago to remove duplicate indexes in compound shards on .com. The code path was active for a short period of time and fulfilled its purpose. Now we can remove it. Background: For a while we saw a handfull of duplicate indexes per shard on .com. The root cause turned out to be a missing lock on index dir during merging. This has since been fixed. However, the duplicates still lingered which is why we added this cleanup code. I have checked several times over the last couple of weeks and can confirm that no new duplicates materialized. As a reference, this is how I checked for duplicates ``` cd /data/index/ && apk add jq && for i in compound-*.zoekt; do zoekt-sourcegraph-indexserver debug meta $i | tail -n1 | jq -r '.[] | select(.Tombstone==false) .Name' >> sh_repos; done && cat sh_repos | sort | uniq -d && rm sh_repos ``` I believe the issue is resolved.

3 years ago

all: remove curl (#401) We only have curl for debugging. We can just use wget which comes with busybox. Additionally bump alpine patch version to latest. Test Plan: ran wget manually to ensure it works. Also built both images locally and had clean scans from snyk. docker build -t zoekt . docker scan zoekt docker build -t zoekt-indexserver . -f Dockerfile.indexserver docker scan zoekt-indexserver docker build -t zoekt-webserver . -f Dockerfile.webserver docker scan zoekt-webserver

4 years ago

default_grpc_service_configuration.json

zoekt-sourcegraph-indexserver: add gRPC support to sourcegraph client (#551) Co-authored-by: Camden Cheek <camden@ccheek.com>

3 years ago

indexserver: include index time when pushing sg index updates (#598) We missed this path in the last PR, since we assumed we did an RPC to get this information. This change has a slightly larger blast radius because indexserver needs to read back the index file to find out the index time. Previously it just used inputs to the index when informing sourcegraph. Test Plan: pretty robust unit tests here.

3 years ago

indexserver: only log if someone else is running (#395) We inverted the condition around alreadyRunning. Luckily the only side-effect was extra debug logs. I noticed this while testing with debug logs on. Test Plan: Run with debug logs and notice the already running messages disappear. Add in a sleep for 2 minutes in indexing then check already running messages pop up. SRC_LOG_LEVEL=dbug go run ./cmd/zoekt-sourcegraph-indexserver \ -listen 127.0.0.1:6072 \ -index_concurrency 4 \ -sourcegraph_url ~/src \ -index /tmp/zoekt-test plz-review-url: https://plz.review/review/6899

4 years ago

indexserver: remove unused GetRepoRank (#604) I think this is our older ranking stuff. Test Plan: go test

3 years ago

indexserver: include index time when pushing sg index updates (#598) We missed this path in the last PR, since we assumed we did an RPC to get this information. This change has a slightly larger blast radius because indexserver needs to read back the index file to find out the index time. Previously it just used inputs to the index when informing sourcegraph. Test Plan: pretty robust unit tests here.

3 years ago

cmd/zoekt-sourcegraph-indexserver: Add default port to root URL (#559)

3 years ago

indexserver: expose all merge options (#527) With this change we expose "the maximum allowed priority" and "the minimum age of the latest commit" as command line flags and ENVs. At the same time we clean up the Server struct a bit and bundle all fields related to shard merging in a dedicated options struct. The motiviation is to make shard merging fully configurable before rolling it out to all customers.

3 years ago

indexserver: expose all merge options (#527) With this change we expose "the maximum allowed priority" and "the minimum age of the latest commit" as command line flags and ENVs. At the same time we clean up the Server struct a bit and bundle all fields related to shard merging in a dedicated options struct. The motiviation is to make shard merging fully configurable before rolling it out to all customers.

3 years ago

peterguy/cross platform build (#535) * Replace unix/syscall `mmap` with `mmap-go` `mmap-go` is a cross-platform memory-mapping package. It uses `Mmap` on *Nix systems and `CreateFileMapping` + `MapViewOfFile` on Windows. `mmap-go` benchmarks the same as `unix.Mmap` (see [https://github.com/peterguy/benchmark-mmap](https://github.com/peterguy/benchmark-mmap)). BONUS! `indexfile_other.go` used straight file reads (`os.File::ReadAt`) to support the same kind of behavior as mmap on non-*Nix platforms; `mmap-go` uses Windows APIs to do real memory mapping on Windows, which is faster by several magnitudes. Also change the rounding/padding of the memory buffer being allocated. It was hard-coded to round up to the nearest 4096 bytes in order to page align for mmap, but not all operating systems use 4k pages. Instead of the hard coded value, use `os.Getpagesize()`. * Build-constrain usages of `unix.Umask` * build-constrain zoekt webserver for cross-platform * Fix buffer size in Windows and refacter to a method that calculates the correct buffer size for the platform * use gopsutil v3 for disk space metrics instead of unix/syscall Statfs * Build-constrain the Prometheus memory map metrics Windows does not have the concept of a count of mapped memory areas. Non-linux operating systems do not use the /sys pseudo-filesystem. Constrain `mustRegisterMemoryMapMetrics` to linux-only builds. Other platforms use a no-op function. * cleanup - go get -u -t ./.. - remove unused import * Update cmd/zoekt-webserver/main.go Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com> --------- Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>

3 years ago

all: rename module to github.com/sourcegraph/zoekt We are a hard enough for now that we might as well have our own module name. This will also make it possible to more directly use go tooling.

3 years ago

peterguy/cross platform build (#535) * Replace unix/syscall `mmap` with `mmap-go` `mmap-go` is a cross-platform memory-mapping package. It uses `Mmap` on *Nix systems and `CreateFileMapping` + `MapViewOfFile` on Windows. `mmap-go` benchmarks the same as `unix.Mmap` (see [https://github.com/peterguy/benchmark-mmap](https://github.com/peterguy/benchmark-mmap)). BONUS! `indexfile_other.go` used straight file reads (`os.File::ReadAt`) to support the same kind of behavior as mmap on non-*Nix platforms; `mmap-go` uses Windows APIs to do real memory mapping on Windows, which is faster by several magnitudes. Also change the rounding/padding of the memory buffer being allocated. It was hard-coded to round up to the nearest 4096 bytes in order to page align for mmap, but not all operating systems use 4k pages. Instead of the hard coded value, use `os.Getpagesize()`. * Build-constrain usages of `unix.Umask` * build-constrain zoekt webserver for cross-platform * Fix buffer size in Windows and refacter to a method that calculates the correct buffer size for the platform * use gopsutil v3 for disk space metrics instead of unix/syscall Statfs * Build-constrain the Prometheus memory map metrics Windows does not have the concept of a count of mapped memory areas. Non-linux operating systems do not use the /sys pseudo-filesystem. Constrain `mustRegisterMemoryMapMetrics` to linux-only builds. Other platforms use a no-op function. * cleanup - go get -u -t ./.. - remove unused import * Update cmd/zoekt-webserver/main.go Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com> --------- Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>

3 years ago

meta_windows.go

peterguy/cross platform build (#535) * Replace unix/syscall `mmap` with `mmap-go` `mmap-go` is a cross-platform memory-mapping package. It uses `Mmap` on *Nix systems and `CreateFileMapping` + `MapViewOfFile` on Windows. `mmap-go` benchmarks the same as `unix.Mmap` (see [https://github.com/peterguy/benchmark-mmap](https://github.com/peterguy/benchmark-mmap)). BONUS! `indexfile_other.go` used straight file reads (`os.File::ReadAt`) to support the same kind of behavior as mmap on non-*Nix platforms; `mmap-go` uses Windows APIs to do real memory mapping on Windows, which is faster by several magnitudes. Also change the rounding/padding of the memory buffer being allocated. It was hard-coded to round up to the nearest 4096 bytes in order to page align for mmap, but not all operating systems use 4k pages. Instead of the hard coded value, use `os.Getpagesize()`. * Build-constrain usages of `unix.Umask` * build-constrain zoekt webserver for cross-platform * Fix buffer size in Windows and refacter to a method that calculates the correct buffer size for the platform * use gopsutil v3 for disk space metrics instead of unix/syscall Statfs * Build-constrain the Prometheus memory map metrics Windows does not have the concept of a count of mapped memory areas. Non-linux operating systems do not use the /sys pseudo-filesystem. Constrain `mustRegisterMemoryMapMetrics` to linux-only builds. Other platforms use a no-op function. * cleanup - go get -u -t ./.. - remove unused import * Update cmd/zoekt-webserver/main.go Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com> --------- Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>

3 years ago

indexserver: assert and check for ownership on index directory (#347) We had a customer misconfigure a zoekt setup such that there were multiple replicas owning the same directory. This lead to hard to debug bugs. Instead we will now write a file (owner.txt) which contains the identity of the owner and proactively look for changes in ownership. Test Plan: go test and the following manual test plan $ go run ./cmd/zoekt-sourcegraph-indexserver -sourcegraph_url ~/src -debug # hit ctrl-c # now I expect it to have my computers hostname "habitat" $ cat ~/.zoekt/owner.txt DO NOT EDIT! generated by zoekt-sourcegraph-indexserver. This file records the identity of the owner of this zoekt index directory. If it changes zoekt-sourcegraph-indexserver will exit with a non-zero exit code. This is to prevent multiple owners/writers. hostname=habitat # Now I will run with an overrided hostname, I expect a warning and a new hostname $ go run ./cmd/zoekt-sourcegraph-indexserver -hostname=foobarbaz -sourcegraph_url ~/src -debug 2022/05/19 16:43:10 WARN: detected a change in ownership at startup. You can ignore this if you only have one zoekt replica: detected a change of ownership of /home/keegan/.zoekt/owner.txt. In multiple replica setups this can lead to un-needed rebalancing or bugs if there are multiple writers: owner="habitat" current="foobarbaz" # Now I mutate ~/.zoekt/owner.txt to have a different hostname and I expect the program to crash 2022/05/19 16:47:47 detected a change of ownership of /home/keegan/.zoekt/owner.txt. In multiple replica setups this can lead to un-needed rebalancing or bugs if there are multiple writers: owner="boom" current="foobarbaz" exit status 1

4 years ago

indexserver: assert and check for ownership on index directory (#347) We had a customer misconfigure a zoekt setup such that there were multiple replicas owning the same directory. This lead to hard to debug bugs. Instead we will now write a file (owner.txt) which contains the identity of the owner and proactively look for changes in ownership. Test Plan: go test and the following manual test plan $ go run ./cmd/zoekt-sourcegraph-indexserver -sourcegraph_url ~/src -debug # hit ctrl-c # now I expect it to have my computers hostname "habitat" $ cat ~/.zoekt/owner.txt DO NOT EDIT! generated by zoekt-sourcegraph-indexserver. This file records the identity of the owner of this zoekt index directory. If it changes zoekt-sourcegraph-indexserver will exit with a non-zero exit code. This is to prevent multiple owners/writers. hostname=habitat # Now I will run with an overrided hostname, I expect a warning and a new hostname $ go run ./cmd/zoekt-sourcegraph-indexserver -hostname=foobarbaz -sourcegraph_url ~/src -debug 2022/05/19 16:43:10 WARN: detected a change in ownership at startup. You can ignore this if you only have one zoekt replica: detected a change of ownership of /home/keegan/.zoekt/owner.txt. In multiple replica setups this can lead to un-needed rebalancing or bugs if there are multiple writers: owner="habitat" current="foobarbaz" # Now I mutate ~/.zoekt/owner.txt to have a different hostname and I expect the program to crash 2022/05/19 16:47:47 detected a change of ownership of /home/keegan/.zoekt/owner.txt. In multiple replica setups this can lead to un-needed rebalancing or bugs if there are multiple writers: owner="boom" current="foobarbaz" exit status 1

4 years ago

indexserver: backoff repo if failing repeatedly (#442) Introduce backoff strategy for failed indexing attempts. The strategy's goal is to backoff subsequent indexing attempts for a given repository for a period of time because we've identified that the preceding indexing attempt has failed. The backoff period of time is finite and increases linearly with consecutive indexing failures. Test Plan: Add queue tests and backoff duration tests

3 years ago

indexserver: backoff repo if failing repeatedly (#442) Introduce backoff strategy for failed indexing attempts. The strategy's goal is to backoff subsequent indexing attempts for a given repository for a period of time because we've identified that the preceding indexing attempt has failed. The backoff period of time is finite and increases linearly with consecutive indexing failures. Test Plan: Add queue tests and backoff duration tests

3 years ago

indexserver: remove unused GetRepoRank (#604) I think this is our older ranking stuff. Test Plan: go test

3 years ago

zoekt-sourcegraph-indexserver: add gRPC support to sourcegraph client (#551) Co-authored-by: Camden Cheek <camden@ccheek.com>

3 years ago