zoekt/shards/ at c4b0bc47a729d20883c7dfe8b126062c87fe8777 · boltless.me/zoekt · Tangled

boltless.me / zoekt

0

fork of https://github.com/sourcegraph/zoekt

0

zoekt / shards /

at c4b0bc47a729d20883c7dfe8b126062c87fe8777 9 files

Keegan Carruthers-Smith shards: handle failed cached List for selectRepoSet (#864) 2y ago

shards: handle failed cached List for selectRepoSet (#864) If we failed to List the repositories when loading a shard we would never search it due to selectRepoSet optimization. In practice this feels very rare to happen (only for logic error or disk corruption). However, in those cases we should surface these crashes searches by attempting to search the shard. Additionally I add logging so we can notice when this happens. I didn't add a metric since this is the sort of thing that I think is so rare we would never think to check the metric (but may notice logs). Note: I used the slightly tricky invariant that repos being nil means error. If the shard is actually empty (eg all repos tombstoned) then we still correctly apply the optimization. In practice having an empty shard shouldn't really happen so I'm open to just treating empty repos list as something we have to search.

2 years ago

tenant: add tenant to trace (#862) This adds the tenant ID to the trace. I also move the pprof logging from `FromContext` to the higher level `typeRepoSearcher`. The number of events was too high, because we logged missing tenants per document. I also fixed a bug where pprof logging didn't work at all, because we read the tenant enforcemnt ENV after we set the pprof profile, so the profile was always nil. Test plan: Checked locally that tenants show up in the traces and "missing_tenant" shows up as pprof profile.

2 years ago

all: rename module to github.com/sourcegraph/zoekt We are a hard enough for now that we might as well have our own module name. This will also make it possible to more directly use go tooling.

3 years ago

scheduler: add some clarifying comments (#508)

3 years ago

all: fix errcheck lints in test files

4 years ago

shards: handle failed cached List for selectRepoSet (#864) If we failed to List the repositories when loading a shard we would never search it due to selectRepoSet optimization. In practice this feels very rare to happen (only for logic error or disk corruption). However, in those cases we should surface these crashes searches by attempting to search the shard. Additionally I add logging so we can notice when this happens. I didn't add a metric since this is the sort of thing that I think is so rare we would never think to check the metric (but may notice logs). Note: I used the slightly tricky invariant that repos being nil means error. If the shard is actually empty (eg all repos tombstoned) then we still correctly apply the optimization. In practice having an empty shard shouldn't really happen so I'm open to just treating empty repos list as something we have to search.

2 years ago

ranking: removing document ranks (#853) Document ranks was an experimental feature of Sourcegraph. We have already removed the code in Sourcegraph in the last release. This is the corresponding cleanup for Zoekt. Test plan: updated tests

2 years ago

shards: only trigger rescan on .zoekt files changing (#801) Any write to the index dir triggered a scan. This means on busy instances we are constantly rescanning, leading to an over-representation in CPU profiles around watch. The events are normally writes to our temporary files. By only considering events for .zoekt files (which is what scan reads) we can avoid the constant scan calls. Just in case we also introduce a re-scan every minute in case we miss an event. There is error handling around this, but I thought it is just more reliable to call scan every once in a while. Note: this doesn't represent significant CPU use, but it does muddy the CPU profiler output. So this makes it easier to understand trends in our continuous cpu profiling. Test Plan: CI

2 years ago

watcher_test.go

all: gofumpt -l -w . gofumpt is a stricter gofmt. I took a look at the changes and in general they are nice. I don't think we need to enforce the use of gofumpt, but I like the idea of running it every once in a while. Test Plan: go test ./...

2 years ago