zoekt/shards/ at a176bde1a312a581c181d8c1ab16b193d0914e18 · boltless.me/zoekt · Tangled

boltless.me / zoekt

0

fork of https://github.com/sourcegraph/zoekt

0

zoekt / shards /

at a176bde1a312a581c181d8c1ab16b193d0914e18 10 files

Keegan Carruthers-Smith matchiter: capture metric NgramLookups (#608) 3y ago

matchiter: capture metric NgramLookups (#608) This instruments our matchiter code to track how often is calls Get on the underlying ngram index. We have some queries on long strings which are going much slower than expected and we suspect it is related to the work done here. This change requires a few moving pieces: - Adding the int to Stats - Instrumenting places we log/emit Stats - grpc support - capturing this work in matchiter code - updating tests to ensure correctness The last two points are the meat of this commit, while the rest was mechanical work. I reworked the TestAndSearch since into TestSearchStats since it was the only test which had useful assertions on Stats. I was surprised to see how high the ngram lookup values were for such simple queries. This makes me more confident in improving our performance here. Test Plan: go test

3 years ago

Ranking: sort and truncate files while collecting (#543) Before, the collector aggregated all file matches before flushing, when it finally sorted and truncated them. Now we sort and limit while collecting results, instead of at the end. This can help cut down on memory in the case there are many shard results. (It won't help with `count: all` queries, where MaxDocDisplayCount is not set.) Another small benefit is that we obey FlushWallTime more closely. Before, we might sort a very large number of results after already using up the whole time.

3 years ago

all: rename module to github.com/sourcegraph/zoekt We are a hard enough for now that we might as well have our own module name. This will also make it possible to more directly use go tooling.

3 years ago

all: rename module to github.com/sourcegraph/zoekt We are a hard enough for now that we might as well have our own module name. This will also make it possible to more directly use go tooling.

3 years ago

scheduler: add some clarifying comments (#508)

3 years ago

all: fix errcheck lints in test files

4 years ago

matchiter: capture metric NgramLookups (#608) This instruments our matchiter code to track how often is calls Get on the underlying ngram index. We have some queries on long strings which are going much slower than expected and we suspect it is related to the work done here. This change requires a few moving pieces: - Adding the int to Stats - Instrumenting places we log/emit Stats - grpc support - capturing this work in matchiter code - updating tests to ensure correctness The last two points are the meat of this commit, while the rest was mechanical work. I reworked the TestAndSearch since into TestSearchStats since it was the only test which had useful assertions on Stats. I was surprised to see how high the ngram lookup values were for such simple queries. This makes me more confident in improving our performance here. Test Plan: go test

3 years ago

shards: populate RepoList.Stats.Repos (#605) Previously this was only populated by the web package for serving a specific endpoint. We now populate it in all places we do aggregation. Test Plan: updated unit tests. Additionally loaded zoekt-webserver then added to the index a repository which required 2 shards to index. The repository count shown only increased by 1.

3 years ago

Fix no-shards-loaded being reported as crashes (#512) Full debug context here: https://sourcegraph.slack.com/archives/C023ELQLV7F/p1672835804462349 Commit 9899a9b3f475ef066ed70c395f8b303268f5d00c changed what the response looks like when Zoekt has never loaded a shard: it now reports a `Crashes = 1`, even if everything's fine. That leads to upstream errors where we show an error message in the Sourcegraph admin UI because the customer hasn't added any repositories to their instance yet. The fix here changes the `loader` to also mark the `shardedSearcher` as ready if there was nothing to load. Previously the `markReady` was skipped, the searcher wasn't marked as "ready" and every search query was replied to with a `Crashes = 1`

3 years ago

watcher_test.go

shards: unexport NewDirectoryWatcher Does not need to be exported.

3 years ago