zoekt/testdata/ at 4a31578bbe1de7d4a824c96e5cba1770fc1630ca · boltless.me/zoekt · Tangled

boltless.me / zoekt

0

fork of https://github.com/sourcegraph/zoekt

0

zoekt / testdata /

at 4a31578bbe1de7d4a824c96e5cba1770fc1630ca 6 folders 2 files

Keegan Carruthers-Smith repoID bitmap for speeding up findShard in compound shards (#899) 1y ago

repoID bitmap for speeding up findShard in compound shards (#899) We add a new section to shards which contains a roaring bitmap for quickly checking if a shard contains a repo ID. We then can load just this (small amount) of data to rule out a compound shard. We use roaring bitmaps since we already have that dependency in our codebase. The reason we speed up this operation is we found on a large instance which contained thousands of tiny repos we spent so much time in findShard that our indexing queue would always fall behind. It is possible this new section won't speed this up enough and we need some sort of global oracle (or in-memory cache in indexserver?). This is noted in the code for future travellers. Test Plan: the existing unit tests already cover if this is forwards and backwards compatible. Additionally I added some logging to zoekt to test if older version of shards still work correctly in findShard, as well as if older versions of zoekt can read the new shards. Added a benchmark to check the impact. See comments in the code. --------- Co-authored-by: Stefan Hengl <stefan@sourcegraph.com>

1 year ago

grpc: RepoList: actually persist "repos" field when converting to protobuf message (#644)

2 years ago

scoring: use repo freshness as tiebreaker (#832) We ignore priority and instead use the latest commit date as repo rank. This has a big impact for Sourcegraph because it means we switch from star count to repo freshness as tiebreaker. As a minor tweak, we also separate query based scores from tiebreakers. To achieve this we reserve the last 7 digits of a score for tiebreakers: - 5 digits (maxUint16) for repo rank - 2 digits ([0,10]) for file order (2 digits). Example: Before: score: 8775.35 <- atom(2):200, fragment:8550.00, repo-rank: 19, doc-order:6.35 After: score: 8750_00019_06.35 <- atom(2):200, fragment:8550.00, repo-rank: 19, doc-order:6.35

2 years ago

Add small test for v15 backwards compatibility (#23) * add test for backwards compatibility for v15 * update version * WIP use smaller index Change-Id: Id28f9477a400b7d5649bbc0e8a4d567813792fae * reduce test index + use golden file * cleanup * update test

6 years ago

merging: support exploding compound shards (#271) This change let's us split a compound shard into its constituent repos. In the future this should happen instead of deleting too small compound shards. Fow now, the feature is behind a feature flag. To activate, place a file EXPLODE in the index dir.

4 years ago

repoID bitmap for speeding up findShard in compound shards (#899) We add a new section to shards which contains a roaring bitmap for quickly checking if a shard contains a repo ID. We then can load just this (small amount) of data to rule out a compound shard. We use roaring bitmaps since we already have that dependency in our codebase. The reason we speed up this operation is we found on a large instance which contained thousands of tiny repos we spent so much time in findShard that our indexing queue would always fall behind. It is possible this new section won't speed this up enough and we need some sort of global oracle (or in-memory cache in indexserver?). This is noted in the code for future travellers. Test Plan: the existing unit tests already cover if this is forwards and backwards compatible. Additionally I added some logging to zoekt to test if older version of shards still work correctly in findShard, as well as if older versions of zoekt can read the new shards. Added a benchmark to check the impact. See comments in the code. --------- Co-authored-by: Stefan Hengl <stefan@sourcegraph.com>

1 year ago

ci: add shellcheck step (#316)

4 years ago

search_result_1.pb

Add a gRPC API (#577) This adds a gRPC API alongside the existing gob API. It is enabled whenever the RPC setting is enabled. I didn't think it made sense to have a separate setting to enable it. It is only used when a gRPC request is detected (Content-Type: application/grpc). Eventually, we should likely open a separate port for gRPC traffic, but this should be okay for now. In order to minimize the footprint of this change, we only use the protobuf definitions in the RPC layer. They are translated to/from the existing go types. This results in a small perf penalty.

3 years ago