zoekt/internal/ at 05b2b92c2ebd0927249db50155dfe7375e0ed525 · boltless.me/zoekt · Tangled

boltless.me / zoekt

0

fork of https://github.com/sourcegraph/zoekt

0

zoekt / internal /

at 05b2b92c2ebd0927249db50155dfe7375e0ed525 15 folders

Julie Tibshirani ranking: downweight binary files (#924) 1y ago

ranking: downweight binary files (#924) In testing, I noticed another problem with BM25: sometimes a binary file is ranked highly because of a match on its filename. In classic Zoekt scoring, these are ranked low because they are skipped, and we always sort skipped docs to the end of the index. This PR ensures they're also ranked low for BM25 by adding a 'binary' category, and marking it low priority. Adding this category required updating `SkipReason` to track the reason a document was skipped. This is necessary because we set the content of skipped docs to `nil`, and `SkipReason` is the only lasting indication that it was binary.

1 year ago

Move several packages to internal/ (#901) This PR moves the following packages to `internal` to avoid exposing them in the API: * `ctags` * `debugserver` * `gitindex` * `shards` * `trace`

1 year ago

Move root-level index code to index package (#902) In the repo root, we have a bunch of low level logic around index building and searching. So we end up exposing internal logic through the main public `zoekt` package, for example `zoekt.Merge(...)`. This PR moves it into the `build` package, so all code related to index building lives together. It then renames `build` to `index` to reflect the broader focus on indexing and searching the index.

1 year ago

ranking: downweight binary files (#924) In testing, I noticed another problem with BM25: sometimes a binary file is ranked highly because of a match on its filename. In classic Zoekt scoring, these are ranked low because they are skipped, and we always sort skipped docs to the end of the index. This PR ensures they're also ranked low for BM25 by adding a 'binary' category, and marking it low priority. Adding this category required updating `SkipReason` to track the reason a document was skipped. This is necessary because we set the content of skipped docs to `nil`, and `SkipReason` is the only lasting indication that it was binary.

1 year ago

ranking: downweight binary files (#924) In testing, I noticed another problem with BM25: sometimes a binary file is ranked highly because of a match on its filename. In classic Zoekt scoring, these are ranked low because they are skipped, and we always sort skipped docs to the end of the index. This PR ensures they're also ranked low for BM25 by adding a 'binary' category, and marking it low priority. Adding this category required updating `SkipReason` to track the reason a document was skipped. This is necessary because we set the content of skipped docs to `nil`, and `SkipReason` is the only lasting indication that it was binary.

1 year ago

Move several packages to internal/ (#901) This PR moves the following packages to `internal` to avoid exposing them in the API: * `ctags` * `debugserver` * `gitindex` * `shards` * `trace`

1 year ago

feat(Search): Add support for all Apex language extensions (#799) * feat(Search): Add support for all Apex language extensions * clean up comment * Fix typo

2 years ago

remove bazel (#634)

2 years ago

remove bazel (#634)

2 years ago

Move root-level index code to index package (#902) In the repo root, we have a bunch of low level logic around index building and searching. So we end up exposing internal logic through the main public `zoekt` package, for example `zoekt.Merge(...)`. This PR moves it into the `build` package, so all code related to index building lives together. It then renames `build` to `index` to reflect the broader focus on indexing and searching the index.

1 year ago

Rename IndexBuilder -> ShardBuilder (#908) When navigating the code, I've often forgotten the difference between `NewBuilder` and `NewIndexBuilder`. This rename clarifies that one of these indexes a whole repo, while the other builds individual shards. Also `index.NewShardBuilder` sounds better.

1 year ago

all: use a faster vendored regexp/syntax/Regexp.String (#753) We replace all calls to Regexp.String with a vendored version which is faster. go1.22 introduced a commit which "minimizes" the string returned by Regexp.String(). Part of what it does is run enumerate through literals runes in your string to see calculate flags related to unicode and case sensitivity. This can be quite slow, but is made worse by the fact we call it per shard per regexp in your query.Q to construct the matchtree. Currently Regexp.String() represents 40% of CPU time on sourcegraph.com. Before go1.22 it was ~0%. Note: This is a temporary change to resolve the issue. I have a deeper change to make this less clumsy. Note: In one place we remove the use of string by relying on Regexp.Equal instead. Test Plan: go test

2 years ago

Move several packages to internal/ (#901) This PR moves the following packages to `internal` to avoid exposing them in the API: * `ctags` * `debugserver` * `gitindex` * `shards` * `trace`

1 year ago

Move several packages to internal/ (#901) This PR moves the following packages to `internal` to avoid exposing them in the API: * `ctags` * `debugserver` * `gitindex` * `shards` * `trace`

1 year ago

remove bazel (#634)

2 years ago