zoekt/build/ at ebb3ca24249dfa7d3be2bb48899b86e68bed2f6c · boltless.me/zoekt · Tangled

boltless.me / zoekt

0

fork of https://github.com/sourcegraph/zoekt

0

zoekt / build /

at ebb3ca24249dfa7d3be2bb48899b86e68bed2f6c 1 folder 7 files

Stefan Hengl ranking: add IDF to BM25 score calculation (#788) 2y ago

Add benchmark for ctags conversion (#679) This change adds a benchmark for the conversion from ctags output to Zoekt document data, plus a tiny optimization to presize the symbol slices.

2 years ago

ranking: add IDF to BM25 score calculation (#788) So far, we didn't include IDF in our BM25 score function. Zoekt uses a trigram index and hence doesn't compute document frequency during indexing. We could add this information to the index, but it is not immediately obvious how to tokenize code in a way that is compatible with tokens from a natural language query. Here we calulate the document frequency at query time under the assumption that we visit all documents containing any of the query terms. Notes: Also fixed an off-by-1 bug with how we count documents. Test plan: - Updated unit test - Context evaluation results are slightly worse with a decrease from 64/89 to 63/89

2 years ago

builder_test.go

all: gofumpt -l -w . gofumpt is a stricter gofmt. I took a look at the changes and in general they are nice. I don't think we need to enforce the use of gofumpt, but I like the idea of running it every once in a while. Test Plan: go test ./...

2 years ago

builder_unix.go

Allow wasm compilation (#786)

2 years ago

all: use stdlib slices package (#735) Noticed we weren't using this yet and that the API signatures had changed. Test Plan: go test

2 years ago

Indexing: clean up ctags parser wrapper (#708) This change cleans up the Go ctags parser wrapper as a follow-up to #702. Specific changes: * Remove synchronization in `lockedParser` and rename it to `CTagsParser` * Push delegation to universal vs. SCIP ctags into parser wrapper * Simplify document timeout logic * Rename some files

2 years ago

all: gofumpt -l -w . gofumpt is a stricter gofmt. I took a look at the changes and in general they are nice. I don't think we need to enforce the use of gofumpt, but I like the idea of running it every once in a while. Test Plan: go test ./...

2 years ago

scoring_test.go

ranking: add IDF to BM25 score calculation (#788) So far, we didn't include IDF in our BM25 score function. Zoekt uses a trigram index and hence doesn't compute document frequency during indexing. We could add this information to the index, but it is not immediately obvious how to tokenize code in a way that is compatible with tokens from a natural language query. Here we calulate the document frequency at query time under the assumption that we visit all documents containing any of the query terms. Notes: Also fixed an off-by-1 bug with how we count documents. Test plan: - Updated unit test - Context evaluation results are slightly worse with a decrease from 64/89 to 63/89

2 years ago