zoekt/build/ at db067d1294f3ac13d77c19c8259baa3a8c6c1b6a · boltless.me/zoekt · Tangled

boltless.me / zoekt

0

fork of https://github.com/sourcegraph/zoekt

0

zoekt / build /

at db067d1294f3ac13d77c19c8259baa3a8c6c1b6a 1 folder 7 files

Keegan Carruthers-Smith build: faster newLinesIndices via bytes.IndexByte and buffer re-use (#680) 2y ago

Add benchmark for ctags conversion (#679) This change adds a benchmark for the conversion from ctags output to Zoekt document data, plus a tiny optimization to presize the symbol slices.

2 years ago

build: faster newLinesIndices via bytes.IndexByte and buffer re-use (#680) Firstly we use bytes.IndexByte for faster newLinesIndices. On my machine this reduces wall clock time of BenchmarkTagsToSections by 38%. This is faster since bytes.IndexByte relies on CPU specific optimizations to find the next new line (eg uses AVX2 if available). Secondly we reuse nls slice between calls to tagsToSections. I noticed in the profiler a nonsignificant chunk in the garbage collector. The slice built by newLinesIndices is allocated and thrown away for each call to tagsToSections. This means we can re-use it which this commit implements by introducing a struct storing the buffer. We now use this buffer per shard of symbols we analyse. old time/op new time/op delta 188µs ± 7% 101µs ± 3% -46.10% (p=0.000 n=10+10) old alloc/op new alloc/op delta 79.3kB ± 0% 36.3kB ± 0% -54.24% (p=0.000 n=9+10) old allocs/op new allocs/op delta 443 ± 0% 441 ± 0% -0.45% (p=0.000 n=10+10) Test Plan: go test -bench BenchmarkTagsToSections

2 years ago

builder_test.go

indexserver: include index time when pushing sg index updates (#598) We missed this path in the last PR, since we assumed we did an RPC to get this information. This change has a slightly larger blast radius because indexserver needs to read back the index file to find out the index time. Previously it just used inputs to the index when informing sourcegraph. Test Plan: pretty robust unit tests here.

3 years ago

builder_unix.go

Swap out all usages of the `syscall` package (#513) with the `golang.org/x/sys/unix` package. `syscall` has been frozen since Go 1.3 and deprecated (https://go.dev/doc/go1.4#major_library_changes). Using the `golang.org/x/sys/unix` package will bring in bug fixes and enhancements since `syscall` was frozen in 1.3, and will pave the way for multi-platform builds (which will affect only the single-program local install, most likely).

3 years ago

build: faster newLinesIndices via bytes.IndexByte and buffer re-use (#680) Firstly we use bytes.IndexByte for faster newLinesIndices. On my machine this reduces wall clock time of BenchmarkTagsToSections by 38%. This is faster since bytes.IndexByte relies on CPU specific optimizations to find the next new line (eg uses AVX2 if available). Secondly we reuse nls slice between calls to tagsToSections. I noticed in the profiler a nonsignificant chunk in the garbage collector. The slice built by newLinesIndices is allocated and thrown away for each call to tagsToSections. This means we can re-use it which this commit implements by introducing a struct storing the buffer. We now use this buffer per shard of symbols we analyse. old time/op new time/op delta 188µs ± 7% 101µs ± 3% -46.10% (p=0.000 n=10+10) old alloc/op new alloc/op delta 79.3kB ± 0% 36.3kB ± 0% -54.24% (p=0.000 n=9+10) old allocs/op new allocs/op delta 443 ± 0% 441 ± 0% -0.45% (p=0.000 n=10+10) Test Plan: go test -bench BenchmarkTagsToSections

2 years ago

build: faster newLinesIndices via bytes.IndexByte and buffer re-use (#680) Firstly we use bytes.IndexByte for faster newLinesIndices. On my machine this reduces wall clock time of BenchmarkTagsToSections by 38%. This is faster since bytes.IndexByte relies on CPU specific optimizations to find the next new line (eg uses AVX2 if available). Secondly we reuse nls slice between calls to tagsToSections. I noticed in the profiler a nonsignificant chunk in the garbage collector. The slice built by newLinesIndices is allocated and thrown away for each call to tagsToSections. This means we can re-use it which this commit implements by introducing a struct storing the buffer. We now use this buffer per shard of symbols we analyse. old time/op new time/op delta 188µs ± 7% 101µs ± 3% -46.10% (p=0.000 n=10+10) old alloc/op new alloc/op delta 79.3kB ± 0% 36.3kB ± 0% -54.24% (p=0.000 n=9+10) old allocs/op new allocs/op delta 443 ± 0% 441 ± 0% -0.45% (p=0.000 n=10+10) Test Plan: go test -bench BenchmarkTagsToSections

2 years ago

Scoring: test against local scip-ctags (#677) This change refactors our end-to-end scoring tests and enables local testing using the scip-ctags binary: * Split scoring tests out of `e2e_test` and into their own file `scoring_test` * Split huge test methods into targeted ones like `TestFileNameMatch`, `TestJava`, `TestGo`, etc. * For languages that scip-ctags supports, rerun the same cases using the scip-ctags binary To run scip-ctags tests locally, you can set the env variable ``` SCIP_CTAGS_COMMAND=<sourcegraph-repo>/dev/scip-ctags-dev ``` This doesn't yet update Zoekt CI to run scip-ctags tests. That will be tackled in a follow-up.

2 years ago

scoring_test.go

Scoring: test against local scip-ctags (#677) This change refactors our end-to-end scoring tests and enables local testing using the scip-ctags binary: * Split scoring tests out of `e2e_test` and into their own file `scoring_test` * Split huge test methods into targeted ones like `TestFileNameMatch`, `TestJava`, `TestGo`, etc. * For languages that scip-ctags supports, rerun the same cases using the scip-ctags binary To run scip-ctags tests locally, you can set the env variable ``` SCIP_CTAGS_COMMAND=<sourcegraph-repo>/dev/scip-ctags-dev ``` This doesn't yet update Zoekt CI to run scip-ctags tests. That will be tackled in a follow-up.

2 years ago