zoekt/build/ at 8801747561381ac59c6e5fc246b6991898692f49 · boltless.me/zoekt · Tangled

boltless.me / zoekt

0

fork of https://github.com/sourcegraph/zoekt

0

zoekt / build /

at 8801747561381ac59c6e5fc246b6991898692f49 1 folder 7 files

Keegan Carruthers-Smith build: ignore out of bound lines from ctags (#694) 2y ago

Add benchmark for ctags conversion (#679) This change adds a benchmark for the conversion from ctags output to Zoekt document data, plus a tiny optimization to presize the symbol slices.

2 years ago

build: ignore out of bound lines from ctags (#694) universal-ctags sometimes returns lines that are out of bounds. In practice it seems to only do an off by one. We haven't noticed the linenum error until a recent change of mine which didn't append an extra entry to NLS if the file was terminated by "\n". In practice this would end up being filtered out later on. So we update to just continue rather than error here. An example is https://github.com/sourcegraph/sourcegraph/blob/v5.2.2/client/web-sveltekit/.storybook/main.ts $ universal-ctags '--fields=*' --output-format=json main.ts | grep 22 {"_type": "tag", "name": "config", "path": "main.ts", "pattern": "/^export default config$/", "language": "TypeScript", "line": 22, "kind": "constant", "roles": "def"} $ wc -l main.ts 21 main.ts $ tail -n1 main.ts export default config Test Plan: added a unit test

2 years ago

builder_test.go

Indexing: respect indexing buffer limit (#686) When indexing documents, we buffer up documents until we reach the shard size limit (100MB), then flush the shard. If we decide to skip a document because it's a binary file, then (naturally) we don't count its content size towards the shard limit. But we still buffered the full document. So if there are a large number of binary files, we could easily blow past the 100MB limit and run into memory issues. This change simply clears `Content` whenever `SkipReason` is set. The invariant: a buffered document should only ever have `SkipReason` or `Content`, not both.

2 years ago

builder_unix.go

Swap out all usages of the `syscall` package (#513) with the `golang.org/x/sys/unix` package. `syscall` has been frozen since Go 1.3 and deprecated (https://go.dev/doc/go1.4#major_library_changes). Using the `golang.org/x/sys/unix` package will bring in bug fixes and enhancements since `syscall` was frozen in 1.3, and will pave the way for multi-platform builds (which will affect only the single-program local install, most likely).

3 years ago

build: ignore out of bound lines from ctags (#694) universal-ctags sometimes returns lines that are out of bounds. In practice it seems to only do an off by one. We haven't noticed the linenum error until a recent change of mine which didn't append an extra entry to NLS if the file was terminated by "\n". In practice this would end up being filtered out later on. So we update to just continue rather than error here. An example is https://github.com/sourcegraph/sourcegraph/blob/v5.2.2/client/web-sveltekit/.storybook/main.ts $ universal-ctags '--fields=*' --output-format=json main.ts | grep 22 {"_type": "tag", "name": "config", "path": "main.ts", "pattern": "/^export default config$/", "language": "TypeScript", "line": 22, "kind": "constant", "roles": "def"} $ wc -l main.ts 21 main.ts $ tail -n1 main.ts export default config Test Plan: added a unit test

2 years ago

build: ignore out of bound lines from ctags (#694) universal-ctags sometimes returns lines that are out of bounds. In practice it seems to only do an off by one. We haven't noticed the linenum error until a recent change of mine which didn't append an extra entry to NLS if the file was terminated by "\n". In practice this would end up being filtered out later on. So we update to just continue rather than error here. An example is https://github.com/sourcegraph/sourcegraph/blob/v5.2.2/client/web-sveltekit/.storybook/main.ts $ universal-ctags '--fields=*' --output-format=json main.ts | grep 22 {"_type": "tag", "name": "config", "path": "main.ts", "pattern": "/^export default config$/", "language": "TypeScript", "line": 22, "kind": "constant", "roles": "def"} $ wc -l main.ts 21 main.ts $ tail -n1 main.ts export default config Test Plan: added a unit test

2 years ago

Indexing: improve skipped doc handling (#687) This change makes a couple small improvements to how we handle skipped docs: * Immediately skip ctags parsing if the content is `nil` * Always sort skipped docs to the end of the shard. This seems like a nice invariant. And generally it's good for performance to group data that is expected to be accessed together and has similar content.

2 years ago

scoring_test.go

build: ignore out of bound lines from ctags (#694) universal-ctags sometimes returns lines that are out of bounds. In practice it seems to only do an off by one. We haven't noticed the linenum error until a recent change of mine which didn't append an extra entry to NLS if the file was terminated by "\n". In practice this would end up being filtered out later on. So we update to just continue rather than error here. An example is https://github.com/sourcegraph/sourcegraph/blob/v5.2.2/client/web-sveltekit/.storybook/main.ts $ universal-ctags '--fields=*' --output-format=json main.ts | grep 22 {"_type": "tag", "name": "config", "path": "main.ts", "pattern": "/^export default config$/", "language": "TypeScript", "line": 22, "kind": "constant", "roles": "def"} $ wc -l main.ts 21 main.ts $ tail -n1 main.ts export default config Test Plan: added a unit test

2 years ago