zoekt/gitindex/ at 59884de44b986c8e4d74e1bad5bf49d7b9093c8d · boltless.me/zoekt · Tangled

boltless.me / zoekt

0

fork of https://github.com/sourcegraph/zoekt

0

zoekt / gitindex /

at 59884de44b986c8e4d74e1bad5bf49d7b9093c8d 14 files

Julie Tibshirani Enable go-git optimization by default (#856) 2y ago

Enable go-git optimization by default (#856) The go-git optimizations have been running well on dot-com, so this PR enables them by default. I'll remove the flag entirely once these optimizations have been released to more instances.

2 years ago

all: use git >= 2.38.5 (#580) This is to resolve CVE-2023-25652 and CVE-2023-29007. Additionally we update the gitindex tests to specify the default branch to use. This was to fix local failures I was experiencing. Test Plan: go test inside of nix develop. Additionally ensure the version of git was larger than 2.38.5 when testing.

3 years ago

zoekt-mirror-gerrit: delete stale repos (#282) This change adds an option on gerrit mirror to delete repos that don't match mirror criteria anymore. It also changes the DeleteRepos to handle repository names with multiple "/".

4 years ago

tests: replace ioutil.TempDir with testing.T.TempDir (#359) testing.T.TempDir was introduced in Go 1.15 (August 2020) and the majority of tests had been written earlier.

4 years ago

all: gofumpt -l -w . gofumpt is a stricter gofmt. I took a look at the changes and in general they are nice. I don't think we need to enforce the use of gofumpt, but I like the idea of running it every once in a while. Test Plan: go test ./...

2 years ago

Make IndexGitRepo signal whether there were changes (#781) As described, we'd like this information to be externally available to drive metrics for incremental indexing. Closes #780.

2 years ago

Enable go-git optimization by default (#856) The go-git optimizations have been running well on dot-com, so this PR enables them by default. I'll remove the flag entirely once these optimizations have been released to more instances.

2 years ago

Avoid reopening packfile on every object access (#852) By default, the `go-git` library will open the packfile on every call to `Repository.BlobObject`, then close it. During indexing, we collect the list of files to index, then iterate through each one calling `Repository.BlobObject`. So on every object access the packfile reopened, and `go-git` reallocates some in-memory buffers. This PR bypasses `git.PlainOpen` to allow us to enable the `KeepDescriptors` option. This option keeps packfile files open, and caches wrappers for them. The files then need to be explicitly closed when done with the repo. Benefits: * Avoid reallocating the memory buffers on every object access (see benchmark results below) * (Highly speculative) I suspect this could improve OS decisions around when to cache portions of the packfile. Maybe constantly reopening and seeking within the file makes it harder for the OS to determine the true access pattern, which is roughly random access. This can affect decisions like readahead and whether to consider pages 'active'.

2 years ago

all: migrate go-git to new package go-git now lives at github.com/go-git/go-git. v5 exists to allow the migration to the new location and is API compatible with v4 [1]. [1]: https://github.com/go-git/go-git/releases/tag/v5.0.0 Change-Id: If68ba927e38ae8ef81447f995e449994928f0cfa

6 years ago

repocache_test.go

tests: replace ioutil.TempDir with testing.T.TempDir (#359) testing.T.TempDir was introduced in Go 1.15 (August 2020) and the majority of tests had been written earlier.

4 years ago

gitindex: correctly skip UTF-8 BOM (#230) Previously we called buf.ReadRune to detect if we had a BOM. However, buf.ReadRune on the BOM just consumes the first byte and returns "\uFFFD". So this code accidently worked. In the case of the BOM the UnreadRune call actually returned an error which we didn't check. This updates the code to correctly detect a BOM. Additionally it doesn't have to rely on reading then unreading. Instead we can peak at what is remaining since this is a bytes.Buffer. There is a risk that there are submodule files that don't have the exact BOM we are detecting and somehow we are skipping over and working on them due to the code before. So when this code rolls out we should monitor production.

4 years ago

submodule_test.go

all: gofumpt -l -w . gofumpt is a stricter gofmt. I took a look at the changes and in general they are nice. I don't think we need to enforce the use of gofumpt, but I like the idea of running it every once in a while. Test Plan: go test ./...

2 years ago

Use single map for collecting files across branches (#839) When looking at large profiles for `inuse_space` on dot com, I noticed the filename maps in `prepareNormalBuild` taking a bunch of memory. This PR avoids allocating a separate map per branch, instead having `RepoWalker` collect all the entries in a single instance variable.

2 years ago

Use single map for collecting files across branches (#839) When looking at large profiles for `inuse_space` on dot com, I noticed the filename maps in `prepareNormalBuild` taking a bunch of memory. This PR avoids allocating a separate map per branch, instead having `RepoWalker` collect all the entries in a single instance variable.

2 years ago