fork of https://github.com/sourcegraph/zoekt
Disable most caching for packfile objects (#854)
By default go-git maintains an LRU cache of git objects of size 96MB. When an
object's contents are loaded, it's stored as a MemoryObject in this cache. This
cache is not super useful in the indexing access pattern, which accesses each
file only once. And in many profiles, we see a substantial number of
allocations from these memory objects.
This PR disables caching for most git objects by setting LargeObjectThreshold:
1. go-git still proactively caches packfile objects under 16KB (see
smallObjectThreshold here).
Follow up to #852. This change is also gated by the
ZOEKT_ENABLE_GOGIT_OPTIMIZATION feature flag.
author
Julie Tibshirani
committer
GitHub
date
(Oct 31, 2024, 8:08 AM -0700)
commit
27429a1e
27429a1e828f23ab3c97f64f318c11d984d24f73
parent
c7f1e697
c7f1e697599c4662ab7b984c12eb4ddaf9d23c59
+6
-2
+6
-2
gitindex/index.go
+6
-2
gitindex/index.go
···
565
565
566
566
// openRepo opens a git repository in a way that's optimized for indexing.
567
567
//
568
-
// It copies the relevant logic from git.PlainOpen, and enables the filesystem KeepDescriptors option. This
569
-
// caches the packfile handles, preventing the packfile from being opened then closed on every object access.
568
+
// It copies the relevant logic from git.PlainOpen, and tweaks certain filesystem options.
570
569
func openRepo(repoDir string) (*git.Repository, io.Closer, error) {
571
570
fs := osfs.New(repoDir)
572
571
···
586
585
}
587
586
588
587
s := filesystem.NewStorageWithOptions(fs, cache.NewObjectLRUDefault(), filesystem.Options{
588
+
// Cache the packfile handles, preventing the packfile from being opened then closed on every object access
589
589
KeepDescriptors: true,
590
+
// Disable caching for most objects, by setting the threshold to 1 byte. This avoids allocating a bunch of
591
+
// in-memory objects that are unlikely to be reused, since we only read each file once. Note: go-git still
592
+
// proactively caches objects under 16KB (see smallObjectThreshold in packfile logic).
593
+
LargeObjectThreshold: 1,
590
594
})
591
595
592
596
// Because we're keeping descriptors open, we need to close the storage object when we're done.