boltless.me / zoekt
fork of https://github.com/sourcegraph/zoekt
0
Fork 0

Configure Feed

Select the types of activity you want to include in your feed.

shards: only trigger rescan on .zoekt files changing (#801)

Any write to the index dir triggered a scan. This means on busy
instances we are constantly rescanning, leading to an
over-representation in CPU profiles around watch. The events are
normally writes to our temporary files. By only considering events for
.zoekt files (which is what scan reads) we can avoid the constant scan
calls.

Just in case we also introduce a re-scan every minute in case we miss an
event. There is error handling around this, but I thought it is just
more reliable to call scan every once in a while.

Note: this doesn't represent significant CPU use, but it does muddy the
CPU profiler output. So this makes it easier to understand trends in our
continuous cpu profiling.

Test Plan: CI

author
Keegan Carruthers-Smith committer
GitHub date (Aug 2, 2024, 12:00 PM +0200) commit acacc5ed parent 764fe4f9
+23 -4
+23 -4
shards/watcher.go
··· 117 117 } 118 118 119 119 func (s *DirectoryWatcher) scan() error { 120 + // NOTE: if you change which file extensions are read, please update the 121 + // watch implementation. 120 122 fs, err := filepath.Glob(filepath.Join(s.dir, "*.zoekt")) 121 123 if err != nil { 122 124 return err ··· 216 218 signal := make(chan struct{}, 1) 217 219 218 220 go func() { 221 + notify := func() { 222 + select { 223 + case signal <- struct{}{}: 224 + default: 225 + } 226 + } 227 + 228 + ticker := time.NewTicker(time.Minute) 229 + 219 230 for { 220 231 select { 221 - case <-watcher.Events: 222 - select { 223 - case signal <- struct{}{}: 224 - default: 232 + case event := <-watcher.Events: 233 + // Only notify if a file we read in has changed. This is important to 234 + // avoid all the events writing to temporary files. 235 + if strings.HasSuffix(event.Name, ".zoekt") || strings.HasSuffix(event.Name, ".meta") { 236 + notify() 225 237 } 238 + 239 + case <-ticker.C: 240 + // Periodically just double check the disk 241 + notify() 242 + 226 243 case err := <-watcher.Errors: 227 244 // Ignore ErrEventOverflow since we rely on the presence of events so 228 245 // safe to ignore. 229 246 if err != nil && err != fsnotify.ErrEventOverflow { 230 247 log.Println("watcher error:", err) 231 248 } 249 + 232 250 case <-s.quit: 233 251 watcher.Close() 252 + ticker.Stop() 234 253 close(signal) 235 254 return 236 255 }