zoekt/cmd/zoekt-webserver/ at adf376d3ed3cb2bb20dd66c35e2885b886b05ed9 · boltless.me/zoekt · Tangled

boltless.me / zoekt

0

fork of https://github.com/sourcegraph/zoekt

0

zoekt / cmd / zoekt-webserver /

at adf376d3ed3cb2bb20dd66c35e2885b886b05ed9 1 folder 5 files

Keegan Carruthers-Smith web: informative and verbose error message when watchdog fails (#647) 2y ago

web: informative and verbose error message when watchdog fails (#647) Right now we use panicf which leads a stack trace which is misleading at what is happening and fills up the space used by kubernetes error reporting. Additionally a few times we have had bug reports about the watchdog failing. This commit updates the message to be far more informative about next steps. Additionally we update the watchdog error to include the response body in case that contains useful information for debugging. Test Plan: Updated the serveHealthz handler to always return an error. Then ran the following $ ZOEKT_WATCHDOG_TICK=1s go run ./cmd/zoekt-webserver 2023/09/14 15:55:27 custom ZOEKT_WATCHDOG_TICK=1s 2023/09/14 15:55:27 loading 1 shard(s): github.com%2Fsourcegraph%2Fzoekt_v16.00000.zoekt 2023/09/14 15:55:28 watchdog: failed, will try 2 more times: watchdog: status=500 body="not ready: boom\n" 2023/09/14 15:55:29 watchdog: failed, will try 1 more times: watchdog: status=500 body="not ready: boom\n" 2023/09/14 15:55:30 watchdog health check has consecutively failed 3 times indicating is likely an unrecoverable error affecting zoekt. As such this process will exit with code 3. Final error: watchdog: status=500 body="not ready: boom\n" Possible Remediations: - If this rarely happens, ignore and let your process manager restart zoekt. - Possibly under provisioned. Try increasing CPU or disk IO. - A bug. Reach out with logs and screenshots of metrics when this occurs. exit status 3

2 years ago

web: informative and verbose error message when watchdog fails (#647) Right now we use panicf which leads a stack trace which is misleading at what is happening and fills up the space used by kubernetes error reporting. Additionally a few times we have had bug reports about the watchdog failing. This commit updates the message to be far more informative about next steps. Additionally we update the watchdog error to include the response body in case that contains useful information for debugging. Test Plan: Updated the serveHealthz handler to always return an error. Then ran the following $ ZOEKT_WATCHDOG_TICK=1s go run ./cmd/zoekt-webserver 2023/09/14 15:55:27 custom ZOEKT_WATCHDOG_TICK=1s 2023/09/14 15:55:27 loading 1 shard(s): github.com%2Fsourcegraph%2Fzoekt_v16.00000.zoekt 2023/09/14 15:55:28 watchdog: failed, will try 2 more times: watchdog: status=500 body="not ready: boom\n" 2023/09/14 15:55:29 watchdog: failed, will try 1 more times: watchdog: status=500 body="not ready: boom\n" 2023/09/14 15:55:30 watchdog health check has consecutively failed 3 times indicating is likely an unrecoverable error affecting zoekt. As such this process will exit with code 3. Final error: watchdog: status=500 body="not ready: boom\n" Possible Remediations: - If this rarely happens, ignore and let your process manager restart zoekt. - Possibly under provisioned. Try increasing CPU or disk IO. - A bug. Reach out with logs and screenshots of metrics when this occurs. exit status 3

2 years ago

peterguy/cross platform build (#535) * Replace unix/syscall `mmap` with `mmap-go` `mmap-go` is a cross-platform memory-mapping package. It uses `Mmap` on *Nix systems and `CreateFileMapping` + `MapViewOfFile` on Windows. `mmap-go` benchmarks the same as `unix.Mmap` (see [https://github.com/peterguy/benchmark-mmap](https://github.com/peterguy/benchmark-mmap)). BONUS! `indexfile_other.go` used straight file reads (`os.File::ReadAt`) to support the same kind of behavior as mmap on non-*Nix platforms; `mmap-go` uses Windows APIs to do real memory mapping on Windows, which is faster by several magnitudes. Also change the rounding/padding of the memory buffer being allocated. It was hard-coded to round up to the nearest 4096 bytes in order to page align for mmap, but not all operating systems use 4k pages. Instead of the hard coded value, use `os.Getpagesize()`. * Build-constrain usages of `unix.Umask` * build-constrain zoekt webserver for cross-platform * Fix buffer size in Windows and refacter to a method that calculates the correct buffer size for the platform * use gopsutil v3 for disk space metrics instead of unix/syscall Statfs * Build-constrain the Prometheus memory map metrics Windows does not have the concept of a count of mapped memory areas. Non-linux operating systems do not use the /sys pseudo-filesystem. Constrain `mustRegisterMemoryMapMetrics` to linux-only builds. Other platforms use a no-op function. * cleanup - go get -u -t ./.. - remove unused import * Update cmd/zoekt-webserver/main.go Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com> --------- Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>

3 years ago

peterguy/cross platform build (#535) * Replace unix/syscall `mmap` with `mmap-go` `mmap-go` is a cross-platform memory-mapping package. It uses `Mmap` on *Nix systems and `CreateFileMapping` + `MapViewOfFile` on Windows. `mmap-go` benchmarks the same as `unix.Mmap` (see [https://github.com/peterguy/benchmark-mmap](https://github.com/peterguy/benchmark-mmap)). BONUS! `indexfile_other.go` used straight file reads (`os.File::ReadAt`) to support the same kind of behavior as mmap on non-*Nix platforms; `mmap-go` uses Windows APIs to do real memory mapping on Windows, which is faster by several magnitudes. Also change the rounding/padding of the memory buffer being allocated. It was hard-coded to round up to the nearest 4096 bytes in order to page align for mmap, but not all operating systems use 4k pages. Instead of the hard coded value, use `os.Getpagesize()`. * Build-constrain usages of `unix.Umask` * build-constrain zoekt webserver for cross-platform * Fix buffer size in Windows and refacter to a method that calculates the correct buffer size for the platform * use gopsutil v3 for disk space metrics instead of unix/syscall Statfs * Build-constrain the Prometheus memory map metrics Windows does not have the concept of a count of mapped memory areas. Non-linux operating systems do not use the /sys pseudo-filesystem. Constrain `mustRegisterMemoryMapMetrics` to linux-only builds. Other platforms use a no-op function. * cleanup - go get -u -t ./.. - remove unused import * Update cmd/zoekt-webserver/main.go Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com> --------- Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>

3 years ago

main_unsupported.go

peterguy/cross platform build (#535) * Replace unix/syscall `mmap` with `mmap-go` `mmap-go` is a cross-platform memory-mapping package. It uses `Mmap` on *Nix systems and `CreateFileMapping` + `MapViewOfFile` on Windows. `mmap-go` benchmarks the same as `unix.Mmap` (see [https://github.com/peterguy/benchmark-mmap](https://github.com/peterguy/benchmark-mmap)). BONUS! `indexfile_other.go` used straight file reads (`os.File::ReadAt`) to support the same kind of behavior as mmap on non-*Nix platforms; `mmap-go` uses Windows APIs to do real memory mapping on Windows, which is faster by several magnitudes. Also change the rounding/padding of the memory buffer being allocated. It was hard-coded to round up to the nearest 4096 bytes in order to page align for mmap, but not all operating systems use 4k pages. Instead of the hard coded value, use `os.Getpagesize()`. * Build-constrain usages of `unix.Umask` * build-constrain zoekt webserver for cross-platform * Fix buffer size in Windows and refacter to a method that calculates the correct buffer size for the platform * use gopsutil v3 for disk space metrics instead of unix/syscall Statfs * Build-constrain the Prometheus memory map metrics Windows does not have the concept of a count of mapped memory areas. Non-linux operating systems do not use the /sys pseudo-filesystem. Constrain `mustRegisterMemoryMapMetrics` to linux-only builds. Other platforms use a no-op function. * cleanup - go get -u -t ./.. - remove unused import * Update cmd/zoekt-webserver/main.go Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com> --------- Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>

3 years ago

main_windows.go

peterguy/cross platform build (#535) * Replace unix/syscall `mmap` with `mmap-go` `mmap-go` is a cross-platform memory-mapping package. It uses `Mmap` on *Nix systems and `CreateFileMapping` + `MapViewOfFile` on Windows. `mmap-go` benchmarks the same as `unix.Mmap` (see [https://github.com/peterguy/benchmark-mmap](https://github.com/peterguy/benchmark-mmap)). BONUS! `indexfile_other.go` used straight file reads (`os.File::ReadAt`) to support the same kind of behavior as mmap on non-*Nix platforms; `mmap-go` uses Windows APIs to do real memory mapping on Windows, which is faster by several magnitudes. Also change the rounding/padding of the memory buffer being allocated. It was hard-coded to round up to the nearest 4096 bytes in order to page align for mmap, but not all operating systems use 4k pages. Instead of the hard coded value, use `os.Getpagesize()`. * Build-constrain usages of `unix.Umask` * build-constrain zoekt webserver for cross-platform * Fix buffer size in Windows and refacter to a method that calculates the correct buffer size for the platform * use gopsutil v3 for disk space metrics instead of unix/syscall Statfs * Build-constrain the Prometheus memory map metrics Windows does not have the concept of a count of mapped memory areas. Non-linux operating systems do not use the /sys pseudo-filesystem. Constrain `mustRegisterMemoryMapMetrics` to linux-only builds. Other platforms use a no-op function. * cleanup - go get -u -t ./.. - remove unused import * Update cmd/zoekt-webserver/main.go Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com> --------- Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>

3 years ago