zoekt/cmd/ at 120aebf6db233e5f5179f0e1223e3a7b6b70ec5c · boltless.me/zoekt · Tangled

boltless.me / zoekt

0

fork of https://github.com/sourcegraph/zoekt

0

zoekt / cmd /

at 120aebf6db233e5f5179f0e1223e3a7b6b70ec5c 17 folders 1 file

Julie Tibshirani Ranking: adapt to new ranking service API (#553) 3y ago

all: rename module to github.com/sourcegraph/zoekt We are a hard enough for now that we might as well have our own module name. This will also make it possible to more directly use go tooling.

3 years ago

zoekt-archive-index

Exclude files in search.largeFiles (#480) A pattern within search.largeFiles can negate files which match an earlier pattern, which would exclude that file from being able to ignore/override the size max threshold.

3 years ago

zoekt-dynamic-indexserver

Change data_dir argument in zoekt-dynamic-indexserver to repo_dir (#544) This previously was only being used for storing thre repos. Since bare repos take up much more disk space that the indexes and have different IO requirements they need to be on a separate disk. We may later introduce a more generic data_dir which can then set default values for repo_dir and index_dir but for now it's simpler to make these 2 options explicit.

3 years ago

zoekt-git-clone

Add a zoekt-dynamic-indexserver cmd with HTTP interface for indexing (#496) This is intended to be used for making it possible for Zoekt to power code search in GitLab. Co-authored-by: Dmitry Gruzd <dgruzd@gitlab.com>

3 years ago

zoekt-git-index

indexserver: introduce DocumentRankVersion (#460) This field allows us to indicate when the ranks have changed, which forces us to re-index. We introduce DocumentRankVersion as a builder options such that it can influence that behavior. By making a corresponding change to Sourcegraph to set this field we will automatically update indexes as document ranks change for a repository. Note: We remove the document ranking feature flag from index server. This is now completely controlled by frontend. Note: I got started on a larger refactor here to make this more clean. The idea is to make DocumentRanksPath a URL which is controlled by Sourcegraph. We can then use the URL changing to indicate re-indexing for example. For now this is a much smaller change which will unblock us sooner. Test Plan: Ran a local server and tested that when the ranks are missing nothing happens. When the ranks change or are added we reindex. Additionally when the ranks do not change it does nothing. go install ./cmd/zoekt-git-index go run ./cmd/zoekt-sourcegraph-indexserver \ -sourcegraph_url ~/src/github.com/sourcegraph/ \ -listen 127.0.0.1:6072 # We expect this to trigger an index echo -e "README.md\t0.1" > ~/src/github.com/sourcegraph/zoekt/SG_DOCUMENT_RANKS pkill -SIGUSR1 zoekt-sourcegra # We expect this to do nothing since the ranks have not changed pkill -SIGUSR1 zoekt-sourcegra # We expect this to trigger a re-index echo -e "README.md\t0.2" > ~/src/github.com/sourcegraph/zoekt/SG_DOCUMENT_RANKS pkill -SIGUSR1 zoekt-sourcegra

3 years ago

all: rename module to github.com/sourcegraph/zoekt We are a hard enough for now that we might as well have our own module name. This will also make it possible to more directly use go tooling.

3 years ago

zoekt-indexserver

all: rename module to github.com/sourcegraph/zoekt We are a hard enough for now that we might as well have our own module name. This will also make it possible to more directly use go tooling.

3 years ago

zoekt-merge-index

all: rename module to github.com/sourcegraph/zoekt We are a hard enough for now that we might as well have our own module name. This will also make it possible to more directly use go tooling.

3 years ago

zoekt-mirror-bitbucket-server

all: rename module to github.com/sourcegraph/zoekt We are a hard enough for now that we might as well have our own module name. This will also make it possible to more directly use go tooling.

3 years ago

zoekt-mirror-gerrit

all: fix golangci-lint errors Only ignored errcheck and composites.

3 years ago

zoekt-mirror-github

all: rename module to github.com/sourcegraph/zoekt We are a hard enough for now that we might as well have our own module name. This will also make it possible to more directly use go tooling.

3 years ago

zoekt-mirror-gitiles

all: rename module to github.com/sourcegraph/zoekt We are a hard enough for now that we might as well have our own module name. This will also make it possible to more directly use go tooling.

3 years ago

zoekt-mirror-gitlab

all: gofmt -w -s .

3 years ago

zoekt-repo-index

all: gofmt -w -s .

3 years ago

zoekt-sourcegraph-indexserver

Ranking: adapt to new ranking service API (#553) We are updating the ranking service API to only return a single rank representing the log of each file's reference counts. This PR adapts zoekt-indexserver to the new API shape, and updates the normalization strategy. For now, it doesn't change the on-disk format to keep the change simple.

3 years ago

all: rename module to github.com/sourcegraph/zoekt We are a hard enough for now that we might as well have our own module name. This will also make it possible to more directly use go tooling.

3 years ago

zoekt-webserver

peterguy/cross platform build (#535) * Replace unix/syscall `mmap` with `mmap-go` `mmap-go` is a cross-platform memory-mapping package. It uses `Mmap` on *Nix systems and `CreateFileMapping` + `MapViewOfFile` on Windows. `mmap-go` benchmarks the same as `unix.Mmap` (see [https://github.com/peterguy/benchmark-mmap](https://github.com/peterguy/benchmark-mmap)). BONUS! `indexfile_other.go` used straight file reads (`os.File::ReadAt`) to support the same kind of behavior as mmap on non-*Nix platforms; `mmap-go` uses Windows APIs to do real memory mapping on Windows, which is faster by several magnitudes. Also change the rounding/padding of the memory buffer being allocated. It was hard-coded to round up to the nearest 4096 bytes in order to page align for mmap, but not all operating systems use 4k pages. Instead of the hard coded value, use `os.Getpagesize()`. * Build-constrain usages of `unix.Umask` * build-constrain zoekt webserver for cross-platform * Fix buffer size in Windows and refacter to a method that calculates the correct buffer size for the platform * use gopsutil v3 for disk space metrics instead of unix/syscall Statfs * Build-constrain the Prometheus memory map metrics Windows does not have the concept of a count of mapped memory areas. Non-linux operating systems do not use the /sys pseudo-filesystem. Constrain `mustRegisterMemoryMapMetrics` to linux-only builds. Other platforms use a no-op function. * cleanup - go get -u -t ./.. - remove unused import * Update cmd/zoekt-webserver/main.go Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com> --------- Co-authored-by: Keegan Carruthers-Smith <keegan.csmith@gmail.com>

3 years ago

Ranking: adapt to new ranking service API (#553) We are updating the ranking service API to only return a single rank representing the log of each file's reference counts. This PR adapts zoekt-indexserver to the new API shape, and updates the normalization strategy. For now, it doesn't change the on-disk format to keep the change simple.

3 years ago