zoekt/cmd/ at 05b2b92c2ebd0927249db50155dfe7375e0ed525 · boltless.me/zoekt · Tangled

boltless.me / zoekt

0

fork of https://github.com/sourcegraph/zoekt

0

zoekt / cmd /

at 05b2b92c2ebd0927249db50155dfe7375e0ed525 18 folders 1 file

Julie Tibshirani ranking: downweight binary files (#924) 1y ago

Document all the commands + packages (#904) This PR adds doc comments for all packages/ commands.

1 year ago

zoekt-archive-index

Document all the commands + packages (#904) This PR adds doc comments for all packages/ commands.

1 year ago

zoekt-dynamic-indexserver

Document all the commands + packages (#904) This PR adds doc comments for all packages/ commands.

1 year ago

zoekt-git-clone

Document all the commands + packages (#904) This PR adds doc comments for all packages/ commands.

1 year ago

zoekt-git-index

Document all the commands + packages (#904) This PR adds doc comments for all packages/ commands.

1 year ago

ranking: downweight binary files (#924) In testing, I noticed another problem with BM25: sometimes a binary file is ranked highly because of a match on its filename. In classic Zoekt scoring, these are ranked low because they are skipped, and we always sort skipped docs to the end of the index. This PR ensures they're also ranked low for BM25 by adding a 'binary' category, and marking it low priority. Adding this category required updating `SkipReason` to track the reason a document was skipped. This is necessary because we set the content of skipped docs to `nil`, and `SkipReason` is the only lasting indication that it was binary.

1 year ago

zoekt-indexserver

Document all the commands + packages (#904) This PR adds doc comments for all packages/ commands.

1 year ago

zoekt-merge-index

sourcegraph-indexserver: GRPC, implement DeleteAllData (#923) Relates to #920 Relates to SPLF-874 This implements DeleteAllData. We hold the global lock while deleting all simple shards belonging to a tenant. We also handle compound shards by disassembling them first. Note that this "only" deletes persisted data. Updating the queue, for example, seems fragile because it might immediately get updated by Sourcegraph. This implies that Sourcegraph first has to delete the tenant in the Sourcegraph DB first and then call this new endpoint. Even if the queue still has a reference to a deleted tenant, indexserver won't be able to retrieve index options or clone the repo from gitserver. Test plan: - new unit tests - manual testing: I ran this together with Sourcegraph and triggered a delete by calling DeleteAllData directly. I confirmed that all shards, including compound shards are deleted.

1 year ago

zoekt-mirror-bitbucket-server

Document all the commands + packages (#904) This PR adds doc comments for all packages/ commands.

1 year ago

zoekt-mirror-gerrit

Document all the commands + packages (#904) This PR adds doc comments for all packages/ commands.

1 year ago

zoekt-mirror-gitea

Document all the commands + packages (#904) This PR adds doc comments for all packages/ commands.

1 year ago

zoekt-mirror-github

Document all the commands + packages (#904) This PR adds doc comments for all packages/ commands.

1 year ago

zoekt-mirror-gitiles

Document all the commands + packages (#904) This PR adds doc comments for all packages/ commands.

1 year ago

zoekt-mirror-gitlab

Document all the commands + packages (#904) This PR adds doc comments for all packages/ commands.

1 year ago

zoekt-repo-index

Document all the commands + packages (#904) This PR adds doc comments for all packages/ commands.

1 year ago

zoekt-sourcegraph-indexserver

sourcegraph-indexserver: GRPC, implement DeleteAllData (#923) Relates to #920 Relates to SPLF-874 This implements DeleteAllData. We hold the global lock while deleting all simple shards belonging to a tenant. We also handle compound shards by disassembling them first. Note that this "only" deletes persisted data. Updating the queue, for example, seems fragile because it might immediately get updated by Sourcegraph. This implies that Sourcegraph first has to delete the tenant in the Sourcegraph DB first and then call this new endpoint. Even if the queue still has a reference to a deleted tenant, indexserver won't be able to retrieve index options or clone the repo from gitserver. Test plan: - new unit tests - manual testing: I ran this together with Sourcegraph and triggered a delete by calling DeleteAllData directly. I confirmed that all shards, including compound shards are deleted.

1 year ago

Document all the commands + packages (#904) This PR adds doc comments for all packages/ commands.

1 year ago

zoekt-webserver

sourcegraph-indexserver: add grpc server (#920) Relates to SPLF-874 This adds a grpc server to sourcegraph-indexserver. For now it supports just one method. The diff is quite big, so I left comments to mark the most important bits. I used the opportunity to clean up a bit (=> hence the big diff): - Reuse grpc logic from webserver and move those bits to "/gprc/..." - Move "protos" inside the new "grpc" directory (=> requires changes of import statements in Sourcegraph) - Refactor import aliases for grpc packages across the codebase Test plan: I tested this locally by calling the new grpc endpoint directly.

1 year ago

ranking: downweight binary files (#924) In testing, I noticed another problem with BM25: sometimes a binary file is ranked highly because of a match on its filename. In classic Zoekt scoring, these are ranked low because they are skipped, and we always sort skipped docs to the end of the index. This PR ensures they're also ranked low for BM25 by adding a 'binary' category, and marking it low priority. Adding this category required updating `SkipReason` to track the reason a document was skipped. This is necessary because we set the content of skipped docs to `nil`, and `SkipReason` is the only lasting indication that it was binary.

1 year ago