The parser used to lift `type:` directives before converting `or` placeholders into real boolean nodes. That let parse-time `orOp` sentinels leak into `Type` children for unparenthesized queries and allowed malformed inputs like `type:repo or` to parse successfully until runtime.

Resolve operator placeholders before wrapping with `Type` so unparenthesized `or` inside a type scope produces a valid `Or` subtree and malformed `or` placement is rejected during parsing. The docs now clarify that `type:` applies to the whole expression in its current scope, including `or` clauses.

Amp-Thread-ID: https://ampcode.com/threads/T-019d0be7-4c9d-76fd-b556-103420d14be9
Co-authored-by: Amp <amp@ampcode.com>

Previously only the "top-level" "case:" affected the query. So if you
had a "case" in any sub expression it would have no effect. This adjusts
our implementation to track nested cases in our query parser and allow
them to affect the final case sensitivity on query.Qs.

Test Plan: added more test cases to demonstrate the problem. These
failed before this commit.

TestDirWatcherUnloadOnce occasionally failed in CI with a queued
"spurious load" event after Stop().

This is expected under fsnotify: a single logical write can emit multiple
filesystem events, and DirectoryWatcher can therefore enqueue extra load
notifications before shutdown completes. The old assertion treated any
leftover load event as a bug, which made the test timing-sensitive.

Update the test to drain queued load notifications after Stop() and keep
asserting that no extra drop event is emitted. That preserves the behavior
we care about (delete triggers unload, and unload is not repeated) while
removing a brittle assumption about load event multiplicity.

Reproduced flake before with:
go test ./search -run TestDirWatcherUnloadOnce -count=500

Validated after change with:
go test ./search -run TestDirWatcherUnloadOnce -count=500

Amp-Thread-ID: https://ampcode.com/threads/T-019d0104-367c-719f-a5f0-e7a95136407c
Co-authored-by: Amp <amp@ampcode.com>

This had likely been copied from the `zoekt-mirror-gerrit` documentation.

Bumped a bunch of dependencies which trivy reported had CVEs to latest:

go get \
github.com/cloudflare/circl@latest \
github.com/go-git/go-git/v5@latest \
go.opentelemetry.io/otel/sdk@latest
go mod tidy

Co-authored-by: Ravi Kumar <rkumar@gitlab.com>

`escape` escapes certain characters, but those don't include `+`, so clicking a `C++` language link will result in the `+`s not being escaped in the URL. This causes the search to search for `"C "` which finds no results.

`encodeURIComponent` is what should be used.

This fixes a bug caused by jsonv2 json.Unmarshal() behavior that is not backward compatible.
The infinite recursion occurs when UnmarshalJSON calls json.Unmarshal on itself.

See: https://github.com/golang/go/issues/75361

Currently the zoekt command's output isn't able to surface information
like which branch a match corresponds to. By streaming out the matches
as lines of json, it can be easily processed by other programs like jq.

This is potentially useful for a lot of cli tools. Personally I'm
looking to use this with fzf and bat to build a little search tui.

- github.com/cloudflare/circl v1.5.0 -> v1.6.1 (fixes CVE-2025-8556)
- golang.org/x/crypto v0.41.0 -> v0.45.0 (fixes CVE-2025-47914, CVE-2025-58181)

Amp-Thread-ID: https://ampcode.com/threads/T-542737d5-98ed-4e85-879f-76abc31d5b77
Co-authored-by: Amp <amp@ampcode.com>

- Replace otelgrpc.StreamServerInterceptor() and otelgrpc.UnaryServerInterceptor() with otelgrpc.NewServerHandler()
- Upgrade go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc from v0.58.0 to v0.63.0
- Stats handler approach is the recommended way forward and more efficient than interceptor-based approach

Amp-Thread-ID: https://ampcode.com/threads/T-2b4e5684-212d-42e5-b790-82569bc9f447

Co-authored-by: Amp <amp@ampcode.com>

I think these snuck in as we changed go versions

Includes one update to prevent some WARN log spam if your data directory
is not backed by a block device.

The /a/ prefix is required for authenticated git operations in default Gerrit
configurations. Gerrit's HttpScheme intentionally includes /a/ in clone URLs
to trigger authentication. Removing it breaks instances using default config.

The ServerInfo API already returns the correct URL format based on the
instance's configuration, so we should use it as-is.

Amp-Thread-ID: https://ampcode.com/threads/T-6944ab20-837d-4e78-a9ad-51083263baec
Co-authored-by: Amp <amp@ampcode.com>

Co-authored-by: John Mason <jmason@gitlab.com>

* Fix git.Open() fs argument for .git dir instead of worktree

The filesystem argument submitted to git.Open() is of the .git
directory, it should be for the worktree, as in go-git this logic is copied from:

https://github.com/go-git/go-git/blob/145daf2492dd86b1d7e7787555ebd9837b93bff8/repository.go#L373-L375

One side effect of this bug was that go-git would refuse to fetch repo
submodules

* Fix missing RepoURLs and LineFragments for result subrepositories

RepoURLs for subrepositories are currently filtered out when results
are streamed per-repo from shards.

* Add submodule indexing without repo cache

When not using repo cache, index git submodules recursively into

subrepos using the go-git API

* Adjust subrepo naming conventions

- Revert repo-cache behavior to unchanged
- Add support for naming repos and modules with no remote (use the dirname for the
former and subrepopath for the latter)
- Add test for submodule no-repo-cache behavior

* Fix overallocation in repoURL and lineFragment filtering

* Revert basename treatment from 08f67597

Missed the fact that it is already done in zoekt-git-index
command, the test was improperly configured

https://github.com/sourcegraph/zoekt/blob/4e4a529c3b63c7d4c7897ba736f1cd52cc163134/cmd/zoekt-git-index/main.go#L93-L100

Co-authored-by: ARyzhikov <ARyzhikov@nota.tech>

We want Zoekt and Sourcegraph to use the same language package. In this PR we move the languages package from Sourcegraph to Zoekt, so that Zoekt can use it and Sourcegraph can import it.

Notes:
- Zoekt doesn't need to fetch content async which is why I added a little helper func `GetLanguagesFromContent` to make the call sites in Zoekt less awkward.
- Sourcegraph's languages package always classified .cls files as Apex, while Zoekt did a content based check. With this PR we follow Zoekt's approach. Specifically, I removed .cls from `unsupportedByEnryExtensionToNameMap`. I added an additional unit test to cover this case.

Test plan:
I appended the test cases from the old Zoekt languages packages to the tests I copied over from Sourcegraph

In https://github.com/sourcegraph/zoekt/pull/962 a new field metadata
was added to zoekt.Repository, but it was not added to the grpc
types and converter methods, which is why the fuzz test occasionally
fails.

Test plan:
CI

This was part of an effort to move the queue from Zoekt to Sourcegraph.
However, we are not going to pursue this for now and we can remove
the corresponging grpc methods.

Sourcegraph never had callers of Index and Delete, so both are save to remove.

`DeleteAllData` is still being called from Sourcegraph.

Test plan:
CI

e2e tests were failing because github.com/sourcegraph/cody repository has been set to private and all tags were removed

Test plan:
CI

Currently if you switch to universal-ctags 6.2.0 it will log a few
warnings on startup. The old version of go-ctags would error out. Now
instead it will log to its info log.

Test Plan: Added the problematic file in the linked issue to ./foo/repo
directory. Then ran "go run ./cmd/zoekt-index -require_ctags -index
./foo/index ./foo/repo" with uctags 6.2.0.

This fixes https://github.com/sourcegraph/zoekt/issues/967.

I reported that also here: https://gitlab.com/gitlab-org/gitlab-zoekt-indexer/-/issues/91

Previously we silently ignored it. For example a search like `(foo))`
would just work since we would only parse `(foo)`. We now report back to
the user the problem.

Note: we already had special handling for unbalanced parenthesis like
this `((foo)`.

Test Plan: added more test cases. Especially tried to explore areas we
could validly not consume the whole input but couldn't find one.

This was a leftover from the commit merged yesterday for code that was
removed.

Test Plan: go test ./...

At GitLab, we encountered limitations when searching within large namespaces
containing thousands of repositories. Specifically, we cannot pass a complete
list of RepoIDs due to size constraints.

This change introduces support for indexing and searching on custom repository
metadata by extending Repository to include an additional Metadata field.

All fields within Repository.Metadata are searchable using a regular
expression evaluator.

This enables more scalable filtering by allowing clients to express regular
expression prefix queries on metadata fields, such as:

traversal_ids:123-456-.*

Or any field really:

haystack:nee.*le

github.com/xanzy/go-gitlab has been archived and is now maintained by
gitlab. Update the import path and catch-up with the API changes.

The only place this was called outside of the tenant package was for
deciding if we should call tenant.Log. However, we can inline that check
and both simplify the tenant exported API as well as how you call
tenant.Log.

Test Plan: CI

We only want to use ID based shard names on multi-tenant instances.
Before this change we decided this based on if tenant enforcement was
turned on. However, we would like to always have tenant enforcment on at
Sourcegraph but not migrate all shards to the new naming scheme (just
yet).

This change is quite simple. However, it works since we never actually
read the filenames to extract tenant ID or repository ID. We only ever
use the data that is persisted inside of the shard to do the
enforcement.

The environment variable we read (WORKSPACES_API_URL) is the same one we
use in the Sourcegraph codebase to determine multi-tenant specific code.

Test Plan: Added a unit test. Will do a much larger manual E2E test
with the Sourcegraph project.

It had exactly one call site and the logic was super simple. So I think
it is clearer to just inline the logic here.

Test Plan: mechanical refactor so just CI. In a future PR I will be
adding better test coverage just on this function.

These are duplicated with RepositoryDescription. There is exactly one
place we set these values, and in that case we set it in the repository
description as well.

Note: we update some the calls in builder.go for o.IndexOptions.RepoID
to be just o.RepoID. This is to make it consistent with how we access
TenantID. IndexOptions is embedded in that struct (indexArgs) which is
why we can do the shortcut.

Test Plan: go test is sufficient for this change. I will do a larger
round of manual tests soon with all my changes stacked.

Asked an agent to inspect the codebase and docs and update the docs to
match reality more. I ended up not using all of its suggestions, but
these are the ones that seem reasonable.

Test Plan: n/a

We have two places with duplicated logic around how it decides the layout of
shards on disk. This now moves that decision into one place.

Additionally we can now unexport index.ShardName. It was only used in one
place outside the package, and that was easy to replace with a hardcoded
string since it is just a test.

Test Plan: Just CI. This has no actual change in functionality, just
refactoring.

We no longer consume these images so lets stop building them.
Additionally we recently rotated some credentials and this step broke.
Rather than fixing it I think we should just remove it since we don't
use the artifacts anymore.

Test Plan: watch CI on main

Generated this and refined it to make sense. Hopefully this helps.

Test Plan: n/a

We instead rely on the environment which is inheritted to make this
decision. The logic will become a little different in the next PR which
will instead decide layout based on if we are in "workspaces" mode vs
just wanting to enforce tenants.

Note: this is Sourcegraph specific. We want to move to always enforcing
tenant in our environments, even outside of our multitenant
environments.

Test Plan: go test

Hi zoekt-Team,

when testing with our local instance I noticed that it seemed that delta builds where not correctly indexing only the delta that I expected, but much more. I looked at prepareDeltaBuild() and prepareNormalBuild() and found that prepareNormalBuild() expands the branches before processing them with expandBranches(), but prepareDeltaBuild() does not. This leads to a lot more indexing, because prepareDeltaBuild() can't find any known branches (that match the wildcard).
I added a call of expandBranches() to prepareDeltaBuild() to fix this and also added a test.

Regards,
Daniel

This fixes an annoying problem where after running `go test ./...` locally,
your local git `user.name` would be set to `thomas`.

This adds the before and after Lines in the search results. The default is
still 0 so the user has to increase "context lines" to see more.

Somehow I just found out about `gen-proto.sh` recently, after many months of
working on Zoekt! This PR adds brief `README` files in the proto directories to
guide people (and LLMs) to it.

Small refactor to simplify the logic in `trace.New`. It inlines a couple public
methods that don't need to be exposed separately.

This PR re-enables the go-git optimization that was disabled in https://github.com/sourcegraph/zoekt/pull/870 because it caused a huge bug. This PR removes the part of the optimization that was causing the bug, setting `LargeObjectThreshold`.

An alternative would be to track down the root cause of the bug in go-git, so that we could keep setting `LargeObjectThreshold`. However, I don't feel that's worth it as:
* The memory improvement from this change was pretty small (see https://github.com/sourcegraph/zoekt/pull/854)
* Eventually we want to move away from go-git towards a "vanilla" direct git usage

Make the flag "token" optional. Exit if the user provided token does not
exist (as before). Only, fallback to the token from the default location
if the user did not provide a token. Continue with an unauthenticated
GitHub client (`nil` OAuth client) if the fallback did not work (does
not exist, cannot be read, ...).

This PR fixes a bug where Zoekt would not compile on 32-bit architectures. It
also takes the opportunity to start using the `math` library everywhere instead
of our own constants like `maxUInt32` to help prevent this sort of issue in the
future by encouraging devs to select the most accurate "max" type for their
specific situation.

Closes https://github.com/sourcegraph/zoekt/issues/935

Closes SPLF-913

This only affects Sourcegraph.

This adds Index and Delete methods to
sourcegraph-indexserver's GRPC server.

The methods aren't called yet by Sourcegraph.

Notable differences to our current indexing loop:
- IndexOptions are pushed by the client instead of pulled by Zoekt
- A semaphore restricts concurrency
- New ENV `SRC_STOP_INDEXING` stops the "competing" indexing loop. This is just for
testing and will eventually be removed
- The Index method tries to recover a possibly older index from
`indexDir/.trash` first before starting to index

Review:
I recommend reviewing the `.proto` file first, followed by `main.go`

Test plan:
- I did a bunch of manual testing to test the trash recovery and the
semaphore logic. I successfuly triggered index and delete calls to the
GRPC server. I will put the Insomnia collection I used for testing in the ticket.
- New unit tests

In https://github.com/sourcegraph/zoekt/pull/901 I moved several packages to 'internal' to clean up the exported API. This PR moves the `shards` package back to root, since it contains important methods like `NewDirectorySearcher`. It also renames the `shards` package to `search` to clarify the usage.

Relates to https://github.com/sourcegraph/zoekt/pull/901#issuecomment-2703171432, https://github.com/sourcegraph/zoekt/discussions/927

The gatherBranches() function shifts a mask of type uint64 until it is 0, but
uses an id for accessing a map of type uint32. This results in empty branch
names getting returned from this function. It can be fixed by changing the type
of id to also be uint64.

The latest release of gopls has a feature called modernize which will
update your code where it can to use modern go features/pkgs.
https://github.com/golang/tools/releases/tag/gopls%2Fv0.18.0

Generated with:

go run golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize@latest -fix -test ./...

Test Plan: CI

In testing, I noticed another problem with BM25: sometimes a binary file is ranked highly because of a match on its filename. In classic Zoekt scoring, these are ranked low because they are skipped, and we always sort skipped docs to the end of the index.

This PR ensures they're also ranked low for BM25 by adding a 'binary' category, and marking it low priority. Adding this category required updating `SkipReason` to track the reason a document was skipped. This is necessary because we set the content of skipped docs to `nil`, and `SkipReason` is the only lasting indication that it was binary.