Codebase analysis · CoreStory V3

Loki

An open-source log aggregation system, written in Go. It stores and indexes log streams by label and answers LogQL queries. Maintained by Grafana Labs.

Maintained by
Grafana Labs
License
AGPL-3.0
Commit analyzed
9ea9b3b44fae23e8a14cef68532e112e958e5a45
Rules extracted
2026-11-15
Extracted by CoreStory V3
30428
accepted rules, each with a cited source quote
18
capabilities
5512
features
01 · Rules

Business rules, traced to source lines.

These 11 rules are a sample of the 30,428 CoreStory V3 extracted from Loki. Each one links to the exact lines on GitHub at commit 9ea9b3b. The statement is CoreStory V3’s wording; the code excerpt is the source it cites.

limit
pkg/distributor/http.go

If MaxInflightBytes is greater than 0 and the current inflight bytes count is greater than or equal to MaxInflightBytes, the push request is rejected with HTTP 503 Service Unavailable and the error message "the server cannot accept more requests at this time".

  • 0 — threshold above which the inflight bytes limit is enforced
  • 503 — HTTP status code returned when the inflight bytes limit is exceeded
  • the server cannot accept more requests at this time — error message returned when the inflight bytes limit is exceeded
maxInflightBytes := int64(d.cfg.MaxInflightBytes) if maxInflightBytes > 0 && d.inflightBytes.Load() >= maxInflightBytes { circuitBreakerErr = errServiceUnavailableMaxLoad errorWriter(w, circuitBreakerErr.Error(), http.StatusServiceUnavailable, logger) return }

Also cited: L28

validation
pkg/distributor/validator.go

When the tenant's RejectOldSamples setting is enabled and an entry's timestamp in Unix nanoseconds is strictly less than the reject-old-sample max-age threshold (now minus RejectOldSamplesMaxAge), ValidateEntry rejects the entry with reason GreaterThanMaxSampleAge and an error formatted with GreaterThanMaxSampleAgeErrorMsg including labels, formatted entry time, and formatted reject max age time.

  • GreaterThanMaxSampleAge — rejection reason returned for too-old sample
  • GreaterThanMaxSampleAgeErrorMsg — error message format used for too-old sample
if vCtx.rejectOldSample && ts < vCtx.rejectOldSampleMaxAge { // Makes time string on the error message formatted consistently. formatedEntryTime := entry.Timestamp.Format(timeFormat) formatedRejectMaxAgeTime := time.Unix(0, vCtx.rejectOldSampleMaxAge).Format(timeFormat) return validation.GreaterThanMaxSampleAge, fmt.Errorf(validation.GreaterThanMaxSampleAgeErrorMsg, labels, formatedEntryTime, formatedRejectMaxAgeTime) }

Also cited: L69–L70

validation
pkg/distributor/validator.go

If an entry's timestamp in Unix nanoseconds is strictly greater than the creation grace period threshold (now plus CreationGracePeriod), ValidateEntry rejects the entry with reason TooFarInFuture and an error formatted with TooFarInFutureErrorMsg including labels and the formatted entry time.

  • TooFarInFuture — rejection reason returned for entries too far in the future
  • TooFarInFutureErrorMsg — error message format used for future entries
if ts > vCtx.creationGracePeriod { formatedEntryTime := entry.Timestamp.Format(timeFormat) return validation.TooFarInFuture, fmt.Errorf(validation.TooFarInFutureErrorMsg, labels, formatedEntryTime) }

Also cited: L71

validation
pkg/distributor/validator.go

When the configured max line size is non-zero and an entry's line length in bytes exceeds that max, ValidateEntry rejects the entry with reason LineTooLong and an error formatted with LineTooLongErrorMsg including the max size, labels, and actual line length.

  • 0 — maxLineSize disables the check
  • LineTooLong — rejection reason for overly long lines
  • LineTooLongErrorMsg — error message format for overly long lines
if maxSize := vCtx.maxLineSize; maxSize != 0 && len(entry.Line) > maxSize { // I wish we didn't return httpgrpc errors here as it seems // an orthogonal concept (we need not use ValidateLabels in this context) // but the upstream cortex_validation pkg uses it, so we keep this // for parity. return validation.LineTooLong, fmt.Errorf(validation.LineTooLongErrorMsg, maxSize, labels, len(entry.Line)) }

Also cited: L72

policy
pkg/distributor/validator.go

shouldBlockTenant returns false when the tenant blockIngestionUntil is zero, returns true with the configured until time and status code when now is before blockIngestionUntil, and otherwise returns false.

  • 0 — status code returned when not blocked
func (v Validator) shouldBlockTenant(ctx validationContext, now time.Time) (bool, time.Time, int) { if ctx.blockIngestionUntil.IsZero() { return false, time.Time{}, 0 } if now.Before(ctx.blockIngestionUntil) { return true, ctx.blockIngestionUntil, ctx.blockIngestionStatusCode } return false, time.Time{}, 0 }

Also cited: L87–L88

validation
pkg/distributor/distributor.go

When a stream of the push resolves to a policy with reject_partial_writes and the error is a 429 (either stream-limit or validation), the whole push is rejected with that error instead of being partially written; the rejected push still consumes its rate-limit tokens and stream slots.

  • 429 — HTTP status code triggering full rejection
// Reject the whole push with a 429 when a policy asks for it, // instead of partially writing and sending a 429. The rejected push // still consumes its rate-limit tokens and the limits service's // stream slots. if validated.rejectPartialWrites && (limitsErr != nil || isTooManyRequests(validated.err)) { d.m.rejectedPartialWrites.WithLabelValues(tenantID, validated.rejectPartialWritesPolicy).Inc() return nil, finalErr }

Also cited: L976–L980

limit
pkg/logql/engine.go

Each LogRangeExpr interval in a sample expression must not exceed the smallest positive non-zero MaxQueryRange across tenants; if it does, an ErrIntervalLimit error is returned. The interval check only runs when the computed limit is non-zero.

  • 0 — limit value that disables the interval check
func (q *query) checkIntervalLimit(expr syntax.SampleExpr, limit time.Duration) error { var err error expr.Walk(func(e syntax.Expr) bool { switch e := e.(type) { case *syntax.LogRangeExpr: if e.Interval > limit { err = fmt.Errorf("%w: [%s] > [%s]", logqlerr.ErrIntervalLimit, model.Duration(e.Interval), model.Duration(limit)) } } return true }) return err }

Also cited: L419–L426

policy
pkg/querier/http.go

Every wrapped query handler applies the smallest positive non-zero per-tenant QueryTimeout across the request's tenants as a context deadline, with cause error 'query timeout reached'. Tenant IDs are extracted from the context; if extraction fails, the request is rejected with HTTP 400.

  • query timeout reached — context cancellation cause message
timeoutCapture := func(id string) time.Duration { return limits.QueryTimeout(ctx, id) } timeout := util_validation.SmallestPositiveNonZeroDurationPerTenant(tenants, timeoutCapture) newCtx, cancel := context.WithTimeoutCause(ctx, timeout, errors.New("query timeout reached")) defer cancel()

Also cited: L628–L633

limit
pkg/querier/tail/querier.go

When the maximum number of concurrent tailers reported by any active ingester (maxCnt) is greater than or equal to the tenant's configured maximum concurrent tail requests limit l, the tail request is rejected with HTTP 400 status and the error message "max concurrent tail requests limit exceeded, count > limit (%d > %d)" with the compared values maxCnt+1 and l.

  • 400 — HTTP status code used in the error response
l := uint32(q.limits.MaxConcurrentTailRequests(ctx, userID)) if maxCnt >= l { return httpgrpc.Errorf(http.StatusBadRequest, "max concurrent tail requests limit exceeded, count > limit (%d > %d)", maxCnt+1, l) }

Also cited: L144–L163

authorization
pkg/querier/limits/validation.go

Queries that access pattern ingester streams (constants.IsPatternIngesterStream) are rejected with ErrInternalStreamsDrilldownOnly unless the context is a Logs Drilldown request (httpreq.IsLogsDrilldownRequest); non-pattern-ingester queries are permitted. ErrInternalStreamsDrilldownOnly equals 'internal streams can only be queried from Logs Drilldown'.

  • internal streams can only be queried from Logs Drilldown — error message returned when pattern ingester streams are queried outside Logs Drilldown
if !constants.IsPatternIngesterStream(hasMatcher) { return nil } if httpreq.IsLogsDrilldownRequest(ctx) { return nil } return ErrInternalStreamsDrilldownOnly

Also cited: L25

limit
pkg/compactor/deletion/deletion_manifest_builder.go

A delete request batch may contain at most 64 delete requests; if the summed request count across all users exceeds 64, construction fails with error "only upto 64 delete requests allowed, current count: %d".

  • 64 — maximum number of delete requests allowed for a manifest build
if requestCount > 64 { return nil, fmt.Errorf("only upto 64 delete requests allowed, current count: %d", requestCount) }

Also cited: L51–L54, L56–L57

02 · Capabilities

What the application does

18 capabilities identified, grouping 151 of the 5,349 distinct features extracted. Most extracted features are not grouped under a capability, so this list is a partial view of the application. These are descriptions, not the cited rules above.

Label and Metadata Management

Discover, parse, match, filter, and manipulate labels and structured metadata across ingestion and query paths.

  • Detected Fields: Query and display detected fields from Loki for field discovery.
  • Entry Label Labels: Expose labels attached to an entry for the caller to observe.
  • Label Discovery: List available label names and the values associated with a label.
  • Label Filtering: Supports applying and carrying label filter hints to control which labels are matched or retained.
  • Label Lookup: Looking up label names and values and resolving labels to references in an inverted index.
  • Label Matcher Evaluation: Evaluates label sets against and/or label matchers with regex and key-value matching semantics.
  • Label Matcher Matching: Evaluates label matchers and matcher predicates against incoming log entries.
  • Label Matching: Matches and pushes value-based and name-based label predicates, resolving shared names across matchers.
  • Label Name and Value Discovery: Enumerates the available label names and the values associated with labels across a store.
  • Label and Field Discovery: Discover available label names, label values, and detected fields for metrics and streams.
  • Label and Metadata Mapping: Attaches labels and metadata to outgoing log entries according to include_fields configuration.
  • Label-Based Access Control: Define label policies and enforce them via LBAC policy headers during queries.
  • Label-based Stream Selection: Select and iterate log streams filtered by labels, with default labels applied when unspecified.
  • Metric Label Filtering: Filtering of metric labels so that non-Prometheus labels are dropped.
  • Structured Metadata Handling: Associate, iterate, and validate structured metadata alongside chunk entries.

Chunk Storage and Retrieval

Encode, store, fetch, cache, and manage time-series and log data chunks across hot and cold storage.

  • Chunk Decoding: Decode stored chunk bytes back into individual samples via an iterator.
  • Chunk Encoding: The chunkenc feature provides a chunk abstraction that encodes, compresses, and reports metadata about stored data blocks.
  • Chunk Encoding and Decoding: Provides encoding and decoding of time-series data into chunk blocks, with validation of chunk integrity and capacity.
  • Chunk Fetching: A capability that fetches data chunks using chunk length and chunk collection parameters.
  • Chunk Fetching and Transfer: Fetch chunks between ingesters/stores including sizing information for transfer.
  • Chunk Flushing: Flushes in-memory log chunks to long-term storage, triggered by full, idle, max-age, forced, synced, or not-owned conditions.
  • Chunk Lifecycle Management: Index, check existence of, and remove chunks within tables, including per-series cleanup.
  • Chunk Retrieval: Fetches stored chunks for a user's series during query evaluation.
  • Chunk Rewriting: Rewrite chunk contents, optionally filtering entries, while preserving or dropping associated structured metadata.
  • Chunk Storage: Retrieve and persist data chunks, addressing each chunk by a derived key.
  • Chunk Storage and Retrieval: Write, read, and delete chunks of data through a metrics-instrumented chunk client.
  • Chunk and Results Caching: Users configure memcached-backed caching of chunks and query results, including cache freshness and replica settings.

Compaction, Retention, and Deletion

Compact stored data, enforce retention windows, and process data deletion requests.

  • Compaction Planning: Produces a plan of runs and sections to compact by grouping data until a convergence or minimum-size condition is met.
  • Compaction Scheduling: Drives periodic compaction work by starting, running, and stopping a scheduler that orchestrates compactor execution.
  • Delete Request Lifecycle: Create, list, and cancel log deletion requests, letting users manage the deletion of log data.
  • Delete Request Management: Create, list, and cancel Loki log deletion requests.
  • Delete Request Processing: Compactor-driven loop that consumes queued deletion jobs and processes outstanding delete requests through the job queue.
  • Deletion Manifest Building: Constructs a manifest of chunks and series to be deleted, with error handling, and validates cleanup of invalid manifests.
  • Deletion Sweep: Sweeps and removes deleted data as a background maintenance capability.
  • Log Compaction: Run periodic compaction of stored log data across the storage backends at a configured interval.
  • Log Compaction and Retention: Merge stored chunks and apply retention/deletion policies to reclaim storage.
  • Loki Log Compaction: The Loki compactor service consolidates stored log data.
  • Retention Enforcement: Applies configured per-tenant retention limits to a retention manager's configuration.
  • Retention Management: Applies configurable retention policies that expire old log data.

Alerting and Recording Rules

Load, evaluate, shard, and dispatch alerting and recording rules to Alertmanager and remote write.

  • Alert Notification Sending: Dispatches firing alerts to configured notifiers with tenant identity headers.
  • Alerting Rule Evaluation: Evaluates alerting rules periodically and reports the success or failure status of each rule evaluation.
  • Alerting Rules Management: Users define and manage alerting rules, including rule descriptions, grouping, expressions, and intervals that trigger alerts.
  • Alerting and Recording Rule Evaluation: Evaluate and manage alerting and recording rules to trigger alerts from log data.
  • Alerting and Recording Rules: Evaluates and manages alerting and recording rules via the ruler.
  • Alertmanager Configuration: Allows users to configure Alertmanager connection details, discovery, authentication, and TLS settings for sending alerts.
  • Rule Group Management: Create, read, and delete alerting/recording rule groups within a namespace.
  • Rule Group Name Management: Add and remove rule identifiers to and from rule group names, producing scoped group tokens.
  • Rule Sharding: Distribute and resolve rules across ruler instances using sharding so each instance serves its owned rules.
  • Rule Synchronization: Continuously sync rules from the ruler's datasource into the running ruler and keep them up to date.
  • Ruler Remote Write: Forward rule evaluation results to remote write endpoints with configurable queues, retries, and SigV4 auth.

Index Building, Shipping, and Compaction

Build, ship, merge, compact, and retain indexes that map series and labels to stored data.

  • Bucket Index Construction: The system builds searchable index structures over time-series buckets to enable data lookup.
  • Index Building: Builds a tenant-scoped index from posting entries, buffering and flushing index pointers to a destination path.
  • Index Compaction: Merges and consolidates index runs per tenant, tracking index merges by outcome, phase, and failure groups.
  • Index Compaction and Path Resolution: Compacts the index and resolves the resulting index file location for downstream upload.
  • Index Gateway: Serve index queries and deletes through the index gateway and its ring.
  • Index Merge: Executes an index merge operation that combines postings and stats from multiple sequence readers into a merged builder.
  • Index Metadata Caching: Cache per-index metadata with eviction of stale entries and bounded capacity.
  • Index Querying: Executes a query against an open index and returns results.
  • Index Retention Deletion: The service deletes eligible source indexes based on retention limits and compaction state.
  • Index Shipping: Uploads and downloads index files to/from object storage for distributed index delivery.
  • Postings Lookup: Query and retrieve postings lists that map label combinations to matching series.

Log Ingestion and Write Path

Accept, validate, rate-limit, shard, and durably persist incoming log streams through distributors and ingesters.

  • Distributor Tee Forwarding: Forwards ingested streams through the distributor tee mechanism to secondary destinations, with failure handling.
  • Ingest Limits Enforcement: Enforces per-tenant ingestion rate limits by checking tenant limits and sharding requests across distributor instances.
  • Ingestion Rate Limiting: Enforce per-tenant ingestion rate limits, reject over-limit streams and report discarded streams.
  • Log Ingestion: Ship input data to Loki for log aggregation and querying.
  • Log Ingestion (Push): Clients push log entries into the Loki-compatible store via the push endpoint.
  • Log Push: Push log payloads to an upstream endpoint, including authentication via bearer token, credentials, or path prefix.
  • Log Stream Ingestion: Ingests labeled log streams with timestamps, lines, and structured metadata into the log pipeline.
  • Log Stream Push/Ingestion: Accept and decode pushed log streams into structured entries.
  • Push Request Entry Ingestion: Construct entries from an incoming push request.
  • Stream Sharding: Distribute streams across shards and ingesters by computing shard assignments and stream sizes.
  • Write-Ahead Logging: Persists incoming data to a WAL with chunk flushing and configurable memory and size limits.

LogQL Query Execution

Parse, plan, shard, and execute instant and range LogQL queries returning logs, metrics, and streams.

  • Instant Query Execution: Users issue instant log queries and receive vector results for a single point in time.
  • Log Query: Query stored logs, including aggregated log results.
  • Log Query Execution: Execute ad-hoc log queries over a selected time range to retrieve matching log lines.
  • Log Query and Retrieval: Query stored log streams and return matching log lines using selectors and filtering.
  • Log Querying: Execute instant and range LogQL queries returning matrix, vector, and stream results.
  • LogQL Query Execution: Run instant and range LogQL queries against Loki and view results in various output modes.
  • LogQL Query Parsing: Parses LogQL log selectors, pipelines, label filters, line filters, parsers, and formatting stages into structured expressions.
  • Metric Query Evaluation: Parses and evaluates LogQL metric expressions including range and vector aggregations, binary operations, and distinct sketches.
  • Query Range API: Clients execute LogQL range queries against Loki through the query-frontend and querier pipeline with result caching.
  • Query Sharding: Shards large queries for parallel execution when supported by the engine.
  • Query Splitting and Parallelism: Splits incoming queries into sub-queries by time/interval and executes them with configurable parallelism limits.

Multi-Tier Caching

Cache chunks, query results, index data, and metadata across embedded, memcached, and Redis tiers with expiration and generation tracking.

  • Cache Expiration and Eviction: Automatic removal of stale or surplus cache entries via TTL and least-recently-used eviction policies.
  • Cache Generation Number Tracking: Retrieve and update the cache generation number used to coordinate cache state.
  • Cache Generation Tracking: The system exposes and updates a cache generation number so consumers can detect when cached results become stale.
  • Chunk and Results Caching: Users configure memcached-backed caching of chunks and query results, including cache freshness and replica settings.
  • Embedded Cache: Provides an in-process cache for query results and other cached data.
  • Memcached Client: Provides a Memcached client that can be configured, created, and used to store and retrieve cached values.
  • Metadata Caching: Caches file metadata by key so repeated metadata lookups avoid re-fetching, with availability checks for cacheability.
  • Object Metadata Caching: Cache and serve object metadata, reading only queried tenants to object storage.
  • Query Result Caching: Generates cache keys scoped to tenant limits so query results can be cached and reused.
  • Redis Cache: Constructs and manages a Redis-backed cache store with type selection and lifecycle control (store, fetch, stop).
  • Results Caching: Caches query results responses and serves cached responses when request conditions match.

Deployment and Configuration

Configure, validate, and deploy the system across topologies, container images, and Helm charts.

  • CLI Flag Registration: Registers CLI flags, optionally with a prefix, so users can configure the client from the command line.
  • Command-Line Flag Registration: Users can register command-line flags with or without a prefix to configure application behavior.
  • Config Validation: Validates configuration settings such as log key size, runs per task, compaction size, and task retries, rejecting invalid values.
  • Configuration Management: Exposes getters, setters, default values, and deletion for Drain algorithm configuration and limits.
  • Configuration Validation: Validate example configuration files and development cluster configuration.
  • Container Image Build: Builds Docker images for the Loki ecosystem components, including versions and images referenced by CI jobs.
  • Helm Chart Validation: Lint and test the Loki production Helm chart and its test image.
  • Loki Deployment: Deploys Loki in a chosen deployment topology (single binary or microservices) and mode (read/write/backend).
  • Loki Deployment Topology Selection: Supports selecting between deployment topologies such as single binary or separated read/write/backend components.
  • Runtime Configuration Loading: Loads and validates the runtime configuration including retention rules and default limits, with support for overrides.

Object Storage Backends

Read, write, list, and delete objects across pluggable cloud and filesystem storage backends.

  • Azure Storage Configuration: Users can configure Azure Storage connection settings and authentication for storage access.
  • Filesystem Storage: Backs storage with the local filesystem using a configured directory.
  • GCS Client Configuration: Users configure and construct a GCS object client with credentials, endpoint, retry, and transport settings.
  • Object Listing: Enumerate stored objects, optionally scoped to tables, through the object metastore.
  • Object Listing and Iteration: List and iterate over stored objects, including table-of-contents listing and error reporting for failed enumerations.
  • Object Retrieval: Retrieve individual objects and ranges of data from storage, including error handling for failed reads.
  • Object Storage Backend: Persists chunk, rules, and other data to configurable object storage with path and directory settings.
  • Object Storage CRUD: Create, read, check existence of, and delete objects in a storage bucket.
  • Object Storage Client Provisioning: Creates a configured object storage client for a chosen backend provider so other components can read and write objects.
  • S3 Object Storage: Configures connectivity and authentication to S3-compatible object storage for persistence.

Bloom Filter Indexing and Filtering

Build, store, ship, prefetch, and query bloom filters to prune chunks and accelerate log searches.

  • Bloom Block Prefetching: Prefetch bloom blocks across addresses in parallel to warm caches before queries are served.
  • Bloom Filter Building and Gateway: Build, store, plan, and serve bloom filters to accelerate queries.
  • Bloom Filter Chunk Reference Filtering: Filters chunk references against bloom filters for a tenant to skip chunks that cannot contain matching data.
  • Bloom Filter Filtering: Build, ship, and query bloom filters to accelerate and reduce log search workloads.
  • Bloom Filter Generation: Generates bloom filters for a given schema and key range, enabling clients to load and query probabilistic set membership data.
  • Bloom Filter Indexing: Creates, stores, and round-trips bloom filter postings for indexed value lookup, including bitmap correctness and sizing.
  • Bloom Filter Matching: Evaluate membership queries against a bloom filter for efficient candidate matching.
  • Bloom Gateway Service: A hosted RPC service exposing chunk-reference filtering and bloom block prefetch to clients.
  • Bloom Shipper: Supports shipping Bloom filter data to object storage.

Distributed Ring and Cluster Coordination

Coordinate component membership, sharding, and replication across a distributed cluster via rings and KV stores.

  • Cluster Membership: Managing cluster membership and node discovery via memberlist gossip protocol.
  • Consistent Hash Ring Distribution: Distributes requests and data across instances using a consistent hash ring with configurable replication.
  • Distributed Hash Ring: Distributing data across cluster members using a hash ring with replication and heartbeats.
  • KV Store Coordination: Loki components coordinate through a configured key-value store for ring and lifecycler state.
  • Partition Ring Management: Maintains and rebuilds a partition ring, tracking which partitions are active or inactive.
  • Ring Lifecycle Management: Start, stop, and wait for ring services to run or terminate, and check their running state.
  • Ring Membership: Manages distribution of work across instances using replication and instance addressing.
  • Ring Membership and Sharding: Manage distributed component rings and memberlist-based cluster membership.
  • Shuffle Sharding: Ability to assign tenant requests to a reduced, randomized subring of instances so that failures are isolated across tenants.

Metrics, Dashboards, and Observability

Expose operational metrics, Grafana dashboards, and alerting for monitoring the running system.

  • Build Info Display: Exposes version, revision, build date, build user, branch, and Go version metadata about the running application.
  • Grafana Dashboards: Delivers Grafana dashboards for observing the system's metrics.
  • Metrics Collection: Collects and records counter, gauge, and histogram metrics through a configured metrics stage.
  • Metrics Collection and Description: Collect and describe the metrics exposed by the rule manager.
  • Metrics Dashboards: Users can view cluster and service metrics through Grafana dashboards backed by Prometheus.
  • Metrics Registration: Registers and unregisters metrics so they can be tracked and observed.
  • Monitoring and Dashboards: Deploy monitoring, self-monitoring, and dashboard resources for observability of the logging stack.
  • Observability Metrics: Exposes a metrics registry with counters, histograms, connections, queues, and write-busy instrumentation that operators observe.
  • Prometheus Metrics Exposition: Exposes application metrics in Prometheus text format for scraping and monitoring.

Multi-Tenancy and Limits

Isolate tenants, authenticate requests, and enforce per-tenant ingestion, query, and stream limits.

  • Authentication and Multi-Tenancy: Enforces tenant authentication and per-tenant limits across the system.
  • Ingestion Policy Enforcement: Applies per-tenant ingestion policies (rate limits, burst sizes, stream caps, sharding) to incoming streams via configurable stream mappings and overrides.
  • Multi-Tenancy: Isolating data and requests per tenant with authentication and per-tenant limits.
  • Organization Context Resolution: Resolve the organization identifier that scopes queries and deletion requests to a tenant.
  • Per-Tenant Limits Configuration: Define and expose per-tenant limits governing remote evaluation, remote write, and rule counts.
  • Per-Tenant Stream Count Limiting: Enforces and reports per-tenant stream count limits, resolving global versus local limits across ingest topologies.
  • Tenant Limits Configuration: Capability to define and look up tenant-level limits with defaults and overrides.
  • Tenant Limits Enforcement: Capability to enforce per-tenant limits against gateway queries.
  • Tenant Scoping: Associating data with a tenant so multiple isolated tenants can share the same store.

Columnar Storage, Encoding, and Postings

Encode, compress, bitpack, and persist columnar data, postings lists, and bitmap indexes.

  • Bitmap Encoding: Encodes integer sets into a compact bitmap representation, choosing between run-length and bitpacked encodings.
  • Bitmap Set Operations: Compute union, intersection, difference, and symmetric difference between bitmaps so callers can combine sets.
  • Bitpacked Encoding/Decoding: Encoding and decoding of values into and out of a bit-packed representation supporting nullable and non-nullable data with varying bit widths.
  • Block Compression: Applies compression to data blocks, exposing block and uncompressed sizes and a zstd-based compression encoding.
  • Columnar Data Encoding: Encoding of tabular data into a column-oriented format with per-column encoders, buffers, and dictionaries.
  • Elias-Fano Encoding: Build and append Elias-Fano encoded integer sequences for compact storage of monotonic data.
  • Postings Encoding: Encodes postings data into compact, sectioned binary columnar form for storage and retrieval.
  • Postings Merge: Multiple posting lists are merged into a single combined postings stream during index construction.

Data Object Storage and Sharding

Build, shard, fetch, and query compacted data objects and their streaming layouts.

  • Data Object Compaction Configuration: Configuration of data object compaction, including index-only and log-implied modes with combination validation.
  • Data Object Querying: Queries data objects across shards with sharding layout and parallelism controls.
  • Data Object Resolution: Resolves which data objects and time windows a query should read from by planning expressions and checking eligibility.
  • Data Object Sharding: Determines how data objects are partitioned into shards based on byte-size and layout limits.
  • Data Object Storage Layout: Controls data object storage layout, shard sizes, and query start time for objects.
  • Data Object Storage and Exploration: Build, cache, compact, and explore data objects in object storage.
  • Data Object Store: A store for managing and accessing data objects with configurable options, streaming, and metadata caching.
  • Stream Layout Mapping: Compute shard layouts and map streams to their target objects for output.

Query Frontend and Scheduling

Front, queue, schedule, split, and cache incoming queries while coordinating querier workers.

  • Frontend Query Execution: The frontend accepts incoming requests, tracks them as in-progress, and returns a query result and status to the caller.
  • Frontend Query Routing: Routes and rewrites incoming queries through the Loki frontend (v1/v2) to downstream query services.
  • Querier Worker Management: Registration and management of querier workers connecting back to the query frontend, including dynamic concurrency and address tracking.
  • Query Frontend: A query frontend capability that provides query handling, results caching, and query ingester window configuration.
  • Query Parallelism Control: Constrains the degree of parallelism applied when executing queries.
  • Query Result Caching: Caches and retrieves results for instant metric, range, series, volume, index stats, and generic queries to accelerate repeated lookups.
  • Query Scheduling: Queue and schedule queries via the query scheduler and its ring.
  • Query Splitting and Sharding: Splits queries by time range or instant shards and reconstructs responses without changing user-visible results.

Live Log Tailing

Stream live log entries to clients in real time over websocket and tail connections.

  • Live Log Tailing: Clients open a streaming tail connection to receive matching log entries in real time.
  • Log Stream Tailing: Subscribe to and stream log data to clients, with the ability to drop streams when overwhelmed or closed.
  • Log Tail: Stream newly arriving log entries from ingesters and manage active tailers.
  • Log Tailing: Streams newly arriving log entries to a client, governed by a tail delay and bounded by line limits.
  • Loki API Tail: Clients can stream logs in real time via the /loki/api/v1/tail endpoint.
  • Tailer Management: Attach, track, and close tailers that follow new stream data as it arrives.
  • WebSocket Messaging: Provides bidirectional message exchange over a websocket connection, including reading and writing messages.
03 · Size

What was analyzed

  • Go · 2,591 files · 653,227 lines
  • Text · 16 files · 309,614 lines
  • JSON · 48 files · 85,595 lines
  • Markdown · 268 files · 52,112 lines
  • YAML · 293 files · 22,737 lines
  • JavaScript · 2 files · 4,038 lines
  • Protocol Buffers · 41 files · 2,565 lines
  • Bourne Shell · 23 files · 1,458 lines
  • Ruby · 13 files · 1,455 lines
  • make · 14 files · 961 lines
  • HCL · 8 files · 864 lines
  • Other (9 languages) · 43 files · 2,207 lines

Whole repository: 3,578 files. cloc 2.06 counts 1,136,833 lines of code in 3,360 of them (1,375,233 lines with comments and blanks).

04 · Source

Where this came from

Extracted by CoreStory V3 (build 4470 with two small engine fixes) using DeepSeek-V4.1-Flash from the public repository at commit 9ea9b3b on 2026-11-15. Capability descriptions are from the same run.

Code excerpts are from the upstream repository and remain under its AGPL-3.0 license. CoreStory is not affiliated with or endorsed by Grafana Labs.

Want this for your own codebase?

Talk to us about mapping the business rules in your repositories.

Talk to an expert