README
¶
flatten
flatten is a Go package for working with hierarchical key-value data structures, primarily for handling nested data in
formats like JSON, YAML, or TOML.
Features
1. Data Flattening
- Converts nested maps, slices, and arrays into a flat
map[string]string. - Uses dot notation for maps (e.g.,
db.hosts). - Uses index notation for arrays/slices (e.g.,
hosts[0]). - Example:
{"db": {"hosts": ["a", "b"]}}becomes{"db.hosts[0]": "a", "db.hosts[1]": "b"}.
2. Path Handling
- Defines a
Pathabstraction, representing hierarchical keys as a sequence of typed segments (map keys or array indices). - Supports parsing string paths (e.g.,
"foo.bar[0]") intoPathobjects. - Supports converting
Pathobjects back into string paths.
3. Storage Management
- The
Storagetype manages a collection of flattened key-value pairs. - Internally builds and maintains a hierarchical tree structure to prevent key conflicts.
- Associates values with their source files, supporting multi-file merging and source tracking.
4. Querying
- Provides helper methods for retrieving values.
- Checks for the existence of keys.
- Enumerates subkeys.
- Iterates in a deterministic order.
Typical Use Cases
- Standardizing configuration files from multiple sources into a flat key-value map for comparison, merging, or diffing.
- Querying nested data with simple string paths, avoiding direct reflection or manual traversal of nested maps.
- Building tools that unify structured data from multiple files while preserving source information and preventing conflicts.
Example
package main
import (
"fmt"
"github.com/go-spring/stdlib/flatten"
)
func main() {
// Create a nested data structure
data := map[string]interface{}{
"database": map[string]interface{}{
"host": "localhost",
"port": 5432,
"credentials": map[string]interface{}{
"username": "admin",
"password": "secret",
},
},
"features": []interface{}{
"feature1",
"feature2",
map[string]interface{}{
"name": "feature3",
"enabled": true,
},
},
}
// Flatten the data
flat := flatten.Flatten(data)
// Print flattened results
for key, value := range flat {
fmt.Printf("%s: %s\n", key, value)
}
// Use Storage to manage data
storage := flatten.NewStorage()
fileID := storage.AddFile("config.yaml")
// Set values
storage.Set("database.host", "localhost", fileID)
storage.Set("database.port", "5432", fileID)
// Retrieve values
host := storage.Get("database.host")
fmt.Printf("Database host: %s\n", host)
// Check if a key exists
if storage.Has("database.host") {
fmt.Println("Database host exists")
}
}
License
Apache License 2.0
Documentation
¶
Overview ¶
Package flatten provides utilities for validating the *structure* of hierarchical configuration data by converting it into a flat key space.
The primary design goal of this package is *structural key validation* for structured configuration formats such as JSON, YAML, or TOML. Rather than operating directly on nested maps and slices via reflection, flatten converts hierarchical data into a flat key/value representation while retaining enough structural metadata to:
- Validate key paths against structural constraints
- Detect property conflicts early (e.g. map vs array, value vs container)
- Support deterministic traversal and querying
- Track value provenance across multiple configuration sources
Flattened keys use:
- Dot notation for map/object fields: "db.host"
- Index notation for arrays/slices: "servers[0].port"
For example:
{"db": {"hosts": ["a", "b"]}}
becomes:
{
"db.hosts[0]": "a",
"db.hosts[1]": "b",
}
Internally, the package deliberately separates *structure* from *values*:
- Structure is tracked by an internal hierarchical tree that models paths as typed segments (map keys or array indices) and enforces consistency.
- Leaf values are stored in flat maps keyed by normalized string paths.
This separation allows flatten to perform strict structural validation without duplicating data, while still providing a simple flat representation for querying, comparison, merging, and diffing.
Key components include:
- Path: a typed abstraction over hierarchical keys, supporting parsing from and formatting to string paths such as "foo.bar[0]".
- Storage: a container for flattened key/value pairs that maintains the internal structure tree, prevents conflicting writes, and associates values with their source files for provenance tracking.
- Query helpers: utilities for existence checks, subkey enumeration, and deterministic iteration.
Typical use cases:
- Normalizing configuration files from multiple sources for comparison, merging, or diffing.
- Querying deeply nested configuration data using simple string paths without dealing with reflection or nested map structures directly.
- Building configuration tooling that requires strict structural guarantees and reproducible traversal order.
Index ¶
- func Flatten(m map[string]any) map[string]string
- func JoinPath(path []Path) string
- type Path
- type PathType
- type Storage
- func (s *Storage) AddFile(file string) int8
- func (s *Storage) Data() map[string]string
- func (s *Storage) Dump(w io.Writer) error
- func (s *Storage) Exists(key string) bool
- func (s *Storage) Get(key string, def ...string) string
- func (s *Storage) Keys() []string
- func (s *Storage) Lookup(key string) (string, bool)
- func (s *Storage) Merge(p *Storage) error
- func (s *Storage) MergeMap(data map[string]any, file string) error
- func (s *Storage) Set(key string, val string, file int8) error
- func (s *Storage) SubKeys(key string) (_ []string, err error)
- func (s *Storage) SubTree(key string) (map[string]string, error)
- type ValueInfo
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Flatten ¶
Flatten flattens a nested map[string]any into a map[string]string.
This function is intended for data produced by encoding/json.Unmarshal, where values are limited to the following kinds:
- map[string]any
- []any
- primitive JSON types (bool, number, string, nil)
Structs, custom types, and non-string map keys are explicitly out of scope.
Flattening rules:
Nested maps are expanded using dot notation: {"a": {"b": 1}} -> "a.b" = "1"
Slices (and arrays, although arrays do not originate from json.Unmarshal) are expanded using index notation: {"a": [1, 2]} -> "a[0]" = "1", "a[1]" = "2"
Nil values (both untyped nil and typed nil) are represented as "<nil>".
Empty (zero-length but non-nil) maps are represented as "{}".
Empty (zero-length but non-nil) slices are represented as "[]".
Primitive values are converted to strings using deterministic, Go-native formatting (strconv).
The resulting map is intended for display-oriented use cases such as logging, diffing, diagnostics, or inspection. The output is not reversible and must not be treated as a lossless serialization format.
Types ¶
type Path ¶
type Path struct {
// Whether the element is a key or an index.
Type PathType
// Actual key or index value as a string.
// For PathTypeKey, it's the key string;
// for PathTypeIndex, it's the index number as a string.
Elem string
}
Path represents a single segment in a parsed key path. A path is composed of multiple Path elements that can be joined or split. For example, "foo.bar[0]" parses into:
[{Type: PathTypeKey, Elem: "foo"},
{Type: PathTypeKey, Elem: "bar"},
{Type: PathTypeIndex, Elem: "0"}].
func SplitPath ¶
SplitPath parses a hierarchical key string into a slice of Path objects. It supports dot-notation for maps and bracket-notation for arrays. Examples:
"foo.bar[0]" -> [{Key:"foo"}, {Key:"bar"}, {Index:"0"}]
"a[1][2]" -> [{Key:"a"}, {Index:"1"}, {Index:"2"}]
Rules:
- Keys must be non-empty strings without spaces.
- Indices must be unsigned integers (no sign, no decimal).
- Empty maps/slices are not special-cased here.
- Returns an error if the key is malformed (e.g. unbalanced brackets, unexpected characters, or empty keys if disallowed).
type PathType ¶
type PathType int8
PathType represents the type of a path element in a hierarchical key. A path element can either be a key (map field) or an index (array/slice element).
type Storage ¶
type Storage struct {
// contains filtered or unexported fields
}
Storage is the central data structure of this package.
It maintains three logically distinct layers:
- root – a hierarchical tree that models *only structure*
- data – flattened leaf key/value pairs
- empty – flattened keys representing empty containers or nil values
empty is tracked separately to preserve leaf semantics for empty containers and to prevent illegal path extension.
Additionally, Storage tracks file provenance using a compact int8 index.
Core invariants:
- root contains no values, only structure
- data contains only concrete leaf values
- empty contains only leaf paths representing [], {}, or <nil>
- a single path cannot simultaneously be a container and a value
func (*Storage) AddFile ¶
AddFile registers a configuration source and assigns it a compact int8 ID.
If the file has already been registered, the existing ID is returned.
The total number of files is limited to 127, which is considered sufficient for typical configuration-merging scenarios.
func (*Storage) Data ¶
Data returns all flattened key/value pairs currently stored in the Storage.
The result includes both concrete values and empty-container markers (e.g. "[]", "{}", "<nil>"), and is sufficient to reconstruct the Storage structure when re-inserted via Set.
func (*Storage) Dump ¶ added in v0.0.8
Dump writes the contents of Storage to the given writer in a human-readable, deterministic format grouped by source file.
The output is intended for inspection and debugging purposes only. It is not a stable serialization format and should not be parsed or relied upon for programmatic consumption.
func (*Storage) Exists ¶ added in v0.0.9
Exists determines whether a key or path *structurally exists* within the Storage.
This check is intentionally permissive: the key does not need to represent a valid leaf path. Container nodes (including arrays) and intermediate structural nodes are considered existing as long as they are compatible with the current structure.
func (*Storage) Get ¶
Get retrieves the value associated with a flattened key, or returns a default.
Only concrete leaf values are considered valid lookup targets. If the key does not exist, the first provided default value (if any) is returned.
func (*Storage) Keys ¶
Keys returns all flattened keys currently stored in the Storage.
This includes both concrete values and empty-container markers. The result is sorted lexicographically to ensure deterministic iteration.
func (*Storage) Lookup ¶ added in v0.0.7
Lookup retrieves the value associated with a flattened key.
The key must refer to a concrete leaf value. Intermediate nodes and empty-container markers are intentionally excluded.
func (*Storage) Merge ¶
Merge imports all values from another Storage instance.
File identities from the source Storage are remapped to local IDs while preserving provenance semantics.
Note: When merging Storages with identical filenames, the provenance information may become ambiguous as files with the same name from different sources will share the same file ID in the merged result.
func (*Storage) MergeMap ¶
MergeMap flattens a nested map and inserts all resulting key/value pairs into the Storage under the given file identity.
Structural conflicts are detected eagerly during insertion.
func (*Storage) Set ¶
Set inserts or updates a flattened key/value pair while enforcing structural consistency.
During insertion, the key path is validated against the internal tree to ensure that:
- a value is not written where a container already exists
- a map branch is not reinterpreted as an array branch (or vice versa)
- no partial prefix of the key conflicts with existing structure
Any structural violation results in an immediate error.
func (*Storage) SubKeys ¶
SubKeys returns the immediate child keys of a container path.
The path may refer to either a map or an array; child keys are returned uniformly as strings (map keys or numeric indices).
Behavior:
- If the path refers to a leaf value, an error is returned
- If the path does not exist, nil is returned
- If the path refers to an empty container, an empty slice is returned
An empty key indicates traversal starting from the root node.
func (*Storage) SubTree ¶ added in v0.0.8
SubTree extracts all descendant key/value pairs under the given key.
Returned keys have the prefix removed. The result may include empty container markers ([] / {} / <nil>), allowing reconstruction of a Storage instance if desired.
The key must not be empty.
type ValueInfo ¶
ValueInfo stores a flattened value together with its source information.
The File field records the Storage-local numeric identifier of the configuration file from which the value originated. This enables provenance tracking and deterministic merging behavior across multiple inputs.
Source Files