datahash

datahash

datahash provides high-performance, customizable hashing for arbitrary Go values with zero dependencies.
It produces consistent 64-bit hashes by recursively traversing data structures.

Features

• Consistent 64-bit hashing of any Go value.
• Supports structs, slices, iter.Seq, and iter.Seq2 as unordered sets (default: ordered).
• Supports custom hash logic via datahash.HashWriter or encoding.BinaryMarshaler interface.
• Integrates with: encoding.BinaryMarshaler, encoding.TextMarshaler, encoding/json.Marshaler, fmt.Stringer.
• Handles cyclic data structures safely (pointer tracking).
• High performance: type caching and hasher pooling.

Installation

go get -u github.com/go-sqlt/datahash

Usage

package main

import (
	"fmt"
	"math/big"

	"github.com/cespare/xxhash/v2"
	"github.com/go-sqlt/datahash"
)

type MyStruct struct {
	Name  string `datahash:"-"`
	Age   int
	Float *big.Float
}

func main() {
	hasher := datahash.New(xxhash.New, datahash.Options{
		UnorderedStruct: false,
		UnorderedSlice:  false,
		UnorderedArray:  false,
		UnorderedSeq:    false,
		UnorderedSeq2:   false,
		Text:            true, // big.Float implements encoding.TextMarshaler
		JSON:            false,
		String:          false,
		ZeroNil:         false,
		IgnoreZero:      false,
	})

	alice, _ := hasher.Hash(MyStruct{Name: "Alice", Age: 30, Float: big.NewFloat(1.23)})
	bob, _ := hasher.Hash(MyStruct{Name: "Bob", Age: 30, Float: big.NewFloat(1.23)})

	fmt.Println(alice, alice == bob) // Output: 13125691809697640472 true
}

Options

Notes

• By default struct fields are hashed in their declared order.
• Maps and unordered sets are folded using XOR for order-independence.
• Cyclic pointers are detected and skipped safely.
• Use datahash:"-" to exclude fields from hashing.
• Implement datahash.HashWriter or encoding.BinaryMarshaler for custom hash behavior.
• Unexported fields cannot be used with custom marshalers. (!)

Benchmark

These benchmarks demonstrate that datahash is 5–6× faster and significantly more memory-efficient than both mitchellh/hashstructure and gohugoio/hashstructure.

Notably, serializing to JSON and hashing the output with cespare/xxhash/v2 performs reasonably well and can be a viable alternative in some cases. However, JSON-based approaches cannot guarantee consistent map ordering during hashing.

By contrast, datahash always hashes maps as unordered sets and offers the same unordered (set-like) treatment for structs, slices, arrays, and iterators—making it more deterministic and flexible for complex data structures.

go test -bench=. -benchmem
goos: darwin
goarch: arm64
pkg: github.com/go-sqlt/datahash
cpu: Apple M3 Pro
BenchmarkHashers/Simple_struct_/Datahash+fnv____-12             18659014                63.84 ns/op            0 B/op          0 allocs/op
BenchmarkHashers/Simple_struct_/Mitchellh+fnv___-12              3214767               372.2 ns/op           248 B/op         17 allocs/op
BenchmarkHashers/Simple_struct_/Gohugoio+fnv____-12              3149439               378.2 ns/op           248 B/op         17 allocs/op
BenchmarkHashers/Simple_struct_/JSON+fnv________-12             12832291                92.47 ns/op           32 B/op          1 allocs/op
BenchmarkHashers/Simple_struct_/Datahash+xxhash_-12             16018688                75.47 ns/op            0 B/op          0 allocs/op
BenchmarkHashers/Simple_struct_/Mitchellh+xxhash-12              2801804               419.7 ns/op           320 B/op         17 allocs/op
BenchmarkHashers/Simple_struct_/Gohugoio+xxhash_-12              3303172               363.2 ns/op           280 B/op         13 allocs/op
BenchmarkHashers/Simple_struct_/JSON+xxhash_____-12             14811546                80.85 ns/op           32 B/op          1 allocs/op
BenchmarkHashers/Complex_struct/Datahash+fnv____-12              2339106               511.2 ns/op           112 B/op          3 allocs/op
BenchmarkHashers/Complex_struct/Mitchellh+fnv___-12               420381              2806 ns/op            1824 B/op        116 allocs/op
BenchmarkHashers/Complex_struct/Gohugoio+fnv____-12               402007              2910 ns/op            1816 B/op        115 allocs/op
BenchmarkHashers/Complex_struct/JSON+fnv________-12              1000000              1075 ns/op             402 B/op          4 allocs/op
BenchmarkHashers/Complex_struct/Datahash+xxhash_-12              2140148               557.9 ns/op           112 B/op          3 allocs/op
BenchmarkHashers/Complex_struct/Mitchellh+xxhash-12               382479              3082 ns/op            1896 B/op        116 allocs/op
BenchmarkHashers/Complex_struct/Gohugoio+xxhash_-12               428434              2707 ns/op            1632 B/op         87 allocs/op
BenchmarkHashers/Complex_struct/JSON+xxhash_____-12              1549033               773.6 ns/op           402 B/op          4 allocs/op
PASS
ok      github.com/go-sqlt/datahash     19.107s