cleve

package module
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 18, 2025 License: GPL-3.0 Imports: 24 Imported by: 0

README

Cleve — sequencing run database

go build and test golangci-lint


Cleve is a database mainly for sequencing data, with an accompanying CLI, API and web-based dashboard. The goal of Cleve is to make management of sequencing runs with their associated samples and metadata effortless. This is achieved by storing metadata for the sequencing runs, including QC data.

Interactive visualisations of various parameters are included in the dashboard, albeit rudimentary at the moment. In essence, this provides some of the same features as Illumina Sequencing Analysis Viewer, but the difference is that it is much faster, and there is also the possibility to visualise QC parameters across runs. These visualisations provide a quick way of visually evaluate the state of a sequencing run.

Requirements

  • A mongodb instance

Installation

The easiest way to build Cleve is to use make:

git clone https://github.com/gmc-norr/cleve
cd cleve
make

The manual steps required are:

go generate ./...
go build -o ./bin/cleve ./cmd/cleve

The resulting binary is ./bin/cleve.

Configuration

Cleve looks for a yaml config file at startup. The following locations are checked in this order:

  • /etc/cleve/config.yaml
  • $HOME/.config/cleve/config.yaml
  • $PWD/config.yaml

The first config file that is found will be used, and the application will exit with an error if no config file is found. The config can also be supplied with the -c/--config flag.

The config file has the following content:

# Mongo database configuration
database:
  host: localhost   # mongodb host
  port: 27017       # mongodb port; 27017 is the default
  user: cleve       # database user
  password: secret  # password for user
  name: cleve       # database name

# If set, logs will be written to this file in addition to stdout
# If not set, logs will only be written to stdout
logfile: cleve.log

# Host and port where cleve will be served
host: 127.0.0.1
port: 8080

The only part that doesn't have decent defaults is the database. If any required values are undefined the application will exit with an error.

CLI

Interact with the sequencing database

Usage:
  cleve [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  db          Database management
  help        Help about any command
  key         API key management
  platform    Manage sequencing platforms
  run         Interact with sequencing runs
  samplesheet Interact with sample sheets
  serve       Serve the cleve api

Flags:
  -c, --config string   config file
  -h, --help            help for cleve
  -v, --version         version for cleve

Use "cleve [command] --help" for more information about a command.

Use the --help flag on the command line for complete documentation of all commands.

Serving the dashboard and the API

In order to serve the API and the dashboard, run

cleve serve

In a production environment, it is highly recommended to set this up with a reverse proxy.

Once everything is up and running, the API documentation can be found at <cleve-host>:<cleve-port>/api.

Development

Requirements

In addition to the general requirments, the following are recommended in a development environment:

Testing

Unit tests can be run with make test or go test ./....

Where does the name come from?

The name cleve is a tribute to what many consider to be the first female librarian in Sweden, Cecilia Cleve.

Documentation

Index

Constants

This section is empty.

Variables

View Source
var ValidRunStates = map[string]RunState{
	"new":        New,
	"ready":      Ready,
	"pending":    Pending,
	"complete":   Complete,
	"incomplete": Incomplete,
	"error":      Error,
	"moved":      Moved,
	"moving":     Moving,
	"unknown":    Unknown,
}

Functions

func GetAPIDoc

func GetAPIDoc() []byte

func GetAssetFS

func GetAssetFS() (fs.FS, error)

func GetTemplateFS

func GetTemplateFS() (fs.FS, error)

func GetVersion added in v0.2.0

func GetVersion() string

func MostRecentSamplesheet

func MostRecentSamplesheet(path string) (string, error)

Find the SampleSheet with the most recent modification time in a directory. The file name must be on the format `SampleSheet*.csv`.

Types

type APIKey

type APIKey struct {
	Key     string
	User    string
	Created time.Time
}

func NewAPIKey

func NewAPIKey(user string) *APIKey

type Analysis

type Analysis struct {
	AnalysisId string           `bson:"analysis_id" json:"analysis_id"`
	Path       string           `bson:"path" json:"path"`
	State      RunState         `bson:"state" json:"state"`
	Summary    *AnalysisSummary `bson:"summary" json:"summary"`
}

func NewAnalysis

func NewAnalysis(path string, state RunState, data []byte) (Analysis, error)

type AnalysisSummary

type AnalysisSummary struct {
	RunID           string `bson:"run_id" json:"run_id"`
	Result          string `bson:"result" json:"result"`
	SoftwareVersion string `bson:"software_version" json:"software_version"`
	Workflows       []struct {
		WorkflowName      string `bson:"workflow_name" json:"workflow_name"`
		ReportAggregation string `bson:"report_aggregation" json:"report_aggregation"`
		Samples           []struct {
			SampleID          string `bson:"sample_id" json:"sample_id"`
			BclToFastq        string `bson:"bcl_to_fastq" json:"bcl_to_fastq"`
			OraCompression    string `bson:"ora_compression" json:"ora_compression"`
			SecondaryAnalysis string `bson:"secondary_analysis" json:"secondary_analysis"`
			ReportGeneration  string `bson:"report_generation" json:"report_generation"`
		} `bson:"samples" json:"samples"`
	} `bson:"workflows" json:"workflows"`
}

func ParseAnalysisSummary

func ParseAnalysisSummary(data []byte) (AnalysisSummary, error)

type Gene added in v0.5.0

type Gene struct {
	HGNC    int
	Symbol  string
	Aliases []string
}

type GenePanel added in v0.5.0

type GenePanel struct {
	GenePanelVersion `bson:",inline" json:",inline"`
	Id               string    `json:"id"`
	Name             string    `json:"name"`
	Description      string    `json:"description"`
	Categories       []string  `json:"categories"`
	Genes            []Gene    `json:"genes,omitzero"`
	Archived         bool      `json:"archived"`
	ArchivedAt       time.Time `bson:",omitzero" json:"archived_at,omitzero"`
}

func GenePanelFromCsv added in v0.5.0

func GenePanelFromCsv(r io.Reader) (GenePanel, error)

func GenePanelFromTsv added in v0.5.0

func GenePanelFromTsv(r io.Reader) (GenePanel, error)

func GenePanelFromYaml added in v0.5.0

func GenePanelFromYaml(r io.Reader) (GenePanel, error)

func NewGenePanel added in v0.5.0

func NewGenePanel(name string, description string) GenePanel

func (*GenePanel) Add added in v0.5.0

func (p *GenePanel) Add(gene Gene)

func (*GenePanel) AddCategory added in v0.5.0

func (p *GenePanel) AddCategory(category string)

func (GenePanel) Validate added in v0.5.0

func (p GenePanel) Validate() error

type GenePanelVersion added in v0.5.0

type GenePanelVersion struct {
	Version Version   `bson:"-" json:"version"`
	Date    time.Time `json:"date"`
}

type PaginationFilter added in v0.3.0

type PaginationFilter struct {
	Page     int `form:"page,default=1"`
	PageSize int `form:"page_size"`
}

Pagination filtering.

func (PaginationFilter) Validate added in v0.3.0

func (f PaginationFilter) Validate() error

type PaginationMetadata

type PaginationMetadata struct {
	TotalCount int `bson:"total_count" json:"total_count"`
	Count      int `bson:"count" json:"count"`
	Page       int `bson:"page" json:"page"`
	PageSize   int `bson:"page_size" json:"page_size"`
	TotalPages int `bson:"total_pages" json:"total_pages"`
}

type PanelFilter added in v0.5.0

type PanelFilter struct {
	Category  string `form:"category"`
	Name      string `form:"name"`
	NameQuery string `form:"name_query"`
	Gene      string `form:"gene"`
	GeneQuery string `form:"gene_query"`
	HGNC      string `form:"hgnc"`
	Version   string `form:"version"`
	Archived  bool   `form:"archived"`
}

func NewPanelFilter added in v0.5.0

func NewPanelFilter() PanelFilter

type Pipeline added in v0.3.0

type Pipeline struct {
	Name    string `bson:"name" json:"name"`
	Version string `bson:"version" json:"version"`
	URL     string `bson:"url" json:"url"`
}

Pipeline represents an analysis pipeline.

type Platform

type Platform struct {
	Name          string
	InstrumentIds []string
	Aliases       []string
	ReadyMarker   string
	RunCount      int
}

Platform represents a sequencing platform.

func (*Platform) UnmarshalBSON added in v0.4.0

func (p *Platform) UnmarshalBSON(data []byte) error

UnmarshalBSON unmarshals a BSON representation into a Platform struct.

type Platforms added in v0.4.0

type Platforms struct {
	Platforms []Platform
	// contains filtered or unexported fields
}

Platforms represents a collection of sequencing platforms.

func (*Platforms) Add added in v0.4.0

func (p *Platforms) Add(platform Platform)

Add adds a platform to the collection.

func (Platforms) Condense added in v0.4.0

func (p Platforms) Condense() Platforms

Condense merges platforms by name. This collects all the instrument IDs and aliases for platforms with the same name, as well as sums up the run count for all individual instruments. The function will leave the original object intact and return a new Platforms object. Platforms will be sorted lexicographically by name.

func (Platforms) Get added in v0.4.0

func (p Platforms) Get(name string) (platform Platform, ok bool)

Get retrieves a platform by name. If a platform with that name is found, the platform is returned with ok set to true. Otherwise a default struct for the platform is returned with ok set to false.

func (Platforms) Names added in v0.4.0

func (p Platforms) Names() []string

Names returns a slice of platform names that are present in the collection. If the platforms have not been condensed there might be duplicates in the resulting slice.

type QcFilter

type QcFilter struct {
	RunId      string    `form:"run_id"`
	RunIdQuery string    `form:"run_id_query"`
	Platform   string    `form:"platform"`
	StartDate  time.Time `form:"start_time"`
	EndDate    time.Time `form:"end_time"`
	PaginationFilter
}

QC filtering.

func NewQcFilter added in v0.4.0

func NewQcFilter() QcFilter

func (QcFilter) UrlParams

func (f QcFilter) UrlParams() string

type QcResult

type QcResult struct {
	PaginationMetadata `bson:"metadata" json:"metadata"`
	InteropSummary     []interop.InteropSummary `bson:"interop" json:"interop"`
}

type Run

type Run struct {
	RunID            string                `bson:"run_id" json:"run_id"`
	ExperimentName   string                `bson:"experiment_name" json:"experiment_name"`
	Path             string                `bson:"path" json:"path"`
	Platform         string                `bson:"platform" json:"platform"`
	Created          time.Time             `bson:"created" json:"created"`
	StateHistory     []TimedRunState       `bson:"state_history" json:"state_history"`
	SampleSheet      *SampleSheetInfo      `bson:"samplesheet,omitempty" json:"samplesheet"`
	SampleSheetFiles []SampleSheetInfo     `bson:"samplesheets,omitempty" json:"samplesheets"`
	RunParameters    interop.RunParameters `bson:"run_parameters,omitzero" json:"run_parameters,omitzero"`
	RunInfo          interop.RunInfo       `bson:"run_info,omitzero" json:"run_info,omitzero"`
	Analysis         []*Analysis           `bson:"analysis,omitempty" json:"analysis,omitempty"`
	AnalysisCount    int                   `bson:"analysis_count" json:"analysis_count"`
}

Run represents an Illumina sequencing run.

func (*Run) UnmarshalBSON

func (r *Run) UnmarshalBSON(data []byte) error

Unmarshals a BSON representation of a run. This supports schema version 1 and 2. If the schema verison is not defined in the document, it is assumed to be version 1. The goal is to eventually deprecate version 1.

type RunCompletionStatus

type RunCompletionStatus struct {
	Status  string
	Message string
}

func ParseRunCompletionStatus

func ParseRunCompletionStatus(data []byte) (RunCompletionStatus, error)

func ReadRunCompletionStatus

func ReadRunCompletionStatus(filename string) (RunCompletionStatus, error)

type RunFilter

type RunFilter struct {
	RunID            string    `form:"run_id"`
	RunIdQuery       string    `form:"run_id_query"`
	Brief            bool      `form:"brief"`
	Platform         string    `form:"platform"`
	State            string    `form:"state"`
	From             time.Time `form:"from"`
	To               time.Time `form:"to"`
	PaginationFilter `form:",inline"`
}

Run filtering.

func NewRunFilter added in v0.4.0

func NewRunFilter() RunFilter

func (RunFilter) UrlParams

func (f RunFilter) UrlParams() string

Convert a run filter to URL query parameters.

type RunResult

type RunResult struct {
	PaginationMetadata `bson:"metadata" json:"metadata"`
	Runs               []*Run `bson:"runs" json:"runs"`
}

type RunState

type RunState int
const (
	New RunState = iota
	Ready
	Pending
	Complete
	Incomplete
	Error
	Moved
	Moving
	Unknown
)

func (RunState) MarshalBSONValue

func (r RunState) MarshalBSONValue() (bsontype.Type, []byte, error)

func (RunState) MarshalJSON

func (r RunState) MarshalJSON() ([]byte, error)

func (*RunState) Set

func (s *RunState) Set(v string) error

func (RunState) String

func (s RunState) String() string

func (*RunState) Type

func (s *RunState) Type() string

func (*RunState) UnmarshalBSONValue

func (r *RunState) UnmarshalBSONValue(t bsontype.Type, data []byte) error

func (*RunState) UnmarshalJSON

func (r *RunState) UnmarshalJSON(data []byte) error

type RunningSummary

type RunningSummary[T uint32 | uint64 | float32 | float64] struct {
	Sum T

	Mean float64
	// contains filtered or unexported fields
}

func NewRunningSummary

func NewRunningSummary[T uint32 | uint64 | float32 | float64](weighted bool) *RunningSummary[T]

func (*RunningSummary[T]) Push

func (v *RunningSummary[T]) Push(x T, weight ...T) error

func (RunningSummary[T]) SD

func (v RunningSummary[T]) SD() float64

func (RunningSummary[T]) Var

func (v RunningSummary[T]) Var() float64

type Sample added in v0.3.0

type Sample struct {
	// Sample name. If missing it should be set to the sample ID.
	Name string `bson:"name" json:"name"`
	// Sample ID as listed in the samplesheet.
	Id string `bson:"id" json:"id"`
	// Paths to fastq files related to the sample.
	Fastq []string `bson:"fastq" json:"fastq"`
	// Analyses associated with the sample.
	Analyses []*SampleAnalysis `bson:"analyses" json:"analyses"`
}

Sample represents a sequenced sample with associated analyses.

type SampleAnalysis added in v0.3.0

type SampleAnalysis struct {
	Pipeline `bson:"pipeline" json:"pipeline"`
	Results  []SampleAnalysisResult `bson:"path" json:"path"`
}

SampleAnalysis represents a collection of analysis results from an analysis pipeline.

type SampleAnalysisResult added in v0.3.0

type SampleAnalysisResult struct {
	Type        string   `bson:"type" json:"type"`
	Description string   `bson:"description" json:"description"`
	Path        []string `bson:"path" json:"path"`
}

SampleAnalysisResult is a specific result from an analysis pipeline.

type SampleFilter added in v0.3.0

type SampleFilter struct {
	Name             string `form:"sample_name"`
	Id               string `form:"sample_id"`
	RunId            string `form:"run_id"`
	Analysis         string `form:"analysis"`
	PaginationFilter `form:",inline"`
}

Sample filtering.

func NewSampleFilter added in v0.4.0

func NewSampleFilter() SampleFilter

func (SampleFilter) UrlParams added in v0.3.0

func (f SampleFilter) UrlParams() string

Convert a sample filter to URL query parameters.

type SampleResult added in v0.3.0

type SampleResult struct {
	PaginationMetadata `bson:"metadata" json:"metadata"`
	Samples            []Sample `bson:"samples" json:"samples"`
}

type SampleSheet

type SampleSheet struct {
	RunID    *string           `bson:"run_id" json:"run_id"`
	UUID     *uuid.UUID        `bson:"uuid" json:"uuid"`
	Files    []SampleSheetInfo `bson:"files" json:"files"`
	Sections []Section         `bson:"sections" json:"sections"`
}

func ParseSampleSheet

func ParseSampleSheet(r *bufio.Reader) (SampleSheet, error)

func ReadSampleSheet

func ReadSampleSheet(filename string) (SampleSheet, error)

func (SampleSheet) IsValid

func (s SampleSheet) IsValid() bool

func (SampleSheet) LastModified added in v0.3.0

func (s SampleSheet) LastModified() (time.Time, error)

func (SampleSheet) Merge added in v0.3.0

func (s SampleSheet) Merge(other *SampleSheet) (*SampleSheet, error)

Merge two sample sheets. Merging is only allowed if the UUIDs of the sample sheets are the same, and the run IDs are the same. An exception to this is if the run ID of the current sample sheet is nil. If the run ID in the current sample sheet is non-nil and different from the other sample sheet, or if the UUIDs are different, an error is returned.

func (SampleSheet) Section

func (s SampleSheet) Section(name string) *Section

func (SampleSheet) Version

func (s SampleSheet) Version() int

type SampleSheetInfo

type SampleSheetInfo struct {
	Path             string    `bson:"path" json:"path"`
	ModificationTime time.Time `bson:"modification_time" json:"modification_time"`
}

type Section

type Section struct {
	Name string      `bson:"name" json:"name"`
	Type SectionType `bson:"type" json:"type"`
	Rows [][]string  `bson:"rows" json:"rows"`
}

func (Section) Get

func (s Section) Get(name string, index ...int) (string, error)

func (Section) GetColumn

func (s Section) GetColumn(name string) ([]string, error)

func (Section) GetFloat

func (s Section) GetFloat(name string, index ...int) (float64, error)

func (Section) GetInt

func (s Section) GetInt(name string, index ...int) (int, error)

type SectionType

type SectionType int
const (
	UnknownSection SectionType = iota
	SettingsSection
	DataSection
)

func (SectionType) MarshalBSONValue

func (s SectionType) MarshalBSONValue() (bsontype.Type, []byte, error)

func (SectionType) MarshalJSON

func (s SectionType) MarshalJSON() ([]byte, error)

func (SectionType) String

func (s SectionType) String() string

func (*SectionType) UnmarshalBSONValue

func (s *SectionType) UnmarshalBSONValue(t bsontype.Type, b []byte) error

type Time added in v0.5.0

type Time struct {
	time.Time
}

type TimedRunState

type TimedRunState struct {
	State RunState  `bson:"state" json:"state"`
	Time  time.Time `bson:"time" json:"time"`
}

type UpdateResult

type UpdateResult = mongo.UpdateResult

type Version added in v0.2.0

type Version struct {
	Major int
	Minor int
	Patch int
	// contains filtered or unexported fields
}

func NewMinorVersion added in v0.5.0

func NewMinorVersion(major int, minor int) Version

func NewPatchVersion added in v0.5.0

func NewPatchVersion(major int, minor int, patch int) Version

func ParseVersion added in v0.5.0

func ParseVersion(vs string) (Version, error)

func (Version) Equal added in v0.5.0

func (v Version) Equal(other Version) bool

func (Version) HasPatch added in v0.5.0

func (v Version) HasPatch() bool

func (Version) IsZero added in v0.5.0

func (v Version) IsZero() bool

func (Version) MarshalJSON added in v0.5.0

func (v Version) MarshalJSON() ([]byte, error)

func (Version) NewerThan added in v0.5.0

func (v Version) NewerThan(other Version) bool

NewerThan checks if version `v` is newer than verson `other`. Only returns `true` if `v` is strictly larger than `other`. If the versions are identical `false` is returned. If one of the versions has a patch version number and the other doesn't, false is returned.

func (Version) OlderThan added in v0.5.0

func (v Version) OlderThan(other Version) bool

OlderThan checks if version `v` is older than verson `other`. Only returns `true` if `v` is strictly smaller than `other`. If the versions are identical `false` is returned. If one of the versions has a patch version number and the other doesn't, false is returned.

func (Version) String added in v0.5.0

func (v Version) String() string

Directories

Path Synopsis
cmd
cleve command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL