common

package
v1.0.1-0...-23bcde9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 10, 2025 License: Apache-2.0 Imports: 17 Imported by: 0

Documentation

Overview

Package common creates an outline for common functionality across the multiple source databases we support. While adding new methods or code here

  1. Ensure that the changes do not adversely impact any source that uses the common code
  2. Test cases might not sufficiently cover all cases, so integration and manual testing should be done ensure no functionality is breaking. Most of the test cases that cover the code in this package will lie in the implementing source databases, so it might not be required to have unit tests for each method here.
  3. Any functions added here should be used by two or more databases
  4. If it looks like the code is getting more complex due to refactoring, it is probably better off leaving the functionality out of common

Index

Constants

View Source 
const DefaultWorkers = 20 // Default to 20 - observed diminishing returns above this value

Variables

View Source 
var DATATYPE_TO_STORAGE_SIZE = map[string]int{
	ddl.Bool:      1,
	ddl.Date:      4,
	ddl.Float32:   4,
	ddl.Float64:   8,
	ddl.Int64:     8,
	ddl.JSON:      ddl.StringMaxLength,
	ddl.Numeric:   22,
	ddl.Timestamp: 12,
}

Data type sizes are referred from https://cloud.google.com/spanner/docs/reference/standard-sql/data-types#storage_size_for_data_types

View Source 
var ErrorTypeMapping = map[string]internal.SchemaIssue{
	"No matching signature for operator": internal.TypeMismatchError,
	"Syntax error":                       internal.InvalidConditionError,
	"Unrecognized name":                  internal.ColumnNotFoundError,
	"Function not found":                 internal.CheckConstraintFunctionNotFoundError,
	"unhandled error":                    internal.GenericError,
}

Functions

func ComputeNonKeyColumnSize 

func ComputeNonKeyColumnSize(conv *internal.Conv, tableId string)

func ConvertSchemaToSpannerDDL 

func ConvertSchemaToSpannerDDL(conv *internal.Conv, dbDump DbDump, schemaToSpanner SchemaToSpannerInterface) error

func CvtForeignKeysHelper 

func CvtForeignKeysHelper(conv *internal.Conv, spTableName string, srcTableId string, srcKey schema.ForeignKey, isRestore bool) (ddl.Foreignkey, error)

func CvtIndexHelper 

func CvtIndexHelper(conv *internal.Conv, tableId string, srcIndex schema.Index, spColIds []string, spColDef map[string]ddl.ColumnDef) ddl.CreateIndex

func GenerateExpressionDetailList 

func GenerateExpressionDetailList(spschema ddl.Schema) []internal.ExpressionDetail

GenerateExpressionDetailList it will generate the expression detail list which is used in verify expression method as a input

func GetColsAndSchemas 

func GetColsAndSchemas(conv *internal.Conv, tableId string) (schema.Table, string, []string, ddl.CreateTable, error)

GetColsAndSchemas provides information about columns and schema for a table.

func GetCommonColumnIds 

func GetCommonColumnIds(conv *internal.Conv, tableId string, colIds []string) []string

func GetErroredIssue 

func GetErroredIssue(result internal.VerifyExpressionsOutput) map[string][]internal.InvalidCheckExp

GetErroredIssue it will collect all the error and return it

func GetIssue 

func GetIssue(result internal.VerifyExpressionsOutput) (map[string][]internal.InvalidCheckExp, map[string][]string)

GetIssue it will collect all the error and return it

func GetSortedTableIdsBySpName 

func GetSortedTableIdsBySpName(spSchema ddl.Schema) []string

func GetSortedTableIdsBySrcName 

func GetSortedTableIdsBySrcName(srcSchema map[string]schema.Table) []string

func IntersectionOfTwoStringSlices 

func IntersectionOfTwoStringSlices(a []string, b []string) []string

func IsPrimaryKey 

func IsPrimaryKey(colId string, table schema.Table) bool

func IsSchemaIssuePresent 

func IsSchemaIssuePresent(schemaissue []internal.SchemaIssue, issue internal.SchemaIssue) bool

IsSchemaIssuePresent checks if issue is present in the given schemaissue list.

func PrepareColumns 

func PrepareColumns(conv *internal.Conv, tableId string, srcCols []string) ([]string, error)

func PrepareValues 

func PrepareValues[T interface{}](conv *internal.Conv, tableId string, colNameIdMap map[string]string, commonColIds, srcCols []string, values []T) ([]T, error)

func ProcessDbDump 

func ProcessDbDump(conv *internal.Conv, r *internal.Reader, dbDump DbDump, ddlVerifier expressions_api.DDLVerifier, exprVerifier expressions_api.ExpressionVerificationAccessor) error

ProcessDbDump reads dump data from r and does schema or data conversion, depending on whether conv is configured for schema mode or data mode. In schema mode, this method incrementally builds a schema (updating conv). In data mode, this method uses this schema to convert data and writes it to Spanner, using the data sink specified in conv.

func RemoveCheckConstraint 

func RemoveCheckConstraint(checkConstraints []ddl.CheckConstraint, expId string) []ddl.CheckConstraint

RemoveCheckConstraint this method will remove the constraint which has error

func RemoveError 

func RemoveError(tableIssues map[string]internal.TableIssues) map[string]internal.TableIssues

RemoveError it will reset the table issue before re-populating

func SanitizeDefaultValue 

func SanitizeDefaultValue(defaultValue string, ty string, generated bool) string

SanitizeDefaultValue removes extra characters added to Default Value in information schema in MySQL.

func SrcTableToSpannerDDL 

func SrcTableToSpannerDDL(conv *internal.Conv, toddl ToDdl, srcTable schema.Table, ddlVerifier expressions_api.DDLVerifier) error

func ToNotNull 

func ToNotNull(conv *internal.Conv, isNullable string) bool

ToNotNull returns true if a column is not nullable and false if it is.

func ToPGDialectType 

func ToPGDialectType(standardType ddl.Type, isPk bool) (ddl.Type, []internal.SchemaIssue)

Types

type DbDump 

type DbDump interface {
	GetToDdl() ToDdl
	ProcessDump(conv *internal.Conv, r *internal.Reader) error
}

DbDump common interface for database dump functions.

type FkConstraint 

type FkConstraint struct {
	Name     string
	Table    string
	Refcols  []string
	Cols     []string
	OnDelete string
	OnUpdate string
}

FkConstraint contains foreign key constraints

type InfoSchema 

type InfoSchema interface {
	GetToDdl() ToDdl
	GetTableName(schema string, tableName string) string
	GetTables() ([]SchemaAndName, error)
	GetColumns(conv *internal.Conv, table SchemaAndName, constraints map[string][]string, primaryKeys []string) (map[string]schema.Column, []string, error)
	GetRowsFromTable(conv *internal.Conv, srcTable string) (interface{}, error)
	GetRowCount(table SchemaAndName) (int64, error)
	GetConstraints(conv *internal.Conv, table SchemaAndName) ([]string, []schema.CheckConstraint, map[string][]string, error)
	GetForeignKeys(conv *internal.Conv, table SchemaAndName) (foreignKeys []schema.ForeignKey, err error)
	GetIndexes(conv *internal.Conv, table SchemaAndName, colNameIdMp map[string]string) ([]schema.Index, error)
	ProcessData(conv *internal.Conv, tableId string, srcSchema schema.Table, spCols []string, spSchema ddl.CreateTable, additionalAttributes internal.AdditionalDataAttributes) error
	StartChangeDataCapture(ctx context.Context, conv *internal.Conv) (map[string]interface{}, error)
	StartStreamingMigration(ctx context.Context, migrationProjectId string, client *sp.Client, conv *internal.Conv, streamInfo map[string]interface{}) (internal.DataflowOutput, error)
}

InfoSchema contains database information.

type InfoSchemaImpl 

type InfoSchemaImpl struct{}

func (*InfoSchemaImpl) GenerateSrcSchema 

func (is *InfoSchemaImpl) GenerateSrcSchema(conv *internal.Conv, infoSchema InfoSchema, numWorkers int) (int, error)

func (*InfoSchemaImpl) GetIncludedSrcTablesFromConv 

func (is *InfoSchemaImpl) GetIncludedSrcTablesFromConv(conv *internal.Conv) (schemaToTablesMap map[string]internal.SchemaDetails, err error)

getIncludedSrcTablesFromConv fetches the list of tables from the source database that need to be migrated.

func (*InfoSchemaImpl) ProcessData 

func (is *InfoSchemaImpl) ProcessData(conv *internal.Conv, infoSchema InfoSchema, additionalAttributes internal.AdditionalDataAttributes)

ProcessData performs data conversion for source database 'db'. For each table, we extract and convert the data to Spanner data (based on the source and Spanner schemas), and write it to Spanner. If we can't get/process data for a table, we skip that table and process the remaining tables.

func (*InfoSchemaImpl) ProcessTable 

func (is *InfoSchemaImpl) ProcessTable(conv *internal.Conv, table SchemaAndName, infoSchema InfoSchema) (schema.Table, error)

func (*InfoSchemaImpl) SetRowStats 

func (is *InfoSchemaImpl) SetRowStats(conv *internal.Conv, infoSchema InfoSchema)

SetRowStats populates conv with the number of rows in each table.

type InfoSchemaInterface 

type InfoSchemaInterface interface {
	GenerateSrcSchema(conv *internal.Conv, infoSchema InfoSchema, numWorkers int) (int, error)
	ProcessData(conv *internal.Conv, infoSchema InfoSchema, additionalAttributes internal.AdditionalDataAttributes)
	SetRowStats(conv *internal.Conv, infoSchema InfoSchema)
	ProcessTable(conv *internal.Conv, table SchemaAndName, infoSchema InfoSchema) (schema.Table, error)
	GetIncludedSrcTablesFromConv(conv *internal.Conv) (schemaToTablesMap map[string]internal.SchemaDetails, err error)
}

type MockDbDump 

type MockDbDump struct {
	mock.Mock
}

MockDbDump is a mock implementation of the DbDump interface.

func (*MockDbDump) GetToDdl 

func (m *MockDbDump) GetToDdl() ToDdl

GetToDdl provides a mock implementation for GetToDdl.

func (*MockDbDump) ProcessDump 

func (m *MockDbDump) ProcessDump(conv *internal.Conv, r *internal.Reader) error

ProcessDump provides a mock implementation for ProcessDump.

type MockInfoSchema 

type MockInfoSchema struct {
	mock.Mock
}

func (*MockInfoSchema) GenerateSrcSchema 

func (mis *MockInfoSchema) GenerateSrcSchema(conv *internal.Conv, infoSchema InfoSchema, numWorkers int) (int, error)

func (*MockInfoSchema) GetIncludedSrcTablesFromConv 

func (mis *MockInfoSchema) GetIncludedSrcTablesFromConv(conv *internal.Conv) (schemaToTablesMap map[string]internal.SchemaDetails, err error)

func (*MockInfoSchema) ProcessData 

func (mis *MockInfoSchema) ProcessData(conv *internal.Conv, infoSchema InfoSchema, additionalAttributes internal.AdditionalDataAttributes)

func (*MockInfoSchema) SetRowStats 

func (mis *MockInfoSchema) SetRowStats(conv *internal.Conv, infoSchema InfoSchema)

type MockOptionProvider 

type MockOptionProvider struct {
	mock.Mock
}

MockOptionProvider is a mock implementation of OptionProvider

func (*MockOptionProvider) GetColumnAutoGen 

func (m *MockOptionProvider) GetColumnAutoGen(conv *internal.Conv, autoGenCol ddl.AutoGenCol, colId string, tableId string) (*ddl.AutoGenCol, error)

func (*MockOptionProvider) GetTypeOption 

func (m *MockOptionProvider) GetTypeOption(srcTypeName string, spType ddl.Type) string

GetTypeOption is a method of the OptionProvider interface

func (*MockOptionProvider) ToSpannerType 

func (m *MockOptionProvider) ToSpannerType(conv *internal.Conv, spType string, srcType schema.Type, isPk bool) (ddl.Type, []internal.SchemaIssue)

ToSpannerType and GetCOlumnAutoGen are methods of the ToDdl interface

type MockProcessSchema 

type MockProcessSchema struct {
	mock.Mock
}

func (*MockProcessSchema) ProcessSchema 

func (mps *MockProcessSchema) ProcessSchema(conv *internal.Conv, infoSchema InfoSchema, numWorkers int, attributes internal.AdditionalSchemaAttributes, s SchemaToSpannerInterface, uo UtilsOrderInterface, is InfoSchemaInterface) error

type MockRunParallelTasks 

type MockRunParallelTasks[I any, O any] struct {
	mock.Mock
}

func (*MockRunParallelTasks[I, O]) RunParallelTasks 

func (mrpt *MockRunParallelTasks[I, O]) RunParallelTasks(input []I, numWorkers int, f func(i I, mutex *sync.Mutex) task.TaskResult[O],
	fastExit bool) ([]task.TaskResult[O], error)

type MockSchemaToSpanner 

type MockSchemaToSpanner struct {
	mock.Mock
}

func (*MockSchemaToSpanner) SchemaToSpannerDDL 

func (mss *MockSchemaToSpanner) SchemaToSpannerDDL(conv *internal.Conv, toddl ToDdl, attributes internal.AdditionalSchemaAttributes) error

func (*MockSchemaToSpanner) SchemaToSpannerDDLHelper 

func (mss *MockSchemaToSpanner) SchemaToSpannerDDLHelper(conv *internal.Conv, toddl ToDdl, srcTable schema.Table, isRestore bool) error

func (*MockSchemaToSpanner) SchemaToSpannerSequenceHelper 

func (mss *MockSchemaToSpanner) SchemaToSpannerSequenceHelper(conv *internal.Conv, srcSequence ddl.Sequence) error

type MockToDdl 

type MockToDdl struct {
	mock.Mock
}

MockToDdl is a mock implementation of the ToDdl interface.

func (*MockToDdl) GetColumnAutoGen 

func (m *MockToDdl) GetColumnAutoGen(conv *internal.Conv, autoGenCol ddl.AutoGenCol, colId string, tableId string) (*ddl.AutoGenCol, error)

GetColumnAutoGen provides a mock implementation for GetColumnAutoGen.

func (*MockToDdl) ToSpannerType 

func (m *MockToDdl) ToSpannerType(conv *internal.Conv, spType string, srcType schema.Type, isPk bool) (ddl.Type, []internal.SchemaIssue)

ToSpannerType provides a mock implementation for ToSpannerType.

type MockUtilsOrder 

type MockUtilsOrder struct {
	mock.Mock
}

type OptionProvider 

type OptionProvider interface {
	GetTypeOption(srcTypeName string, spType ddl.Type) string
}

CassandraOptionProvider is an interface that can be implemented by ToDdl implementations for sources that provide specific type options, like Cassandra.

type ProcessSchemaImpl 

type ProcessSchemaImpl struct{}

func (*ProcessSchemaImpl) ProcessSchema 

func (ps *ProcessSchemaImpl) ProcessSchema(conv *internal.Conv, infoSchema InfoSchema, numWorkers int, attributes internal.AdditionalSchemaAttributes, s SchemaToSpannerInterface, uo UtilsOrderInterface, is InfoSchemaInterface) error

ProcessSchema performs schema conversion for source database 'db'. Information schema tables are a broadly supported ANSI standard, and we use them to obtain source database's schema information.

type ProcessSchemaInterface 

type ProcessSchemaInterface interface {
	ProcessSchema(conv *internal.Conv, infoSchema InfoSchema, numWorkers int, attributes internal.AdditionalSchemaAttributes, s SchemaToSpannerInterface, uo UtilsOrderInterface, is InfoSchemaInterface) error
}

type SchemaAndName 

type SchemaAndName struct {
	Schema string
	Name   string
	Id     string
}

SchemaAndName contains the schema and name for a table

type SchemaToSpannerImpl 

type SchemaToSpannerImpl struct {
	ExpressionVerificationAccessor expressions_api.ExpressionVerificationAccessor
	DdlV                           expressions_api.DDLVerifier
}

func (*SchemaToSpannerImpl) SchemaToSpannerDDL 

func (ss *SchemaToSpannerImpl) SchemaToSpannerDDL(conv *internal.Conv, toddl ToDdl, attributes internal.AdditionalSchemaAttributes) error

SchemaToSpannerDDL performs schema conversion from the source DB schema to Spanner. It uses the source schema in conv.SrcSchema, and writes the Spanner schema to conv.SpSchema.

func (*SchemaToSpannerImpl) SchemaToSpannerDDLHelper 

func (ss *SchemaToSpannerImpl) SchemaToSpannerDDLHelper(conv *internal.Conv, toddl ToDdl, srcTable schema.Table, isRestore bool) error

func (*SchemaToSpannerImpl) SchemaToSpannerSequenceHelper 

func (ss *SchemaToSpannerImpl) SchemaToSpannerSequenceHelper(conv *internal.Conv, srcSequence ddl.Sequence) error

func (*SchemaToSpannerImpl) VerifyExpressions 

func (ss *SchemaToSpannerImpl) VerifyExpressions(conv *internal.Conv) error

VerifyExpression this function will use expression_api to validate check constraint expressions and add the relevant error to suggestion tab and remove the check constraint which has error

type SchemaToSpannerInterface 

type SchemaToSpannerInterface interface {
	SchemaToSpannerDDL(conv *internal.Conv, toddl ToDdl, attributes internal.AdditionalSchemaAttributes) error
	SchemaToSpannerDDLHelper(conv *internal.Conv, toddl ToDdl, srcTable schema.Table, isRestore bool) error
	SchemaToSpannerSequenceHelper(conv *internal.Conv, srcSequence ddl.Sequence) error
}

type ToDdl 

type ToDdl interface {
	ToSpannerType(conv *internal.Conv, spType string, srcType schema.Type, isPk bool) (ddl.Type, []internal.SchemaIssue)
	GetColumnAutoGen(conv *internal.Conv, autoGenCol ddl.AutoGenCol, colId string, tableId string) (*ddl.AutoGenCol, error)
}

ToDdl interface is meant to be implemented by all sources. When support for a new target database is added, please add a new method here with the output type expected. In case a particular source to target transoformation is not supported, an error is to be returned by the corresponding method.

type UtilsOrderImpl 

type UtilsOrderImpl struct{}

type UtilsOrderInterface 

type UtilsOrderInterface interface {
	// contains filtered or unexported methods
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.