Search Toolkit Book for CBlobMetaDB
Dictionary file, storing references on external BLOB file (super BLOB structure). More...
#include <db/bdb/bdb_ext_blob.hpp>
Inheritance diagram for CBlobMetaDB:
Collaboration diagram for CBlobMetaDB:
Public Types | |
| typedef CBDB_BLobFile | TParent |
| Public Types inherited from CBDB_File | |
| enum | EAfterWrite { eKeepData , eDiscardData } |
| typedef int | TUnifiedFieldIndex |
| CBDB_File keeps data in two buffers (key buffer and data buffer). More... | |
| Public Types inherited from CBDB_RawFile | |
| enum | EOpenMode { eReadWrite , eReadOnly , eCreate , eReadWriteCreate } |
| BDB file open mode. More... | |
| enum | EDBType { eBtree , eQueue , eHash } |
| Berkeley DB database type. More... | |
| enum | EReallocMode { eReallocAllowed , eReallocForbidden } |
| BLOB read mode, controld data buffer reallocation when there is not enough space in buffer. More... | |
| enum | EDuplicateKeys { eDuplicatesDisable , eDuplicatesEnable } |
| Control key duplicates in Btree. More... | |
| enum | EIgnoreError { eIgnoreError , eThrowOnError } |
| enum | ECompact { eCompactNoFree , eCompactFreeExisting , eCompactFreeAll } |
| BerkeleyDB compaction methods and flags. More... | |
| enum | ECachePriority { eCache_Lowest , eCache_Low , eCache_Default , eCache_High , eCache_Highest } |
| Set the priority for this database's pages in the buffer cache This is generally a temporary advisement, and works only if an environment is used. More... | |
| typedef CSimpleBuffer | TBuffer |
| typedef for raw buffer operations More... | |
| typedef bool(* | FContinueCompact) (void) |
| Extended version of compact This version performs iterative compacting and uses a callback to request an exit. More... | |
Public Member Functions | |
| CBlobMetaDB () | |
| EBDB_ErrCode | FetchMeta (Uint4 blob_id, CBDB_BlobMetaContainer *meta_container, Uint4 *id_from=0, Uint4 *id_to=0) |
| Find the meta container storing our target blob_id Function is doing the cursor range scan sequentially reading range-matching BLOB descriptions. More... | |
| EBDB_ErrCode | UpdateInsert (const CBDB_BlobMetaContainer &meta_container) |
| Insert new super BLOB metainfo. More... | |
| Public Member Functions inherited from CBDB_BLobFile | |
| CBDB_BLobFile (EDuplicateKeys dup_keys=eDuplicatesDisable, EDBType db_type=eBtree) | |
| EBDB_ErrCode | Insert (const void *data, size_t size) |
| Insert BLOB into the database. More... | |
| EBDB_ErrCode | Insert (const TBuffer &buf) |
| Insert BLOB. More... | |
| unsigned | Append (const void *data, size_t size) |
| EBDB_ErrCode | UpdateInsert (const void *data, size_t size) |
| Insert or update BLOB. More... | |
| EBDB_ErrCode | UpdateInsert (const TBuffer &buf) |
| Insert or update BLOB. More... | |
| EBDB_ErrCode | Fetch () |
| Fetch the record corresponding to the current key value. More... | |
| EBDB_ErrCode | Fetch (void **buf, size_t buf_size, EReallocMode allow_realloc) |
| Retrieve BLOB data. More... | |
| size_t | LobSize () const |
| Get LOB size. Becomes available right after successfull Fetch. More... | |
| size_t | BlobSize () const |
| Get LOB size. Becomes available right after successfull Fetch. More... | |
| EBDB_ErrCode | ReadRealloc (TBuffer &buffer) |
| Read BLOB into vector. More... | |
| EBDB_ErrCode | GetData (void *buf, size_t size) |
| Copy LOB data into the 'buf'. More... | |
| CBDB_BLobStream * | CreateStream () |
| Creates stream like object to retrieve or write BLOB by chunks. More... | |
| CBDB_BlobReaderWriter * | CreateReaderWriter () |
| Creates stream like object to retrieve or write BLOB by chunks. More... | |
| IReader * | CreateReader () |
| Creates stream like object to read BLOB by chunks. More... | |
| Public Member Functions inherited from CBDB_File | |
| CBDB_File (EDuplicateKeys dup_keys=eDuplicatesDisable, EDBType db_type=eBtree) | |
| void | Open (const string &filename, EOpenMode open_mode, bool support_dirty_read=false, unsigned rec_len=0) |
| Open file with specified access mode. More... | |
| void | Open (const string &filename, const string &database, EOpenMode open_mode, bool support_dirty_read=false, unsigned rec_len=0) |
| Open file with specified filename and database name. More... | |
| void | Reopen (EOpenMode open_mode, bool support_dirty_read=false) |
| Reopen the db file. More... | |
| void | Attach (CBDB_File &db_file) |
| Attach external Berkeley DB file. More... | |
| EBDB_ErrCode | Fetch () |
| Fetches the record corresponding to the current key value. More... | |
| EBDB_ErrCode | FetchForUpdate () |
| Fetche the record corresponding to the current key value. More... | |
| EBDB_ErrCode | Insert (EAfterWrite write_flag=eDiscardData) |
| Insert new record. More... | |
| unsigned | Append (EAfterWrite write_flag=eDiscardData) |
| Append record to the queue (works only for DB_QUEUE database type) More... | |
| EBDB_ErrCode | Delete (EIgnoreError on_error=eThrowOnError) |
| Delete record corresponding to the current key value. More... | |
| EBDB_ErrCode | UpdateInsert (EAfterWrite write_flag=eDiscardData) |
| Update record corresponding to the current key value. More... | |
| void | BindKey (const char *field_name, CBDB_Field *key_field, size_t buf_size=0) |
| void | BindData (const char *field_name, CBDB_Field *data_field, size_t buf_size=0, ENullable is_null=eNullable) |
| void | DuplicateStructure (const CBDB_File &dbf) |
| Create the same fieldset as in dbf and bind them to the current file. More... | |
| const CBDB_BufferManager * | GetKeyBuffer () const |
| Get Buffer manager for key section of the file. More... | |
| const CBDB_BufferManager * | GetDataBuffer () const |
| Get Buffer manager for data section of the file. More... | |
| CBDB_BufferManager * | GetKeyBuffer () |
| Get Buffer manager for key section of the file. More... | |
| CBDB_BufferManager * | GetDataBuffer () |
| Get Buffer manager for data section of the file. More... | |
| void | SetFieldCompareLimit (unsigned int n_fields) |
| Sets maximum number of key fields participating in comparison Should be less than total number of key fields. More... | |
| DBT * | CloneDBT_Key () |
| Create new copy of m_DBT_Key. More... | |
| void | SetLegacyStringsCheck (bool value) |
| Set C-str detection. More... | |
| TUnifiedFieldIndex | GetFieldIdx (const string &name) const |
| Get field index by name. More... | |
| const CBDB_Field & | GetField (TUnifiedFieldIndex idx) const |
| Return field by field index. More... | |
| CBDB_Field & | GetField (TUnifiedFieldIndex idx) |
| void | SetFieldOwnership (bool own_fields) |
| Fields deletion is managed by the class when own_fields is TRUE. More... | |
| bool | IsOwnFields () const |
| Return fields ownership flag. More... | |
| void | CopyFrom (const CBDB_File &dbf) |
| Copy record (fields) from another BDB file (MUST have the same structure) More... | |
| void | Verify (const char *filename, const char *database, FILE *backup) |
| Run database verification (DB->verify) More... | |
| void | EnablePrefixCompression () |
| Turn ON prefix compression. More... | |
| Public Member Functions inherited from CBDB_RawFile | |
| CBDB_RawFile (EDuplicateKeys dup_keys=eDuplicatesDisable, EDBType db_type=eBtree) | |
| virtual | ~CBDB_RawFile () |
| void | SetEnv (CBDB_Env &env) |
| Associate file with environment. More... | |
| CBDB_Env * | GetEnv () |
| Get pointer on file environment Return NULL if no environment has been set. More... | |
| void | Open (const string &filename, EOpenMode open_mode, bool support_dirty_read=false, unsigned rec_len=0) |
| Open file with specified access mode. More... | |
| void | Open (const string &filename, const string &database, EOpenMode open_mode, bool support_dirty_read=false, unsigned rec_len=0) |
| Open file with specified filename and database name. More... | |
| void | Attach (CBDB_RawFile &bdb_file) |
| Attach class to external BerkeleyDB file instance. More... | |
| void | Close () |
| Close file. More... | |
| void | Reopen (EOpenMode open_mode, bool support_dirty_read=false, unsigned rec_len=0) |
| Reopen database file. (Should be already open). More... | |
| void | Remove (const string &filename, const string &database=kEmptyStr) |
| Remove the database specified by the filename and database arguments. More... | |
| unsigned int | Truncate () |
| Empty the database. Return number of records removed. More... | |
| unsigned int | SafeTruncate () |
| Workaround for truncate of large databases. More... | |
| void | Rename (const string &fname, const string &old_name, const string &new_name) |
| Rename a database. NOTE: This cannot be called on an opened file. More... | |
| void | Compact (ECompact compact_type=eCompactNoFree, int target_fill_pct=0) |
| Compact the database. More... | |
| void | CompactEx (FContinueCompact compact_callback, ECompact compact_type=eCompactNoFree, int target_fill_pct=0) |
| void | SetPageSize (unsigned int page_size) |
| unsigned int | GetPageSize () |
| void | SetCacheSize (unsigned int cache_size) |
| Set Berkeley DB memory cache size for the file (default is 256K). More... | |
| void | RevSplitOff () |
| Turn OFF reverse splitting. More... | |
| void | DisableCmpOverride () |
| Disable BTREE comparison override. More... | |
| void | SetCachePriority (ECachePriority priority) |
| const string & | FileName () const |
| const string & | Database () const |
| virtual void | SetHash (DB *) |
| Set hash function. More... | |
| bool | IsOpen () const |
| Return TRUE if the file is open. More... | |
| bool | IsAttached () const |
| bool | IsByteSwapped () const |
| Return TRUE if the if the underlying database files were created on an architecture of the different byte order. More... | |
| bool | DuplicatesAllowed () const |
| Return TRUE if file can contain duplicate keys. More... | |
| EDuplicateKeys | GetDupKeysMode () const |
| Return the key duplicate mode value. More... | |
| const string & | GetFileName () const |
| Return file name. More... | |
| EOpenMode | GetOpenMode () const |
| Return the file open mode. More... | |
| void | Sync () |
| Flush any cached information to disk. More... | |
| unsigned | CountRecs (bool bFast=false) |
| Compute database statistic, return number of records. More... | |
| void | PrintStat (CNcbiOstream &out) |
| Print database statistics. More... | |
| virtual void | SetTransaction (ITransaction *trans) |
| Establish transaction association. More... | |
| virtual void | RemoveTransaction (ITransaction *trans) |
| Remove transaction association (must be established by SetTransaction. More... | |
| virtual ITransaction * | GetTransaction () |
| Get current transaction. More... | |
| CBDB_Transaction * | GetBDBTransaction () |
| Get current transaction. More... | |
| unsigned | GetRecLen () const |
| Get record length Works for fixed length record DBs only (Queue) More... | |
| void | SetHashFillFactor (unsigned h_ffactor) |
| Set hash table density (fill factor) More... | |
| void | SetHashNelem (unsigned h_nelem) |
| Set an estimate of hash table final size. More... | |
| void | DisableHashOverride () |
| Disable hash method override (Berkeley DB will use it's own default hashing method) More... | |
| void | SetBtreeMinKeysPerPage (unsigned int keys_per_page) |
| Set the minimum number of keys per page (BTREE access methods only) More... | |
| unsigned int | GetBtreeMinKeysPerPage () |
| void | SetCompressor (ICompression *compressor, EOwnership own=eTakeOwnership) |
| Set record compressor. More... | |
| Public Member Functions inherited from ITransactional | |
| virtual | ~ITransactional () |
Public Attributes | |
| CBDB_FieldUint4 | id_from |
| Id range from. More... | |
| CBDB_FieldUint4 | id_to |
| Id range to. More... | |
Private Member Functions | |
| CBlobMetaDB (const CBlobMetaDB &) | |
| CBlobMetaDB & | operator= (const CBlobMetaDB &) |
Additional Inherited Members | |
| Static Public Member Functions inherited from CBDB_File | |
| static void | DestroyDBT_Clone (DBT *dbt) |
| Free the DBT structure created by CloneDBT_Key. More... | |
| Static Public Attributes inherited from CBDB_RawFile | |
| static const char | kDefaultDatabase [] = "_table" |
| Protected Member Functions inherited from CBDB_File | |
| void | Discard () |
| Unpack internal record buffers. More... | |
| virtual void | SetCmp (DB *) |
| Set comparison function. More... | |
| EBDB_ErrCode | ReadCursor (DBC *dbc, unsigned int bdb_flag) |
| Read DB cursor. More... | |
| EBDB_ErrCode | ReadCursor (DBC *dbc, unsigned int bdb_flag, void **buf, size_t buf_size, EReallocMode allow_realloc) |
| Read DB cursor (BLOB) More... | |
| EBDB_ErrCode | ReadCursor (DBC *dbc, unsigned int bdb_flag, TBuffer *buf) |
| Read DB cursor (BLOB) More... | |
| EBDB_ErrCode | ReadCursor (DBC *dbc, unsigned int bdb_flag, CBDB_MultiRowBuffer *multirow_buf, bool multirow_only) |
| Multiple-row read into a buffer Buffer is to be traversed using DB_MULTIPLE_KEY_NEXT (BerkeleyDB) More... | |
| EBDB_ErrCode | WriteCursor (DBC *dbc, unsigned int bdb_flag, EAfterWrite write_flag) |
| Write DB cursor. More... | |
| EBDB_ErrCode | WriteCursor (const void *data, size_t size, DBC *dbc, unsigned int bdb_flag, EAfterWrite write_flag) |
| Write BLOB to DB cursor. More... | |
| EBDB_ErrCode | DeleteCursor (DBC *dbc, EIgnoreError) |
| Delete DB cursor. More... | |
| void | CheckNullDataConstraint () const |
| Check if all NOT NULL fields are assigned. More... | |
| void | DisableDataBufProcessing () |
| Function disables processing of m_DBT_data. More... | |
| void | DisableNull () |
| Disable NULL/not NULL in data fields (Performance tweak, call before BindData) More... | |
| void | DisableDataPacking () |
| Disable packing of variable length fields in the data buffer (Call after BindData) More... | |
| EBDB_ErrCode | x_Fetch (unsigned int flags) |
| Wrapper around get operation. More... | |
| virtual void | x_SetByteSwapped (bool bswp) |
| Set byte order swapping. More... | |
| Protected Member Functions inherited from CBDB_RawFile | |
| void | x_Open (const char *filename, const char *database, EOpenMode open_mode, bool support_dirty_read, unsigned rec_len) |
| void | x_Create (const char *filename, const char *database) |
| void | x_Close (EIgnoreError close_mode) |
| void | x_CreateDB (unsigned rec_len) |
| Create m_DB member, set page, cache parameters. More... | |
| void | x_SetTransaction (CBDB_Transaction *trans) |
| Set current transaction. More... | |
| void | x_RemoveTransaction (CBDB_Transaction *trans) |
| DB_TXN * | GetTxn () |
| Get transaction handler. More... | |
| DBC * | CreateCursor (CBDB_Transaction *trans=0, unsigned int flags=0) const |
| Create DB cursor. More... | |
| int | x_DB_Fetch (DBT *key, DBT *data, unsigned flags) |
| Internal override for DB->get(...) This method overrides destination buffer and uses compressor: Should only be used with DB_DBT_USERMEM flag. More... | |
| int | x_DBC_Fetch (DBC *dbc, DBT *key, DBT *data, unsigned flags) |
| Internal override for DBC->c_get(...) This method overrides destination buffer and uses compressor: Should only be used with DB_DBT_USERMEM flag. More... | |
| int | x_DB_Put (DBT *key, DBT *data, unsigned flags) |
| Override for DB->put(...) Handles compression. More... | |
| int | x_DB_CPut (DBC *dbc, DBT *key, DBT *data, unsigned flags) |
| Override for DBC->c_put(...) Handles compression. More... | |
| int | x_FetchBufferDecompress (DBT *data, void *usr_data) |
| Protected Attributes inherited from CBDB_RawFile | |
| EDBType | m_DB_Type |
| DB * | m_DB |
| DBT * | m_DBT_Key |
| DBT * | m_DBT_Data |
| CBDB_Env * | m_Env |
| CBDB_Transaction * | m_Trans |
| int | m_TransAssociation |
| unsigned | m_RecLen |
| unsigned | m_H_ffactor |
| unsigned | m_H_nelem |
| unsigned | m_BT_minkey |
| AutoPtr< ICompression > | m_Compressor |
| Record compressor. More... | |
| TBuffer | m_CompressBuffer |
Detailed Description
Dictionary file, storing references on external BLOB file (super BLOB structure).
All BLOB meta attributes here are stored in this table and can be transactionally protected and caches. External file in this case works as a raw partition stores only BLOB bytes.
An important implication is that we almost immediately know if target range of ids contains our BLOB without reading the super BLOB (could be slow). Search through overlapping ranges can relatively fast.
File encodes id range -> super BLOB description
The general design idea is that we take certain number of BLOBs (each is has a unique id) BLOBs are packed together into one large super BLOB, compressed and stored in an external file. Super BLOB description is stored in a CExtBlobLocDB.
Getting a BLOB is a multi-step procedure:
- first we find the right id range
- read super BLOB description and deserialize it
- before reading the super BLOB
- reassemble super BLOB from chunks (one chunk in most cases) the most efficient read would probably be mmap
- decompress super-BLOB (optional) (may be coupled with mmap)
- find blob id and location in the meta-information
- read BLOB from super BLOB (reassemble using location table)
Definition at line 264 of file bdb_ext_blob.hpp.
The documentation for this struct was generated from the following files:
- include/db/bdb/bdb_ext_blob.hpp
- src/db/bdb/bdb_ext_blob.cpp
