NCBI C++ ToolKit
Classes | Public Types | Public Member Functions | Static Public Member Functions | Private Types | Private Member Functions | Private Attributes | List of all members
CSeqDBAtlas Class Reference

Search Toolkit Book for CSeqDBAtlas

CSeqDBAtlas class. More...

#include <objtools/blast/seqdb_reader/impl/seqdbatlas.hpp>

+ Collaboration diagram for CSeqDBAtlas:

Classes

class  CAtlasMappedFile
 

Public Types

enum  { e_MaxFileDescritors = 950 }
 
enum  EFilesCount { eFileCounterNoChange , eFileCounterIncrement , eFileCounterDecrement }
 
typedef CNcbiStreamoff TIndx
 The type used for file offsets. More...
 

Public Member Functions

 CSeqDBAtlas (bool use_atlas_lock)
 Constructor. More...
 
 ~CSeqDBAtlas ()
 The destructor unmaps and frees all associated memory. More...
 
bool DoesFileExist (const string &fname)
 Check if file exists. More...
 
bool DoesFileExist (const CSeqDB_Path &fname)
 Check if file exists. More...
 
bool GetFileSize (const string &fname, TIndx &length)
 Get size of a file. More...
 
bool GetFileSizeL (const string &fname, TIndx &length)
 Get size of a file. More...
 
void RegisterExternal (CSeqDBMemReg &memreg, size_t bytes, CSeqDBLockHold &locked)
 Register externally allocated memory. More...
 
void UnregisterExternal (CSeqDBMemReg &memreg)
 Unregister externally allocated memory. More...
 
void Lock (CSeqDBLockHold &locked)
 Lock the atlas. More...
 
void Unlock (CSeqDBLockHold &locked)
 Unlock the atlas. More...
 
Uint8 GetSliceSize ()
 Get the current slice size. More...
 
const string GetSearchPath () const
 Get BlastDB search path. More...
 
CMemoryFileGetMemoryFile (const string &fileName)
 
CMemoryFileReturnMemoryFile (const string &fileName)
 
int ChangeOpenedFilseCount (EFilesCount fc)
 
int GetOpenedFilseCount (void)
 

Static Public Member Functions

static void RetRegion (const char *datap)
 Free allocated memory. More...
 
static char * Alloc (size_t length, bool clear=true)
 Allocate memory that atlas will keep track of. More...
 
static const string GenerateSearchPath ()
 Generate search path. More...
 

Private Types

enum  { e_MaxSlice64 = 1 << 30 }
 

Private Member Functions

 CSeqDBAtlas (const CSeqDBAtlas &)
 Private method to prevent copy construction. More...
 

Private Attributes

CMutex m_Lock
 Protects most of the critical regions of the SeqDB library. More...
 
bool m_UseLock
 Use single atlas lock to protect critical regions. More...
 
std::mutex m_FileSizeMutex
 Cache of file existence and length. More...
 
map< string, pair< bool, TIndx > > m_FileSize
 
Uint8 m_MaxFileSize
 Maxium file size. More...
 
std::mutex m_FileMemMapMutex
 
map< string, unique_ptr< CAtlasMappedFile > > m_FileMemMap
 
int m_OpenedFilesCount
 
int m_MaxOpenedFilesCount
 
const string m_SearchPath
 BlastDB search path. More...
 

Detailed Description

CSeqDBAtlas class.

This object manages a collection of (memory) maps. It mmaps or reads data from files on demand, to allow a set of files to be accessed efficiently by SeqDB. The total size of the files used by a multivolume database may exceed the usable address space of the system by several times. SeqDB also registers certain large, dynamically allocated (via new[]) memory blocks with this object, in an effort to limit the total memory usage. This class also contains the primary mutex used to sequentialize access to the various SeqDB critical regions, some of which are outside of this class. ["Atlas: n. 1. A book or bound collection of maps..."; The American Heritage Dictionary of the English Language, 4th Edition. Copyright (c) 2000 by Houghton Mifflin Company.]

Definition at line 298 of file seqdbatlas.hpp.

Member Typedef Documentation

◆ TIndx

The type used for file offsets.

Definition at line 302 of file seqdbatlas.hpp.

Member Enumeration Documentation

◆ anonymous enum

anonymous enum
Enumerator
e_MaxFileDescritors 

Definition at line 318 of file seqdbatlas.hpp.

◆ anonymous enum

anonymous enum
private
Enumerator
e_MaxSlice64 

Definition at line 598 of file seqdbatlas.hpp.

◆ EFilesCount

Enumerator
eFileCounterNoChange 
eFileCounterIncrement 
eFileCounterDecrement 

Definition at line 540 of file seqdbatlas.hpp.

Constructor & Destructor Documentation

◆ CSeqDBAtlas() [1/2]

CSeqDBAtlas::CSeqDBAtlas ( bool  use_atlas_lock)

Constructor.

Initializes the atlas object.

Parameters
use_atlas_lockIf true, the atlas lock will be used to protect critical regions, otherwise the Lock() and Unlock() functions will be noops. Setting the parameter to false improves CPU utilization when each thread access a different database volume. It should be set to true in other cases.

Definition at line 101 of file seqdbatlas.cpp.

References m_MaxOpenedFilesCount, and m_OpenedFilesCount.

◆ ~CSeqDBAtlas()

CSeqDBAtlas::~CSeqDBAtlas ( )

The destructor unmaps and frees all associated memory.

Definition at line 110 of file seqdbatlas.cpp.

◆ CSeqDBAtlas() [2/2]

CSeqDBAtlas::CSeqDBAtlas ( const CSeqDBAtlas )
private

Private method to prevent copy construction.

Member Function Documentation

◆ Alloc()

char * CSeqDBAtlas::Alloc ( size_t  length,
bool  clear = true 
)
static

Allocate memory that atlas will keep track of.

This method allocates memory for the calling code's use. There are three reasons to do this. First, the allocated memory is guaranteed to be deleted when the atlas destructor runs, so using this method ties the lifetime of the allocated memory to that of the atlas, which may prevent memory leaks. Secondly, allocating memory in this way brings the allocated memory under the total memory bound the atlas imposes on itself. The amount of memory assumed to be available for slice allocation will be reduced by the size of these allocations during garbage collection. Thirdly, the memory allocated this way can be freed by the RetRegion(char*) method, so the RetSequence code in the volume layers (and thereabouts) does not need to can return allocated memory to the user as "sequence data", and does not have to track whether the data was allocated or mapped.

Parameters
lengthAmount of memory to allocate in bytes.
lockedThe lock hold object for this thread.
clearSpecify true to zero out memory contents.
Returns
A pointer to the allocation region of memory.

Definition at line 232 of file seqdbatlas.cpp.

References NCBI_THROW.

Referenced by CSeqDBVol::x_AllocType().

◆ ChangeOpenedFilseCount()

int CSeqDBAtlas::ChangeOpenedFilseCount ( EFilesCount  fc)
inline

Definition at line 548 of file seqdbatlas.hpp.

References fc, and max().

Referenced by BOOST_AUTO_TEST_CASE(), GetMemoryFile(), and ReturnMemoryFile().

◆ DoesFileExist() [1/2]

bool CSeqDBAtlas::DoesFileExist ( const CSeqDB_Path fname)
inline

Check if file exists.

This is like the previous but accepts CSeqDB_Path.

Parameters
fnameThe filename of the file to get.
lockedThe lock hold object for this thread.
Returns
True if the file exists.

Definition at line 350 of file seqdbatlas.hpp.

References CSeqDB_Path::GetPathS().

◆ DoesFileExist() [2/2]

bool CSeqDBAtlas::DoesFileExist ( const string fname)

Check if file exists.

If the file exists, this method will return true, otherwise false. Also, if the file exists, the first slice of it will be mapped. This method exists for efficiency reasons; it is cheaper to ask the atlas whether it already knows about a file than to search the filesystem for the file, particularly in the case of network file systems. If a slice of the file has already been mapped, this will return true without consulting the file system, so this method will not detect files that have been deleted since the mapping occurred.

Parameters
fnameThe filename of the file to get.
lockedThe lock hold object for this thread.
Returns
True if the file exists.

Definition at line 148 of file seqdbatlas.cpp.

References GetFileSize().

Referenced by CSeqDBColumn::ColumnExists(), CSeqDB_AtlasAccessor::DoesFileExist(), CSeqDBAliasSets::ReadAliasFile(), and CSeqDBAliasNode::x_ExpandAliases().

◆ GenerateSearchPath()

static const string CSeqDBAtlas::GenerateSearchPath ( )
inlinestatic

◆ GetFileSize()

bool CSeqDBAtlas::GetFileSize ( const string fname,
TIndx length 
)

Get size of a file.

Check whether a file exists and get the file's size.

Parameters
fnameThe filename of the file to get the size of.
lengthThe length of the file is returned here.
lockedThe lock hold object for this thread.
Returns
true if the file exists.

Definition at line 154 of file seqdbatlas.cpp.

References GetFileSizeL().

Referenced by DoesFileExist().

◆ GetFileSizeL()

bool CSeqDBAtlas::GetFileSizeL ( const string fname,
TIndx length 
)

Get size of a file.

Check whether a file exists and get the file's size. This version assumes the atlas lock is held.

Parameters
fnameThe filename of the file to get the size of.
lengthThe length of the file is returned here.
Returns
true if the file exists.

Definition at line 160 of file seqdbatlas.cpp.

References map_checker< Container >::end(), map_checker< Container >::find(), m_FileSize, m_FileSizeMutex, m_MaxFileSize, val, and whole.

Referenced by CSeqDBNodeFileIdList::CSeqDBNodeFileIdList(), GetFileSize(), CSeqDBRawFile::Open(), CSeqDBIsam::x_InitSearch(), CSeqDBAliasNode::x_ReadAliasFile(), and CSeqDBAliasSets::x_ReadAliasSetFile().

◆ GetMemoryFile()

CMemoryFile * CSeqDBAtlas::GetMemoryFile ( const string fileName)

◆ GetOpenedFilseCount()

int CSeqDBAtlas::GetOpenedFilseCount ( void  )
inline

Definition at line 566 of file seqdbatlas.hpp.

Referenced by BOOST_AUTO_TEST_CASE(), CSeqDBFileMemMap::Init(), and ReturnMemoryFile().

◆ GetSearchPath()

const string CSeqDBAtlas::GetSearchPath ( void  ) const
inline

Get BlastDB search path.

Definition at line 505 of file seqdbatlas.hpp.

Referenced by SeqDB_FindBlastDBPath().

◆ GetSliceSize()

Uint8 CSeqDBAtlas::GetSliceSize ( )
inline

Get the current slice size.

This returns the current slice size used for mmap() style memory allocations.

Returns
Atlas will try to map this much data at a time.

Definition at line 497 of file seqdbatlas.hpp.

References min().

Referenced by CSeqDBImpl::x_FillSeqBuffer().

◆ Lock()

void CSeqDBAtlas::Lock ( CSeqDBLockHold locked)
inline

◆ RegisterExternal()

void CSeqDBAtlas::RegisterExternal ( CSeqDBMemReg memreg,
size_t  bytes,
CSeqDBLockHold locked 
)

Register externally allocated memory.

This method tells the atlas code that memory was allocated external to the atlas code, and should be included under the memory bound enforced by the atlas. These areas of memory will not be managed by the atlas, but may influence the atlas by causing database volume files or auxiliary files to be unmapped earlier or more often. This method may trigger atlas garbage collection. RegisterFree() should be called when the memory is freed.

Parameters
memregMemory registration tracking object.
bytesAmount of memory externally allocated.
lockedThe lock hold object for this thread.

Definition at line 264 of file seqdbatlas.cpp.

References _ASSERT, Lock(), and CSeqDBMemReg::m_Bytes.

Referenced by CSeqDBNodeFileIdList::CSeqDBNodeFileIdList().

◆ RetRegion()

void CSeqDBAtlas::RetRegion ( const char *  datap)
static

Free allocated memory.

Releases allocated memory.

This method releases the memory aquired by the Alloc() method. With data known to have originated in Alloc(), it is faster to call the Free() method. This method assumes the lock is held.

Parameters
datapPointer to the data to release or deallocate.

Definition at line 226 of file seqdbatlas.cpp.

Referenced by CSeqDBImpl::RetAmbigSeq().

◆ ReturnMemoryFile()

CMemoryFile * CSeqDBAtlas::ReturnMemoryFile ( const string fileName)

◆ Unlock()

void CSeqDBAtlas::Unlock ( CSeqDBLockHold locked)
inline

Unlock the atlas.

If the lock hold object passed to this method is already in an "unlocked" state, this call is a noop. Otherwise, the lock hold object is put in an unlocked state and the lock is released.

Parameters
lockedThis object tracks whether this thread owns the mutex.

Definition at line 480 of file seqdbatlas.hpp.

References CSeqDBLockHold::m_Locked.

Referenced by CSeqDBColumn::CSeqDBColumn(), CSeqDBVolSet::CSeqDBVolSet(), CSeqDBImpl::GetBioseq(), CSeqDBFileMemMap::Init(), CSeqDBImpl::x_GetCacheID(), CSeqDBImpl::x_GetHdr(), CSeqDBImpl::x_GetOidList(), CSeqDBImpl::x_GetSeqGI(), and CSeqDBLockHold::~CSeqDBLockHold().

◆ UnregisterExternal()

void CSeqDBAtlas::UnregisterExternal ( CSeqDBMemReg memreg)

Unregister externally allocated memory.

This method tells the atlas that external memory registered with RegisterExternal() has been freed. The atlas lock is assumed to be held.

Parameters
memregMemory registration tracking object.

Definition at line 275 of file seqdbatlas.cpp.

References CSeqDBMemReg::m_Bytes.

Referenced by CSeqDBMemReg::~CSeqDBMemReg().

Member Data Documentation

◆ m_FileMemMap

map<string, unique_ptr<CAtlasMappedFile> > CSeqDBAtlas::m_FileMemMap
private

Definition at line 607 of file seqdbatlas.hpp.

Referenced by GetMemoryFile(), and ReturnMemoryFile().

◆ m_FileMemMapMutex

std::mutex CSeqDBAtlas::m_FileMemMapMutex
private

Definition at line 606 of file seqdbatlas.hpp.

Referenced by GetMemoryFile(), and ReturnMemoryFile().

◆ m_FileSize

map< string, pair<bool, TIndx> > CSeqDBAtlas::m_FileSize
private

Definition at line 602 of file seqdbatlas.hpp.

Referenced by GetFileSizeL().

◆ m_FileSizeMutex

std::mutex CSeqDBAtlas::m_FileSizeMutex
private

Cache of file existence and length.

Definition at line 601 of file seqdbatlas.hpp.

Referenced by GetFileSizeL().

◆ m_Lock

CMutex CSeqDBAtlas::m_Lock
private

Protects most of the critical regions of the SeqDB library.

Definition at line 592 of file seqdbatlas.hpp.

◆ m_MaxFileSize

Uint8 CSeqDBAtlas::m_MaxFileSize
private

Maxium file size.

Definition at line 604 of file seqdbatlas.hpp.

Referenced by GetFileSizeL().

◆ m_MaxOpenedFilesCount

int CSeqDBAtlas::m_MaxOpenedFilesCount
private

Definition at line 609 of file seqdbatlas.hpp.

Referenced by CSeqDBAtlas().

◆ m_OpenedFilesCount

int CSeqDBAtlas::m_OpenedFilesCount
private

Definition at line 608 of file seqdbatlas.hpp.

Referenced by CSeqDBAtlas().

◆ m_SearchPath

const string CSeqDBAtlas::m_SearchPath
private

BlastDB search path.

Definition at line 612 of file seqdbatlas.hpp.

◆ m_UseLock

bool CSeqDBAtlas::m_UseLock
private

Use single atlas lock to protect critical regions.

The single lock is not needed if each thread access different database volume.

Definition at line 596 of file seqdbatlas.hpp.


The documentation for this class was generated from the following files:
Modified on Wed Feb 21 09:54:55 2024 by modify_doxy.py rev. 669887