Search Toolkit Book for CDataCrawlerCache

Base class for data crawler cache objects. More...

#include <gui/widgets/loaders/data_crawler_cache.hpp>

Inheritance diagram for CDataCrawlerCache:

Collaboration diagram for CDataCrawlerCache:

Public Types
enum	ECacheState { eState_None , eState_InProgress , eState_NotAccessible , eState_Created }
	Cache object states. More...

Public Member Functions
	CDataCrawlerCache ()=default
	Default constructor. More...

ECacheState	GetState () const
	Gets the state of the cache object. More...

ECacheState	SetState (ECacheState desired_state)
	Changes the state of the cache object. More...

size_t	GetReadSize () const
	Returns size of existing BLOB data. More...

std::unique_ptr< CNcbiIstream >	GetReadStream ()
	Returns stream interface to read BLOB data. More...

std::unique_ptr< CNcbiOstream >	GetWriteStream ()
	Returns stream interface to write BLOB data. More...

void	Remove ()
	Removes the cache entry. More...

const std::string &	GetKey () const
	Returns the NetCache key of the BLOB. More...

const std::string	GetCompoundKey () const
	Returns the compound NetCache key of the BLOB. More...

virtual	~CDataCrawlerCache ()
	Destructor. More...

Protected Types
enum	ECacheType { eCache_Blob , eCache_Sync }
	Types of cache objects. More...

Protected Member Functions
void	x_Initialize ()
	Initializes the cache instances. More...

std::string	x_GetCacheKey (ECacheType type)
	Generates the cache key for a given type. More...

virtual CNetICacheClient	x_InitBlobCache ()=0
	Creates the BLOB NetCache instance. More...

virtual CNetICacheClient	x_InitSyncCache ()=0
	Creates the sync NetCache instance. More...

virtual std::string	x_GetCacheKey ()=0
	Generates the cache key. More...

void	x_StartHeartbeat ()
	Starts the heartbeat (thread to keep the sybc object alive) More...

void	x_StopHeartbeat ()
	Stops the heartbeat (thread to keep the sybc object alive) More...

void	x_Heartbeat ()
	The heartbeat thread function. More...

CNetICacheClient	x_InitICache (const std::string &cache, const std::string &service, const std::string &db)
	Creates a NetCache instance. More...

ECacheState	x_ReadState () const
	Reads the state of the sync object. More...

void	x_WriteState (ECacheState state, unsigned int ttl=300)
	Writes a new state to the sync object. More...

Protected Attributes
bool	m_isImmutableKey {false}
	do not try to shorten or otherwise modify the root cache key More...

Private Attributes
std::string	m_BlobKey
	The NetCache key of the BLOB object. More...

std::string	m_SyncKey
	The NetCache key of the sync object. More...

CNetICacheClient	m_BlobCache
	Blobs NetCache instance. More...

CNetICacheClient	m_SyncCache
	Sync objects NetCache instance. More...

std::mutex	m_SyncCacheMutex
	Mutex, controlling the access to the sync cache. More...

std::atomic_bool	m_StopHeartbeat { false }
	Boolean flag to stop the heartbeat thread. More...

std::unique_ptr< std::thread >	m_Heartbeat
	The heartbeat thread. More...

Detailed Description

Base class for data crawler cache objects.

Definition at line 47 of file data_crawler_cache.hpp.

Member Enumeration Documentation

◆ ECacheState

enum CDataCrawlerCache::ECacheState

Cache object states.

Enumerator
eState_None
eState_InProgress	No cache data.
eState_NotAccessible	The remote file is being processed.
eState_Created	The remote file is not accessible. The remote file was processed and its data is in the cache

Definition at line 51 of file data_crawler_cache.hpp.

◆ ECacheType

enum CDataCrawlerCache::ECacheType

protected

Types of cache objects.

Enumerator

eCache_Blob

eCache_Sync

BLOB object.

Synchronization object

Definition at line 101 of file data_crawler_cache.hpp.

Constructor & Destructor Documentation

◆ CDataCrawlerCache()

CDataCrawlerCache::CDataCrawlerCache ( )

default

Default constructor.

◆ ~CDataCrawlerCache()

CDataCrawlerCache::~CDataCrawlerCache ( )

virtual

Destructor.

Definition at line 237 of file data_crawler_cache.cpp.

References x_StopHeartbeat().

Member Function Documentation

◆ GetCompoundKey()

const string CDataCrawlerCache::GetCompoundKey ( ) const

Returns the compound NetCache key of the BLOB.

Returns: String, holding the compound NetCache key.

Definition at line 231 of file data_crawler_cache.cpp.

References CNetStorageObjectLoc::Create(), CNetICacheClient::GetCacheName(), CNetICacheClient::GetService(), CNetService::GetServiceName(), m_BlobCache, m_BlobKey, and version.

◆ GetKey()

const std::string& CDataCrawlerCache::GetKey ( void ) const

inline

Returns the NetCache key of the BLOB.

Returns: String, holding the NetCache key.

Definition at line 90 of file data_crawler_cache.hpp.

References m_BlobKey.

◆ GetReadSize()

size_t CDataCrawlerCache::GetReadSize ( ) const

Returns size of existing BLOB data.

Returns: size in bytes

Definition at line 107 of file data_crawler_cache.cpp.

References CNetICacheClient::GetSize(), m_BlobCache, and m_BlobKey.

◆ GetReadStream()

unique_ptr< CNcbiIstream > CDataCrawlerCache::GetReadStream ( )

Returns stream interface to read BLOB data.

Returns: Smart pointer to read stream.

Definition at line 112 of file data_crawler_cache.cpp.

References CRWStreambuf::fOwnReader, CNetICacheClient::GetReadStream(), m_BlobCache, and m_BlobKey.

◆ GetState()

CDataCrawlerCache::ECacheState CDataCrawlerCache::GetState ( void ) const

Gets the state of the cache object.

Returns: The current state

Definition at line 48 of file data_crawler_cache.cpp.

References eState_Created, eState_None, CNetICacheClient::HasBlob(), Info(), LOG_POST, m_BlobCache, m_BlobKey, m_SyncCache, m_SyncKey, and x_ReadState().

Referenced by SetState().

◆ GetWriteStream()

unique_ptr< CNcbiOstream > CDataCrawlerCache::GetWriteStream ( )

Returns stream interface to write BLOB data.

Returns: Smart pointer to write stream.

Definition at line 118 of file data_crawler_cache.cpp.

References CRWStreambuf::fOwnWriter, CNetICacheClient::GetWriteStream(), m_BlobCache, and m_BlobKey.

Referenced by ColumnarVcfCache::CreateBlob().

◆ Remove()

void CDataCrawlerCache::Remove ( void )

Removes the cache entry.

Definition at line 124 of file data_crawler_cache.cpp.

References m_BlobCache, m_BlobKey, and CNetICacheClient::Remove().

◆ SetState()

CDataCrawlerCache::ECacheState CDataCrawlerCache::SetState ( ECacheState desired_state )

Changes the state of the cache object.

Parameters

[in] desired_state The desired state to switch to.

Returns: The actual state of the cache object.

Definition at line 63 of file data_crawler_cache.cpp.

References eState_Created, eState_InProgress, eState_None, eState_NotAccessible, GetState(), m_SyncCache, m_SyncCacheMutex, m_SyncKey, CNetICacheClient::Remove(), x_StartHeartbeat(), x_StopHeartbeat(), and x_WriteState().

Referenced by ColumnarVcfCache::CreateBlob().

◆ x_GetCacheKey() [1/2]

virtual std::string CDataCrawlerCache::x_GetCacheKey ( )

protectedpure virtual

Generates the cache key.

Returns: String, holding the cache key.

Implemented in CColumnarVcfCache.

Referenced by x_GetCacheKey(), and x_Initialize().

◆ x_GetCacheKey() [2/2]

string CDataCrawlerCache::x_GetCacheKey ( ECacheType type )

protected

Generates the cache key for a given type.

Parameters

[in] type The desired cache type.

Returns: String, holding the cache key.

Definition at line 138 of file data_crawler_cache.cpp.

References eCache_Blob, eCache_Sync, ncbi::grid::netcache::search::fields::key, m_isImmutableKey, md5(), and x_GetCacheKey().

◆ x_Heartbeat()

void CDataCrawlerCache::x_Heartbeat ( )

protected

The heartbeat thread function.

Definition at line 194 of file data_crawler_cache.cpp.

References Error(), eState_InProgress, LOG_POST, m_StopHeartbeat, and x_WriteState().

Referenced by x_StartHeartbeat().

◆ x_InitBlobCache()

virtual CNetICacheClient CDataCrawlerCache::x_InitBlobCache ( )

protectedpure virtual

Creates the BLOB NetCache instance.

Returns: The NetCache instance.

Implemented in CColumnarVcfCache.

Referenced by x_Initialize().

◆ x_Initialize()

void CDataCrawlerCache::x_Initialize ( )

protected

Initializes the cache instances.

Note: Must be invoked by the constructors of the derived classes.

Definition at line 129 of file data_crawler_cache.cpp.

References eCache_Blob, eCache_Sync, ICache::fBestReliability, m_BlobCache, m_BlobKey, m_SyncCache, m_SyncKey, CNetICacheClient::SetFlags(), x_GetCacheKey(), x_InitBlobCache(), and x_InitSyncCache().

Referenced by CColumnarVcfCache::CColumnarVcfCache().

◆ x_InitICache()

CNetICacheClient CDataCrawlerCache::x_InitICache	(	const std::string &	cache,
		const std::string &	service,
		const std::string &	db
	)

protected

Creates a NetCache instance.

Parameters

[in]	cache	Name of the registry section to look for the configuration parameters in.
[in]	service	The service name.
[in]	db	The cache name.

Returns: The NetCache instance.

Definition at line 207 of file data_crawler_cache.cpp.

References CNcbiApplicationAPI::GetConfig(), IRegistry::GetString(), Info(), CNcbiApplication::Instance(), and LOG_POST.

Referenced by CColumnarVcfCache::x_InitBlobCache(), and CColumnarVcfCache::x_InitSyncCache().

◆ x_InitSyncCache()

virtual CNetICacheClient CDataCrawlerCache::x_InitSyncCache ( )

protectedpure virtual

Creates the sync NetCache instance.

Returns: The NetCache instance.

Implemented in CColumnarVcfCache.

Referenced by x_Initialize().

◆ x_ReadState()

CDataCrawlerCache::ECacheState CDataCrawlerCache::x_ReadState ( ) const

inlineprotected

Reads the state of the sync object.

Returns: The cache state, read from the sync object.

Definition at line 216 of file data_crawler_cache.cpp.

References eState_None, m_SyncCache, m_SyncCacheMutex, m_SyncKey, CNetICacheClient::Read(), and result.

Referenced by GetState().

◆ x_StartHeartbeat()

void CDataCrawlerCache::x_StartHeartbeat ( )

protected

Starts the heartbeat (thread to keep the sybc object alive)

Definition at line 166 of file data_crawler_cache.cpp.

References Error(), eUnknown, Info(), LOG_POST, m_Heartbeat, m_StopHeartbeat, NCBI_THROW, and x_Heartbeat().

Referenced by SetState().

◆ x_StopHeartbeat()

void CDataCrawlerCache::x_StopHeartbeat ( )

protected

Stops the heartbeat (thread to keep the sybc object alive)

Definition at line 179 of file data_crawler_cache.cpp.

References Error(), Info(), LOG_POST, m_Heartbeat, and m_StopHeartbeat.

Referenced by SetState(), and ~CDataCrawlerCache().

◆ x_WriteState()

void CDataCrawlerCache::x_WriteState	(	ECacheState	state,
		unsigned int	ttl = `300`
	)

inlineprotected

Writes a new state to the sync object.

Parameters

[in]	state	The state to write.
[in]	ttl	Time to live in seconds. 5 minutes by default.

Definition at line 224 of file data_crawler_cache.cpp.

References m_SyncCache, m_SyncCacheMutex, m_SyncKey, and CNetICacheClient::Store().

Referenced by SetState(), and x_Heartbeat().

Member Data Documentation

◆ m_BlobCache

CNetICacheClient CDataCrawlerCache::m_BlobCache

mutableprivate

Blobs NetCache instance.

Definition at line 168 of file data_crawler_cache.hpp.

Referenced by GetCompoundKey(), GetReadSize(), GetReadStream(), GetState(), GetWriteStream(), Remove(), and x_Initialize().

◆ m_BlobKey

std::string CDataCrawlerCache::m_BlobKey

private

The NetCache key of the BLOB object.

Definition at line 164 of file data_crawler_cache.hpp.

Referenced by GetCompoundKey(), GetKey(), GetReadSize(), GetReadStream(), GetState(), GetWriteStream(), Remove(), and x_Initialize().

◆ m_Heartbeat

std::unique_ptr<std::thread> CDataCrawlerCache::m_Heartbeat

private

The heartbeat thread.

Definition at line 176 of file data_crawler_cache.hpp.

Referenced by x_StartHeartbeat(), and x_StopHeartbeat().

◆ m_isImmutableKey

bool CDataCrawlerCache::m_isImmutableKey {false}

protected

do not try to shorten or otherwise modify the root cache key

Definition at line 180 of file data_crawler_cache.hpp.

Referenced by CColumnarVcfCache::CColumnarVcfCache(), and x_GetCacheKey().

◆ m_StopHeartbeat

std::atomic_bool CDataCrawlerCache::m_StopHeartbeat { false }

private

Boolean flag to stop the heartbeat thread.

Definition at line 174 of file data_crawler_cache.hpp.

Referenced by x_Heartbeat(), x_StartHeartbeat(), and x_StopHeartbeat().

◆ m_SyncCache

CNetICacheClient CDataCrawlerCache::m_SyncCache

mutableprivate

Sync objects NetCache instance.

Definition at line 170 of file data_crawler_cache.hpp.

Referenced by GetState(), SetState(), x_Initialize(), x_ReadState(), and x_WriteState().

◆ m_SyncCacheMutex

std::mutex CDataCrawlerCache::m_SyncCacheMutex

mutableprivate

Mutex, controlling the access to the sync cache.

Definition at line 172 of file data_crawler_cache.hpp.

Referenced by SetState(), x_ReadState(), and x_WriteState().

◆ m_SyncKey

std::string CDataCrawlerCache::m_SyncKey

private

The NetCache key of the sync object.

Definition at line 166 of file data_crawler_cache.hpp.

Referenced by GetState(), SetState(), x_Initialize(), x_ReadState(), and x_WriteState().

The documentation for this class was generated from the following files:

include/gui/widgets/loaders/data_crawler_cache.hpp
src/gui/widgets/loaders/data_crawler_cache.cpp

Public Types

Public Member Functions

Protected Types

Protected Member Functions

Protected Attributes

Private Attributes

Detailed Description

Member Enumeration Documentation

◆ ECacheState

◆ ECacheType

Constructor & Destructor Documentation

◆ CDataCrawlerCache()

◆ ~CDataCrawlerCache()

Member Function Documentation

◆ GetCompoundKey()

◆ GetKey()

◆ GetReadSize()

◆ GetReadStream()

◆ GetState()

◆ GetWriteStream()

◆ Remove()

◆ SetState()

◆ x_GetCacheKey() [1/2]

◆ x_GetCacheKey() [2/2]

◆ x_Heartbeat()

◆ x_InitBlobCache()

◆ x_Initialize()

◆ x_InitICache()

◆ x_InitSyncCache()

◆ x_ReadState()

◆ x_StartHeartbeat()

◆ x_StopHeartbeat()

◆ x_WriteState()

Member Data Documentation

◆ m_BlobCache

◆ m_BlobKey

◆ m_Heartbeat

◆ m_isImmutableKey

◆ m_StopHeartbeat

◆ m_SyncCache

◆ m_SyncCacheMutex

◆ m_SyncKey