NCBI C++ ToolKit
|
Search Toolkit Book for bm::aggregator::pipeline
Pipeline vector for running a group of aggregation operations on a family of vectors. More...
#include <util/bitset/bmaggregator.h>
Public Types | |
typedef Opt | options_type |
Public Member Functions | |
pipeline () | |
run_options & | options () noexcept |
Set pipeline run options. More... | |
const run_options & | get_options () const noexcept |
Get pipeline run options. More... | |
Aggregator traits and control constants | |
~pipeline () noexcept | |
pipeline argument groups fill-in methods | |
arg_groups * | add () |
Add new arguments group. More... | |
void | set_or_target (bvector_type *bv_or) noexcept |
Attach OR (aggregator vector). More... | |
void | set_search_count_limit (size_type limit) noexcept |
Set search limit for results. More... | |
void | complete () |
Prepare pipeline for the execution (resize and init internal structures) Once complete, you cannot add() to it. More... | |
bool | is_complete () const noexcept |
return true if pipeline is ready for execution (complete) More... | |
size_type | size () const noexcept |
Return size() of pileine. More... | |
const arg_vector_type & | get_args_vector () const noexcept |
Return argument vector used for pipeline execution. More... | |
bvect_vector_type & | get_bv_res_vector () noexcept |
Return results vector used for pipeline execution. More... | |
bv_count_vector_type & | get_bv_count_vector () noexcept |
Return results vector count used for pipeline execution. More... | |
access to internals | |
class | aggregator |
run_options | options_ |
execution parameters More... | |
bool | is_complete_ = false |
ready to run state flag More... | |
size_t | total_vect_ = 0 |
total number of vector mentions in all groups More... | |
arg_vector_type | arg_vect_ |
input arg. groups More... | |
bvect_vector_type | bv_res_vect_ |
results (bit-vector ptrs) More... | |
bv_count_vector_type | count_res_vect_ |
results (counts) More... | |
size_type | search_count_limit_ {bm::id_max} |
search limit by count More... | |
pipeline_bcache | bcache_ |
blocks cache structure More... | |
unsigned | top_blocks_ = 1 |
top-level structure size, max of all bvectors More... | |
bvector_type * | bv_or_target_ = 0 |
OR target bit-bector ptr. More... | |
const bv_vector_type & | get_all_input_vect () const noexcept |
const count_vector_type & | get_all_input_cnt_vect () const noexcept |
size_t | unique_vectors () const noexcept |
Return number of unique vectors in the pipeline (after complete()) More... | |
size_t | compute_run_batch () const noexcept |
Function looks at the pipeline to apply euristics to suggest optimal run_batch parameter. More... | |
pipeline_bcache & | get_bcache () noexcept |
unsigned | get_top_blocks () const noexcept |
Return number of top blocks after complete. More... | |
pipeline (const pipeline &)=delete | |
pipeline & | operator= (const pipeline &)=delete |
void | complete_arg_group (arg_groups *ag) |
void | complete_arg_sub_group (index_vector_type &idx_vect, const bvector_type_const_ptr *bv_src, size_t size) |
Pipeline vector for running a group of aggregation operations on a family of vectors.
Pipeline is used to run multiple aggregation combinations (searches) for essencially same set of vectors (different combinations of ANDs and SUBs for example). Pipeline execution improves CPU cache reuse and caches the compressed blocks to re-use it for more efficient execution. Essencially it is a tool to run thousads of runs at once faster.
Definition at line 223 of file bmaggregator.h.
typedef Opt bm::aggregator< BV >::pipeline< Opt >::options_type |
Definition at line 226 of file bmaggregator.h.
|
inline |
Definition at line 228 of file bmaggregator.h.
|
noexcept |
Definition at line 2761 of file bmaggregator.h.
References bm::aligned_free(), arr, and i.
|
protecteddelete |
aggregator< BV >::arg_groups * bm::aggregator< BV >::pipeline< Opt >::add |
Add new arguments group.
Definition at line 2888 of file bmaggregator.h.
References BM_ASSERT.
Referenced by AggregatorTest(), StressTestAggregatorAND(), and StressTestAggregatorAND_SUB().
Prepare pipeline for the execution (resize and init internal structures) Once complete, you cannot add() to it.
Definition at line 2783 of file bmaggregator.h.
References bm::aggregator< BV >::arg_groups::arg_bv0, bm::aggregator< BV >::arg_groups::arg_bv1, BM_ASSERT, bm::heap_vector< Val, BVAlloc, trivial_type >::data(), i, and bm::heap_vector< Val, BVAlloc, trivial_type >::size().
Referenced by AggregatorTest(), StressTestAggregatorAND(), and StressTestAggregatorAND_SUB().
|
private |
Definition at line 2836 of file bmaggregator.h.
References bm::aggregator< BV >::arg_groups::arg_bv0, bm::aggregator< BV >::arg_groups::arg_bv1, bm::aggregator< BV >::arg_groups::arg_idx0, bm::aggregator< BV >::arg_groups::arg_idx1, BM_ASSERT, bm::heap_vector< Val, BVAlloc, trivial_type >::data(), and bm::heap_vector< Val, BVAlloc, trivial_type >::size().
|
private |
Definition at line 2850 of file bmaggregator.h.
References BM_ASSERT, bm::find_ptr(), bm::heap_vector< Val, BVAlloc, trivial_type >::push_back(), ncbi::grid::netcache::search::fields::size, and bm::heap_vector< Val, BVAlloc, trivial_type >::size().
|
noexcept |
Function looks at the pipeline to apply euristics to suggest optimal run_batch parameter.
Definition at line 2899 of file bmaggregator.h.
References bm::set_block_size.
|
inlinenoexcept |
Definition at line 297 of file bmaggregator.h.
Referenced by AggregatorTest().
|
inlinenoexcept |
Definition at line 295 of file bmaggregator.h.
Referenced by AggregatorTest().
|
inlinenoexcept |
Return argument vector used for pipeline execution.
Definition at line 279 of file bmaggregator.h.
Referenced by AggregatorTest().
|
inlineprotectednoexcept |
Definition at line 310 of file bmaggregator.h.
|
inlinenoexcept |
Return results vector count used for pipeline execution.
Definition at line 287 of file bmaggregator.h.
Referenced by AggregatorTest().
|
inlinenoexcept |
Return results vector used for pipeline execution.
Definition at line 283 of file bmaggregator.h.
Referenced by AggregatorTest(), StressTestAggregatorAND(), and StressTestAggregatorAND_SUB().
|
inlinenoexcept |
Get pipeline run options.
Definition at line 235 of file bmaggregator.h.
|
inlineprotectednoexcept |
Return number of top blocks after complete.
Definition at line 315 of file bmaggregator.h.
|
inlinenoexcept |
return true if pipeline is ready for execution (complete)
Definition at line 269 of file bmaggregator.h.
Referenced by AggregatorTest().
|
protecteddelete |
|
inlinenoexcept |
Set pipeline run options.
Definition at line 232 of file bmaggregator.h.
|
inlinenoexcept |
Attach OR (aggregator vector).
Pipeline results all will be OR-ed (UNION) into the OR target vector
bv_or | - OR target vector |
Definition at line 251 of file bmaggregator.h.
Referenced by StressTestAggregatorAND(), and StressTestAggregatorAND_SUB().
|
inlinenoexcept |
Set search limit for results.
Requires that bit-counting to be enabled in the template parameters. Warning: search limit is approximate (for performance reasons) so it can occasinally find more than requested. It cannot find less.
limit | - search limit (target population count to search for) |
Definition at line 260 of file bmaggregator.h.
|
inlinenoexcept |
Return size() of pileine.
Definition at line 272 of file bmaggregator.h.
|
inlinenoexcept |
Return number of unique vectors in the pipeline (after complete())
Definition at line 301 of file bmaggregator.h.
Definition at line 323 of file bmaggregator.h.
|
protected |
input arg. groups
Definition at line 332 of file bmaggregator.h.
Referenced by bm::aggregator< bvector_type >::pipeline::get_args_vector(), and bm::aggregator< bvector_type >::pipeline::size().
|
protected |
blocks cache structure
Definition at line 338 of file bmaggregator.h.
Referenced by bm::aggregator< bvector_type >::pipeline::get_all_input_cnt_vect(), bm::aggregator< bvector_type >::pipeline::get_all_input_vect(), bm::aggregator< bvector_type >::pipeline::get_bcache(), and bm::aggregator< bvector_type >::pipeline::unique_vectors().
|
protected |
OR target bit-bector ptr.
Definition at line 340 of file bmaggregator.h.
Referenced by bm::aggregator< bvector_type >::pipeline::set_or_target().
|
protected |
results (bit-vector ptrs)
Definition at line 334 of file bmaggregator.h.
Referenced by bm::aggregator< bvector_type >::pipeline::get_bv_res_vector().
|
protected |
results (counts)
Definition at line 335 of file bmaggregator.h.
Referenced by bm::aggregator< bvector_type >::pipeline::get_bv_count_vector().
|
protected |
ready to run state flag
Definition at line 330 of file bmaggregator.h.
Referenced by bm::aggregator< bvector_type >::pipeline::is_complete().
|
protected |
execution parameters
Definition at line 329 of file bmaggregator.h.
Referenced by bm::aggregator< bvector_type >::pipeline::get_options(), and bm::aggregator< bvector_type >::pipeline::options().
|
protected |
search limit by count
Definition at line 336 of file bmaggregator.h.
Referenced by bm::aggregator< bvector_type >::pipeline::set_search_count_limit().
|
protected |
top-level structure size, max of all bvectors
Definition at line 339 of file bmaggregator.h.
Referenced by bm::aggregator< bvector_type >::pipeline::get_top_blocks().
|
protected |
total number of vector mentions in all groups
Definition at line 331 of file bmaggregator.h.