Skip to content

Commit 4f8f425

Browse files
byrnedjbyrnedj
committed
Background data movement (#20)
Background data movement using periodic workers. Attempts to evict/promote items per given thresholds for each class. These reduce p99 latency since there is a higher chance that an allocation slot is free in the tier we are allocating in. fix race in promotion where releaseBackToAllocator was being called before wakeUpWaiters. reinsert to mm container on failed promotion
1 parent afd1456 commit 4f8f425

24 files changed

+1318
-26
lines changed

‎MultiTierDataMovement.md‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
# Background Data Movement
2+
3+
In order to reduce the number of online evictions and support asynchronous
4+
promotion - we have added two periodic workers to handle eviction and promotion.
5+
6+
The diagram below shows a simplified version of how the background evictor
7+
thread (green) is integrated to the CacheLib architecture.
8+
9+
<p align="center">
10+
<img width="640" height="360" alt="BackgroundEvictor" src="cachelib-background-evictor.png">
11+
</p>
12+
13+
## Background Evictors
14+
15+
The background evictors scan each class to see if there are objects to move the next (lower)
16+
tier using a given strategy. Here we document the parameters for the different
17+
strategies and general parameters.
18+
19+
- `backgroundEvictorIntervalMilSec`: The interval that this thread runs for - by default
20+
the background evictor threads will wake up every 10 ms to scan the AllocationClasses. Also,
21+
the background evictor thread will be woken up everytime there is a failed allocation (from
22+
a request handling thread) and the current percentage of free memory for the
23+
AllocationClass is lower than `lowEvictionAcWatermark`. This may render the interval parameter
24+
not as important when there are many allocations occuring from request handling threads.
25+
26+
- `evictorThreads`: The number of background evictors to run - each thread is a assigned
27+
a set of AllocationClasses to scan and evict objects from. Currently, each thread gets
28+
an equal number of classes to scan - but as object size distribution may be unequal - future
29+
versions will attempt to balance the classes among threads. The range is 1 to number of AllocationClasses.
30+
The default is 1.
31+
32+
- `maxEvictionBatch`: The number of objects to remove in a given eviction call. The
33+
default is 40. Lower range is 10 and the upper range is 1000. Too low and we might not
34+
remove objects at a reasonable rate, too high and it might increase contention with user threads.
35+
36+
- `minEvictionBatch`: Minimum number of items to evict at any time (if there are any
37+
candidates)
38+
39+
- `maxEvictionPromotionHotness`: Maximum candidates to consider for eviction. This is similar to `maxEvictionBatch`
40+
but it specifies how many candidates will be taken into consideration, not the actual number of items to evict.
41+
This option can be used to configure duration of critical section on LRU lock.
42+
43+
44+
### FreeThresholdStrategy (default)
45+
46+
- `lowEvictionAcWatermark`: Triggers background eviction thread to run
47+
when this percentage of the AllocationClass is free.
48+
The default is `2.0`, to avoid wasting capacity we don't set this above `10.0`.
49+
50+
- `highEvictionAcWatermark`: Stop the evictions from an AllocationClass when this
51+
percentage of the AllocationClass is free. The default is `5.0`, to avoid wasting capacity we
52+
don't set this above `10`.
53+
54+
55+
## Background Promoters
56+
57+
The background promoters scan each class to see if there are objects to move to a lower
58+
tier using a given strategy. Here we document the parameters for the different
59+
strategies and general parameters.
60+
61+
- `backgroundPromoterIntervalMilSec`: The interval that this thread runs for - by default
62+
the background promoter threads will wake up every 10 ms to scan the AllocationClasses for
63+
objects to promote.
64+
65+
- `promoterThreads`: The number of background promoters to run - each thread is a assigned
66+
a set of AllocationClasses to scan and promote objects from. Currently, each thread gets
67+
an equal number of classes to scan - but as object size distribution may be unequal - future
68+
versions will attempt to balance the classes among threads. The range is `1` to number of AllocationClasses. The default is `1`.
69+
70+
- `maxProtmotionBatch`: The number of objects to promote in a given promotion call. The
71+
default is 40. Lower range is 10 and the upper range is 1000. Too low and we might not
72+
remove objects at a reasonable rate, too high and it might increase contention with user threads.
73+
74+
- `minPromotionBatch`: Minimum number of items to promote at any time (if there are any
75+
candidates)
76+
77+
- `numDuplicateElements`: This allows us to promote items that have existing handles (read-only) since
78+
we won't need to modify the data when a user is done with the data. Therefore, for a short time
79+
the data could reside in both tiers until it is evicted from its current tier. The default is to
80+
not allow this (0). Setting the value to 100 will enable duplicate elements in tiers.
81+
82+
### Background Promotion Strategy (only one currently)
83+
84+
- `promotionAcWatermark`: Promote items if there is at least this
85+
percent of free AllocationClasses. Promotion thread will attempt to move `maxPromotionBatch` number of objects
86+
to that tier. The objects are chosen from the head of the LRU. The default is `4.0`.
87+
This value should correlate with `lowEvictionAcWatermark`, `highEvictionAcWatermark`, `minAcAllocationWatermark`, `maxAcAllocationWatermark`.
88+
- `maxPromotionBatch`: The number of objects to promote in batch during BG promotion. Analogous to
89+
`maxEvictionBatch`. It's value should be lower to decrease contention on hot items.
90+

‎cachelib/allocator/BackgroundMover-inl.h‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
/*
2+
* Copyright (c) Intel and its affiliates.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
namespace facebook {
18+
namespace cachelib {
19+
20+
template <typename CacheT>
21+
BackgroundMover<CacheT>::BackgroundMover(
22+
Cache& cache,
23+
std::shared_ptr<BackgroundMoverStrategy> strategy,
24+
MoverDir direction)
25+
: cache_(cache), strategy_(strategy), direction_(direction) {
26+
if (direction_ == MoverDir::Evict) {
27+
moverFunc = BackgroundMoverAPIWrapper<CacheT>::traverseAndEvictItems;
28+
29+
} else if (direction_ == MoverDir::Promote) {
30+
moverFunc = BackgroundMoverAPIWrapper<CacheT>::traverseAndPromoteItems;
31+
}
32+
}
33+
34+
template <typename CacheT>
35+
BackgroundMover<CacheT>::~BackgroundMover() {
36+
stop(std::chrono::seconds(0));
37+
}
38+
39+
template <typename CacheT>
40+
void BackgroundMover<CacheT>::work() {
41+
try {
42+
checkAndRun();
43+
} catch (const std::exception& ex) {
44+
XLOGF(ERR, "BackgroundMover interrupted due to exception: {}", ex.what());
45+
}
46+
}
47+
48+
template <typename CacheT>
49+
void BackgroundMover<CacheT>::setAssignedMemory(
50+
std::vector<MemoryDescriptorType>&& assignedMemory) {
51+
XLOG(INFO, "Class assigned to background worker:");
52+
for (auto [tid, pid, cid] : assignedMemory) {
53+
XLOGF(INFO, "Tid: {}, Pid: {}, Cid: {}", tid, pid, cid);
54+
}
55+
56+
mutex.lock_combine([this, &assignedMemory] {
57+
this->assignedMemory_ = std::move(assignedMemory);
58+
});
59+
}
60+
61+
// Look for classes that exceed the target memory capacity
62+
// and return those for eviction
63+
template <typename CacheT>
64+
void BackgroundMover<CacheT>::checkAndRun() {
65+
auto assignedMemory = mutex.lock_combine([this] { return assignedMemory_; });
66+
67+
unsigned int moves = 0;
68+
std::set<ClassId> classes{};
69+
auto batches = strategy_->calculateBatchSizes(cache_, assignedMemory);
70+
71+
for (size_t i = 0; i < batches.size(); i++) {
72+
const auto [tid, pid, cid] = assignedMemory[i];
73+
const auto batch = batches[i];
74+
75+
classes.insert(cid);
76+
const auto& mpStats = cache_.getPoolByTid(pid, tid).getStats();
77+
78+
if (!batch) {
79+
continue;
80+
}
81+
82+
// try moving BATCH items from the class in order to reach free target
83+
auto moved = moverFunc(cache_, tid, pid, cid, batch);
84+
moves += moved;
85+
moves_per_class_[tid][pid][cid] += moved;
86+
totalBytesMoved.add(moved * mpStats.acStats.at(cid).allocSize);
87+
}
88+
89+
numTraversals.inc();
90+
numMovedItems.add(moves);
91+
totalClasses.add(classes.size());
92+
}
93+
94+
template <typename CacheT>
95+
BackgroundMoverStats BackgroundMover<CacheT>::getStats() const noexcept {
96+
BackgroundMoverStats stats;
97+
stats.numMovedItems = numMovedItems.get();
98+
stats.runCount = numTraversals.get();
99+
stats.totalBytesMoved = totalBytesMoved.get();
100+
stats.totalClasses = totalClasses.get();
101+
102+
return stats;
103+
}
104+
105+
template <typename CacheT>
106+
std::map<TierId, std::map<PoolId, std::map<ClassId, uint64_t>>>
107+
BackgroundMover<CacheT>::getClassStats() const noexcept {
108+
return moves_per_class_;
109+
}
110+
111+
} // namespace cachelib
112+
} // namespace facebook

‎cachelib/allocator/BackgroundMover.h‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
/*
2+
* Copyright (c) Intel and its affiliates.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
#pragma once
18+
19+
#include "cachelib/allocator/BackgroundMoverStrategy.h"
20+
#include "cachelib/allocator/CacheStats.h"
21+
#include "cachelib/common/AtomicCounter.h"
22+
#include "cachelib/common/PeriodicWorker.h"
23+
24+
namespace facebook {
25+
namespace cachelib {
26+
27+
// wrapper that exposes the private APIs of CacheType that are specifically
28+
// needed for the cache api
29+
template <typename C>
30+
struct BackgroundMoverAPIWrapper {
31+
static size_t traverseAndEvictItems(C& cache,
32+
unsigned int tid,
33+
unsigned int pid,
34+
unsigned int cid,
35+
size_t batch) {
36+
return cache.traverseAndEvictItems(tid, pid, cid, batch);
37+
}
38+
39+
static size_t traverseAndPromoteItems(C& cache,
40+
unsigned int tid,
41+
unsigned int pid,
42+
unsigned int cid,
43+
size_t batch) {
44+
return cache.traverseAndPromoteItems(tid, pid, cid, batch);
45+
}
46+
};
47+
48+
enum class MoverDir { Evict = 0, Promote };
49+
50+
// Periodic worker that evicts items from tiers in batches
51+
// The primary aim is to reduce insertion times for new items in the
52+
// cache
53+
template <typename CacheT>
54+
class BackgroundMover : public PeriodicWorker {
55+
public:
56+
using Cache = CacheT;
57+
// @param cache the cache interface
58+
// @param strategy the stragey class that defines how objects are
59+
// moved,
60+
// (promoted vs. evicted and how much)
61+
BackgroundMover(Cache& cache,
62+
std::shared_ptr<BackgroundMoverStrategy> strategy,
63+
MoverDir direction_);
64+
65+
~BackgroundMover() override;
66+
67+
BackgroundMoverStats getStats() const noexcept;
68+
std::map<TierId, std::map<PoolId, std::map<ClassId, uint64_t>>>
69+
getClassStats() const noexcept;
70+
71+
void setAssignedMemory(
72+
std::vector<MemoryDescriptorType>&& assignedMemory);
73+
74+
private:
75+
std::map<TierId, std::map<PoolId, std::map<ClassId, uint64_t>>>
76+
moves_per_class_;
77+
// cache allocator's interface for evicting
78+
using Item = typename Cache::Item;
79+
80+
Cache& cache_;
81+
std::shared_ptr<BackgroundMoverStrategy> strategy_;
82+
MoverDir direction_;
83+
84+
std::function<size_t(
85+
Cache&, unsigned int, unsigned int, unsigned int, size_t)>
86+
moverFunc;
87+
88+
// implements the actual logic of running the background evictor
89+
void work() override final;
90+
void checkAndRun();
91+
92+
AtomicCounter numMovedItems{0};
93+
AtomicCounter numTraversals{0};
94+
AtomicCounter totalClasses{0};
95+
AtomicCounter totalBytesMoved{0};
96+
97+
std::vector<MemoryDescriptorType> assignedMemory_;
98+
folly::DistributedMutex mutex;
99+
};
100+
} // namespace cachelib
101+
} // namespace facebook
102+
103+
#include "cachelib/allocator/BackgroundMover-inl.h"

‎cachelib/allocator/BackgroundMoverStrategy.h‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
/*
2+
* Copyright (c) Facebook, Inc. and its affiliates.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* http://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
#pragma once
18+
19+
#include "cachelib/allocator/Cache.h"
20+
21+
22+
namespace facebook {
23+
namespace cachelib {
24+
25+
struct MemoryDescriptorType {
26+
MemoryDescriptorType(TierId tid, PoolId pid, ClassId cid) :
27+
tid_(tid), pid_(pid), cid_(cid) {}
28+
TierId tid_;
29+
PoolId pid_;
30+
ClassId cid_;
31+
};
32+
33+
// Base class for background eviction strategy.
34+
class BackgroundMoverStrategy {
35+
public:
36+
virtual std::vector<size_t> calculateBatchSizes(
37+
const CacheBase& cache,
38+
std::vector<MemoryDescriptorType> acVec) = 0;
39+
};
40+
41+
} // namespace cachelib
42+
} // namespace facebook

‎cachelib/allocator/CMakeLists.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ add_library (cachelib_allocator
3535
CCacheManager.cpp
3636
ContainerTypes.cpp
3737
FreeMemStrategy.cpp
38+
FreeThresholdStrategy.cpp
3839
HitsPerSlabStrategy.cpp
3940
LruTailAgeStrategy.cpp
4041
MarginalHitsOptimizeStrategy.cpp

‎cachelib/allocator/Cache.h‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,12 @@ class CacheBase {
9898
//
9999
// @param poolId The pool id to query
100100
virtual const MemoryPool& getPool(PoolId poolId) const = 0;
101+
102+
// Get the reference to a memory pool using a tier id, for stats purposes
103+
//
104+
// @param poolId The pool id to query
105+
// @param tierId The tier of the pool id
106+
virtual const MemoryPool& getPoolByTid(PoolId poolId, TierId tid) const = 0;
101107

102108
// Get Pool specific stats (regular pools). This includes stats from the
103109
// Memory Pool and also the cache.

0 commit comments

Comments
 (0)