[Enhancement](memory) Add ConcurrentLong2ObjectHashMap and ConcurrentLong2LongHashMap#61332
Open
dataroaring wants to merge 1 commit intomasterfrom
Open
[Enhancement](memory) Add ConcurrentLong2ObjectHashMap and ConcurrentLong2LongHashMap#61332dataroaring wants to merge 1 commit intomasterfrom
dataroaring wants to merge 1 commit intomasterfrom
Conversation
…Long2LongHashMap Add thread-safe primitive-key concurrent hash maps built on fastutil, designed to replace ConcurrentHashMap<Long, V> and ConcurrentHashMap<Long, Long> in memory-sensitive FE paths. These maps eliminate Long autoboxing overhead and reduce per-entry memory from ~64 bytes (ConcurrentHashMap) to ~16 bytes, a 4x improvement. Key design: - Segment-based locking (default 16 segments) for concurrent throughput - Full Map interface compatibility for drop-in replacement - Atomic putIfAbsent, computeIfAbsent, replace, remove operations - Comprehensive unit tests covering CRUD, concurrency, iteration, edge cases, and default value semantics Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
There was a problem hiding this comment.
Pull request overview
Adds two new segmented-lock concurrent hash maps for FE that use fastutil primitive-key/value maps to reduce memory overhead compared to ConcurrentHashMap<Long, ...> while preserving familiar APIs and providing snapshot-based iteration.
Changes:
- Introduce
ConcurrentLong2ObjectHashMap<V>: concurrent long→object map with per-segment RW locks and atomic compute/merge-style operations. - Introduce
ConcurrentLong2LongHashMap: concurrent long→long map with per-segment RW locks plus an atomicaddTocounter helper. - Add comprehensive JUnit tests for correctness, concurrency behavior, iteration snapshots, and Gson round-trip/format compatibility.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 8 comments.
| File | Description |
|---|---|
| fe/fe-core/src/main/java/org/apache/doris/common/ConcurrentLong2ObjectHashMap.java | New segmented concurrent long→object map implementation. |
| fe/fe-core/src/main/java/org/apache/doris/common/ConcurrentLong2LongHashMap.java | New segmented concurrent long→long map implementation with addTo. |
| fe/fe-core/src/test/java/org/apache/doris/common/ConcurrentLong2ObjectHashMapTest.java | New unit tests for object map behavior, concurrency, and Gson. |
| fe/fe-core/src/test/java/org/apache/doris/common/ConcurrentLong2LongHashMapTest.java | New unit tests for long map behavior, concurrency, and Gson. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+263
to
+269
| if (seg.map.containsKey(key)) { | ||
| return seg.map.get(key); | ||
| } | ||
| long newValue = mappingFunction.applyAsLong(key); | ||
| seg.map.put(key, newValue); | ||
| return newValue; | ||
| } finally { |
Comment on lines
+451
to
+453
| // Boxed get via Map<Long,Long> interface returns null for missing keys | ||
| Long boxedResult = map.getOrDefault(999L, map.defaultReturnValue()); | ||
| Assertions.assertEquals(0L, boxedResult); |
Comment on lines
+50
to
+51
| * <p><b>Important:</b> All compound operations from both {@link Long2LongMap} and {@link Map} | ||
| * interfaces are overridden to ensure atomicity within a segment's write lock. |
Comment on lines
+26
to
+29
| import it.unimi.dsi.fastutil.longs.LongBinaryOperator; | ||
| import it.unimi.dsi.fastutil.longs.LongOpenHashSet; | ||
| import it.unimi.dsi.fastutil.longs.LongSet; | ||
| import it.unimi.dsi.fastutil.objects.ObjectArrayList; |
Comment on lines
+255
to
+257
| V newValue = mappingFunction.apply(key); | ||
| seg.map.put(key, newValue); | ||
| return newValue; |
Comment on lines
+272
to
+274
| V newValue = mappingFunction.get(key); | ||
| seg.map.put(key, newValue); | ||
| return newValue; |
Comment on lines
+50
to
+51
| * <p><b>Important:</b> All compound operations (computeIfAbsent, computeIfPresent, compute, merge) | ||
| * from both {@link Long2ObjectMap} and {@link Map} interfaces are overridden to ensure atomicity |
Comment on lines
+223
to
+227
| void testNullValues() { | ||
| ConcurrentLong2ObjectHashMap<String> map = new ConcurrentLong2ObjectHashMap<>(); | ||
| map.put(1L, null); | ||
| Assertions.assertTrue(map.containsKey(1L)); | ||
| Assertions.assertNull(map.get(1L)); |
gavinchou
approved these changes
Mar 14, 2026
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add two thread-safe primitive-key concurrent hash maps built on fastutil, designed as drop-in replacements for
ConcurrentHashMap<Long, V>andConcurrentHashMap<Long, Long>in memory-sensitive FE paths.ConcurrentLong2ObjectHashMap<V>— replacesConcurrentHashMap<Long, V>ConcurrentLong2LongHashMap— replacesConcurrentHashMap<Long, Long>Why
ConcurrentHashMap<Long, V>costs ~64 bytes per entry due to Long boxing, Node wrapper, and segment overhead. These fastutil-based maps reduce that to ~16 bytes per entry — a 4x memory reduction.In Doris FE, several critical data structures use
ConcurrentHashMap<Long, V>at tablet/partition scale (millions of entries), making this a significant memory optimization opportunity.Design
Mapinterface compatibility for drop-in replacementputIfAbsent,computeIfAbsent,replace,remove(key, value)entrySet()/keySet()/values()Memory comparison
ConcurrentHashMap<Long, V>ConcurrentLong2ObjectHashMap<V>ConcurrentHashMap<Long, Long>ConcurrentLong2LongHashMapTest plan
ConcurrentLong2ObjectHashMapTest— 432 lines covering put/get/remove, putIfAbsent, computeIfAbsent, replace, concurrent writes from multiple threads, iteration consistency, empty map edge casesConcurrentLong2LongHashMapTest— 455 lines covering CRUD, default value semantics, concurrent operations, atomic operations, iteration, edge cases🤖 Generated with Claude Code