Skip to content

Bigtable: Implement query sharding by generalizing ReadRows resume request builder.#3103

Merged
garrettjonesgoogle merged 10 commits intogoogleapis:masterfrom
igorbernstein2:query-sharding
Aug 14, 2018
Merged

Bigtable: Implement query sharding by generalizing ReadRows resume request builder.#3103
garrettjonesgoogle merged 10 commits intogoogleapis:masterfrom
igorbernstein2:query-sharding

Conversation

@igorbernstein2
Copy link
Contributor

@igorbernstein2 igorbernstein2 commented Mar 29, 2018

This extends the work done in #2986 to allow map reduce style frameworks like beam to split queries into shards and execute them in parallel. The mechanism for chopping off part of query in a resume request is very similar to splitting a query into multiple shards. The main difference is how many splits are used.

The common functionality is extracted to an internal RowSetUtil class that does all of the heavy lifting. The class is used both byReadRowsResumptionStrategy for computing the resume request and the newly introduced Query#shard method.

Also expose the ability to get a Query's bounding range. The combination of Query#shard & Query#getBound is needed to implement a Beam source

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla: yes This human has signed the Contributor License Agreement.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants