merge bq-perf branch by pongad · Pull Request #2755 · googleapis/google-cloud-java

pongad · 2018-01-08T05:11:23Z

No description provided.

Eventually we'll delete QueryResponse completely so that we can use the faster listTableData RPC.

In the future, we might populate it with result from tabledata.list. The response has fewer fields than that of getQueryResults.

This is the first step in removing QueryResponse. This PR first creates QueryResult then retrieves the Response. Eventually, we'll directly create Result from tabledata.list RPC.

This sped up query benchmark from 470s to 360s. This breaks "schema injection" however, so users cannot call `row.get(String)`. This will be fixed in a follow up. Updates #2591.

@internalapi

Updates #2591. Previous PR, #2642, broke Job.getQueryResults in two ways: 1. It wasn't possible to get total number of rows. 2. The returned iterator doesn't record schema, so it wasn't possible to call `row.get(String)`. This PR prepares the fix. Actually landing the fix will change tests, so it is punted to another PR to keep changes small. From previous meeting, it was our decision to remove - BigQuery.getQueryResults method - QueryResponse class However, the method is being used internally by the client, and interfaces only have public members. Removing them altogether entails a big surgery, so this PR instead mark them @internalapi.

Updates #2591.

This also makes queries require fewer RPCs if detailed job statuses are not required. If a query fails, then query() throws an exception detailing why it fails and provides the job ID so users can retrieve more details themselves. Updates #2591.

listTableData and query now use the same classes.

google-cloud-bigquery/README.md

-    QueryOption.of(QueryResultsOption.maxWaitTime(60000L)),
-    QueryOption.of(QueryResultsOption.pageSize(1000L)));
+QueryJobConfiguration queryConfig =
+    QueryJobConfiguration.newBuilder("SELECT * FROM my_dataset_id.my_table_id").build();


google-cloud-bigquery/README.md

 // Read rows
 System.out.println("Table rows:");
-for (FieldValues row : queryResponse.getResult().iterateAll()) {
+for (FieldValueList row : bigquery.query(queryConfig).iterateAll()) {


google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQuery.java

   * @throws BigQueryException upon failure
   */
-  Page<FieldValueList> listTableData(TableId tableId, TableDataListOption... options);
+  TableResult listTableData(TableId tableId, TableDataListOption... options);


google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQuery.java

+   * String tableName = "my_table_name";
+   * Schema schema = ...;
+   * String field = "my_field";
+   * Page<FieldValueList> tableData =


google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/BigQuery.java

+   *         Field.of("corpus", LegacySQLTypeName.STRING),
+   *         Field.of("corpus_date", LegacySQLTypeName.STRING));
+   * Page<FieldValueList> page =
+   *     bigquery.listTableData(


google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/TableResult.java

+   * Returns the total number of rows in the complete result set, which can be more than the number
+   * of rows in the first page of results returned by {@link #getValues()}.
+   *
+   * <p>If this instance is a result of a dry-run query, returns {@code 0}.


google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/BigQueryImplTest.java

-    for (FieldValueList row : response.getResult().getValues()) {
-      assertEquals(false, row.get(0).getBooleanValue());
-      assertEquals(1L, row.get(1).getLongValue());
+    // TODO(pongad): pagesize = 42


google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/BigQueryImplTest.java

-    for (FieldValueList row : response.getResult().getValues()) {
-      assertEquals(false, row.get(0).getBooleanValue());
-      assertEquals(1L, row.get(1).getLongValue());
+    // TODO(pongad): pagesize = 42


google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/JobTest.java

+    expect(mockOptions.getClock()).andReturn(CurrentMillisClock.getDefaultClock()).times(2);
    Job completedJob = expectedJob.toBuilder().setStatus(status).build();
-    QueryResponse completedQuery = QueryResponse.newBuilder().setJobCompleted(true).build();
+    // TODO(pongad): remove when https://github.com/googleapis/gax-java/pull/431/ lands.


google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/TableResultTest.java

@@ -0,0 +1,103 @@
+/*
+ * Copyright 2015 Google Inc. All Rights Reserved.


pongad · 2018-01-09T03:31:16Z

@tswast PTAL

…18.0 (#2755) * chore(deps): update dependency com.google.cloud:libraries-bom to v26.18.0 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | googleapis/java-cloud-bom | action | digest | `f3c611a` -> `db4265f` | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/googleapis/sdk-platform-java).

🤖 I have created a release *beep* *boop* --- <details><summary>2.40.1</summary> ## [2.40.1](googleapis/sdk-platform-java@v2.40.0...v2.40.1) (2024-05-15) ### Bug Fixes * [common-protos] An existing method `UpdateVehicleLocation` is ([a8848b4](googleapis/sdk-platform-java@a8848b4)) * [common-protos] An existing method `UpdateVehicleLocation` is removed from service `VehicleService` ([#2751](googleapis/sdk-platform-java#2751)) ([a8848b4](googleapis/sdk-platform-java@a8848b4)) * [iam] An existing method `UpdateVehicleLocation` is removed from ([04ab176](googleapis/sdk-platform-java@04ab176)) * [iam] An existing method `UpdateVehicleLocation` is removed from service `VehicleService` ([#2752](googleapis/sdk-platform-java#2752)) ([04ab176](googleapis/sdk-platform-java@04ab176)) * do not populate repo level change while removing library ([#2740](googleapis/sdk-platform-java#2740)) ([8d92d16](googleapis/sdk-platform-java@8d92d16)) * only append `.api.grpc` suffix to group id if the artifact id starts with `proto-` or `grpc-` ([#2731](googleapis/sdk-platform-java#2731)) ([b791c84](googleapis/sdk-platform-java@b791c84)) * opentelemetry-bom to be in third-party-dependencies BOM ([#2736](googleapis/sdk-platform-java#2736)) ([0700180](googleapis/sdk-platform-java@0700180)) * prepare to generate grafeas ([#2761](googleapis/sdk-platform-java#2761)) ([bf26ca1](googleapis/sdk-platform-java@bf26ca1)) * Replace deprecated protobuf methods. ([#2764](googleapis/sdk-platform-java#2764)) ([c1e586d](googleapis/sdk-platform-java@c1e586d)) ### Dependencies * update dependency black to v24.4.2 ([#2660](googleapis/sdk-platform-java#2660)) ([027191f](googleapis/sdk-platform-java@027191f)) * update dependency com.fasterxml.jackson:jackson-bom to v2.17.1 ([#2732](googleapis/sdk-platform-java#2732)) ([3e94c74](googleapis/sdk-platform-java@3e94c74)) * update dependency com.google.cloud:grpc-gcp to v1.6.0 ([#2767](googleapis/sdk-platform-java#2767)) ([e360a3d](googleapis/sdk-platform-java@e360a3d)) * update dependency com.google.errorprone:error_prone_annotations to v2.27.1 ([#2708](googleapis/sdk-platform-java#2708)) ([5ceb5fa](googleapis/sdk-platform-java@5ceb5fa)) * update dependency com.google.errorprone:error_prone_annotations to v2.27.1 ([#2709](googleapis/sdk-platform-java#2709)) ([94df49f](googleapis/sdk-platform-java@94df49f)) * update dependency com.google.oauth-client:google-oauth-client-bom to v1.36.0 ([#2768](googleapis/sdk-platform-java#2768)) ([f8500df](googleapis/sdk-platform-java@f8500df)) * update dependency commons-codec:commons-codec to v1.17.0 ([#2710](googleapis/sdk-platform-java#2710)) ([d1c14e9](googleapis/sdk-platform-java@d1c14e9)) * update dependency jinja2 to v3.1.4 [security] ([#2742](googleapis/sdk-platform-java#2742)) ([e0ebded](googleapis/sdk-platform-java@e0ebded)) * update dependency lxml to v5.2.2 ([#2766](googleapis/sdk-platform-java#2766)) ([99e23f7](googleapis/sdk-platform-java@99e23f7)) * update dependency markupsafe to v2.1.5 ([#2657](googleapis/sdk-platform-java#2657)) ([834b258](googleapis/sdk-platform-java@834b258)) * update dependency net.bytebuddy:byte-buddy to v1.14.15 ([#2753](googleapis/sdk-platform-java#2753)) ([7494d67](googleapis/sdk-platform-java@7494d67)) * update dependency platformdirs to v4.2.1 ([#2662](googleapis/sdk-platform-java#2662)) ([9118e55](googleapis/sdk-platform-java@9118e55)) * update googleapis/java-cloud-bom digest to db4265f ([#2755](googleapis/sdk-platform-java#2755)) ([afbbb78](googleapis/sdk-platform-java@afbbb78)) * update googleapis/java-cloud-bom digest to f3c611a ([#2700](googleapis/sdk-platform-java#2700)) ([de5a108](googleapis/sdk-platform-java@de5a108)) * update opentelemetry-java monorepo to v1.38.0 ([#2769](googleapis/sdk-platform-java#2769)) ([74a2451](googleapis/sdk-platform-java@74a2451)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Co-authored-by: Joe Wang <106995533+JoeWang1127@users.noreply.github.com>

pongad added 20 commits November 8, 2017 16:05

bigquery: make query return a Job (#2566)

6125c56

Eventually we'll delete QueryResponse completely so that we can use the faster listTableData RPC.

Merge branch 'master' into bq-perf

c98a09c

bigquery: give QueryResult fewer fields (#2592)

8259a78

In the future, we might populate it with result from tabledata.list. The response has fewer fields than that of getQueryResults.

bigquery: Job.getQueryResults returns QueryResult (#2593)

4bf9d80

This is the first step in removing QueryResponse. This PR first creates QueryResult then retrieves the Response. Eventually, we'll directly create Result from tabledata.list RPC.

Merge branch 'master' into bq-perf

0aac950

bigquery: read query results only if query finished (#2637)

7b15777

bigquery: use tabledata.list for query (#2642)

5304fbb

This sped up query benchmark from 470s to 360s. This breaks "schema injection" however, so users cannot call `row.get(String)`. This will be fixed in a follow up. Updates #2591.

Merge branch 'master' into bq-perf

5fa8fa6

bigquery: make schema work again (#2660)

66dee5d

Updates #2591.

Merge branch 'master' into bq-perf

f6bcbc2

bigquery: allow schema-ful data listing (#2697)

8ac24ed

listTableData and query now use the same classes.

Merge branch 'master' into bq-perf

8959c2c

bigquery: add test and fix silly bugs (#2712)

ad69088

bigquery: document new methods (#2709)

47e805a

bigquery: properly spell "BigQuery" (#2716)

3313bcb

bigquery: properly handle options to Job.getQueryResults (#2713)

c34e949

add bigquery release notes (#2717)

7d843ad

bigquery: also allow schema-ful list from Table (#2718)

f370cbe

pongad requested review from garrettjonesgoogle and tswast January 8, 2018 05:11

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Jan 8, 2018

tswast requested changes Jan 8, 2018

View reviewed changes

pr comment

47a55dc

tswast approved these changes Jan 9, 2018

View reviewed changes

pongad merged commit 7d144e2 into master Jan 9, 2018

pongad deleted the bq-perf branch January 9, 2018 22:39

		@@ -0,0 +1,103 @@
		/*
		* Copyright 2015 Google Inc. All Rights Reserved.

Conversation

pongad commented Jan 8, 2018

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

pongad commented Jan 9, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants