Add a workflow that builds the docs and deploys them at staged or production by martin-g · Pull Request #104 · apache/datafusion-python

martin-g · 2022-12-05T22:29:59Z

Which issue does this PR close?

Closes #39.

Rationale for this change

Publish the documentation for the Datafusion Python bindings.

When changes are pushed to master then it will copy the generated HTMLs to asf-staging branch and update https://arrow.staged.apache.org/datafusion-python, i.e. the staging site. See Add a workflow that builds the docs and deploys them at staged or production #104 (comment)
When changes are pushed to a non-RC tag, e.g. 0.8.0, then it will copy the generated HTMLs to asf-site branch and update https://arrow.apache.org/datafusion-python, i.e. the production site. See Issue #39 - Add .asf.yaml for asf-site #105

What changes are included in this PR?

A new GitHub Actions workflow is introduced that builds the docs and deploys them at the appropriate site

Are there any user-facing changes?

No API changes!

martin-g · 2022-12-05T22:32:27Z

@andygrove Before merging this PR please create a new branch named asf-staging from asf-site and update its .asf.yaml file contents to be the same as in https://github.com/martin-g/arrow-datafusion-python/blob/asf-staging/.asf.yaml.
It will be used for the staging deployment.

martin-g · 2022-12-05T22:32:51Z

@kou Please review!

…duction Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

.github/workflows/docs.yaml

kou · 2022-12-06T01:52:01Z

.github/workflows/docs.yaml

+      - name: Copy & push the generated HTML
+        run: |
+          cd docs-target
+          find ./ -name "*.html" -type f -print0 | xargs -0 /bin/rm -f


This may not work for other extensions such as *.css. How about using rsync?

rsync \ -a \ --delete \ --exclude '/.git/' \ ../docs/build/html/ \ ./

Right! There are .css/.js/images/... !
I have to test your approach!
The important thing is to preserve anything that is not autogenerated, especially .asf.yaml

How about copying .asf.yaml from master to asf-site?
apache/arrow-site uses this approach: apache/arrow-site#116

This will work for asf-site, but it will break asf-staging, because it uses staging: instead of publish:

find ./ -not -name ".asf.yaml" | grep -v ".git" | xargs rm -rf seems to do the trick.

This will work for asf-site, but it will break asf-staging, because it uses staging: instead of publish:

Can we write both staging: and publish: in master's .asf.yaml?

staging: whoami: asf-staging subdir: datafusion-python publish: whoami: asf-site subdir: datafusion-python

I think this should work!
The only way to find out is to try it! :-)

I think so too. :-)

done with d5e32a2!
depends on #107

.github/workflows/docs.yaml

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

Use special email address for the Git user, so that it has an avatar/icon. Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

kou

+1

This almost looks good to me. I think that we can merge this after we confirm that this works with 0.8.0-rc1 and 0.8.0 tags on your fork. (We need to merge this branch to master on your fork temporary to test these tags.)

.github/workflows/docs.yaml

kou · 2022-12-07T20:22:35Z

.github/workflows/docs.yaml

+on:
+  push:
+    branches:
+      - docs-workflow


We need to change this to master when we merge this.

Right! I used it to be able to test on my fork!

kou · 2022-12-07T20:41:22Z

.github/workflows/docs.yaml

+      - name: Checkout docs target branch
+        uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
+          ref: ${{ steps.target-branch.outputs.value }}
+          path: docs-target


It seems that we need to prepare asf-site and asf-staging branches for this: https://github.com/martin-g/arrow-datafusion-python/actions/runs/3633415170/jobs/6130403408

How about using git worktree instead?

diff --git a/.github/workflows/docs.yaml b/.github/workflows/docs.yaml index 806202a..bfafafa 100644 --- a/.github/workflows/docs.yaml +++ b/.github/workflows/docs.yaml @@ -26,12 +26,6 @@ jobs: fi - name: Checkout docs sources uses: actions/checkout@v3 - - name: Checkout docs target branch - uses: actions/checkout@v3 - with: - fetch-depth: 0 - ref: ${{ steps.target-branch.outputs.value }} - path: docs-target - name: Setup Python uses: actions/setup-python@v4 with: @@ -61,7 +55,12 @@ jobs: - name: Copy & push the generated HTML run: | set -x + git worktree add docs-target cd docs-target + if ! git checkout ${{ steps.target-branch.outputs.value }}; then + git checkout --orphan ${{ steps.target-branch.outputs.value }} + git rm -rf . + fi # delete anything but: 1) '.'; 2) '..'; 3) .git/ find ./ | grep -vE "^./$|^../$|^./.git" | xargs rm -rf cp ../.asf.yaml .

IMO this is more complex/confusing than the current solution.

A task before merging this PR is to create asf-staging branch.

Created: https://github.com/apache/arrow-datafusion-python/tree/asf-staging

.github/workflows/docs.yaml

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

martin-g · 2022-12-07T21:00:22Z

This almost looks good to me. I think that we can merge this after we confirm that this works with 0.8.0-rc1 and 0.8.0 tags on your fork. (We need to merge this branch to master on your fork temporary to test these tags.)

Actually I already tested tags - see testN and test-rcN at https://github.com/martin-g/arrow-datafusion-python/actions
Those tags were created from docs-workflow branch, not from master.

kou · 2022-12-07T21:41:59Z

This almost looks good to me. I think that we can merge this after we confirm that this works with 0.8.0-rc1 and 0.8.0 tags on your fork. (We need to merge this branch to master on your fork temporary to test these tags.)

Actually I already tested tags - see testN and test-rcN at https://github.com/martin-g/arrow-datafusion-python/actions Those tags were created from docs-workflow branch, not from master.

Could you retry testN after you create asf-site branch on your fork?

martin-g · 2022-12-07T21:50:26Z

Could you retry testN after you create asf-site branch on your fork?

Done!
There is no Deploy DataFusion Python site for test-rc10 and there is such for test10.

kou · 2022-12-07T23:35:09Z

Thanks. Committed: martin-g@34ed8b8

I merge this.

martin-g force-pushed the docs-workflow branch from 6056d73 to a0e9bd6 Compare December 5, 2022 22:33

Add a workflow that builds the docs and deploys them at staged or pro…

aea783e

…duction Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

martin-g force-pushed the docs-workflow branch from a0e9bd6 to aea783e Compare December 5, 2022 22:36

martin-g mentioned this pull request Dec 5, 2022

Issue #39 - Add .asf.yaml for asf-site #105

Closed

kou reviewed Dec 6, 2022

View reviewed changes

martin-g and others added 2 commits December 6, 2022 09:19

Add a branch for unsupported branch/tag

91d97e6

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

Improve the glob to ignored RCx tags

289f36e

martin-g force-pushed the docs-workflow branch 2 times, most recently from 52bcc20 to 64dc8d9 Compare December 6, 2022 20:38

"git commit" only if there are changed files

e0a6910

Use special email address for the Git user, so that it has an avatar/icon. Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

martin-g force-pushed the docs-workflow branch from 64dc8d9 to e0a6910 Compare December 6, 2022 20:49

martin-g added 4 commits December 6, 2022 23:00

Test building tags

617ead0

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

Delete everything but .asf.yaml and .git/ when copying the new docs

832cc6d

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

Remove the filtering by paths

8f46ef9

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

Do not try to delete . and ..

822153e

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

martin-g force-pushed the docs-workflow branch from d27c482 to 822153e Compare December 6, 2022 21:47

martin-g mentioned this pull request Dec 6, 2022

Add entries for publishing production (asf-site) and staging docs #107

Merged

Use .asf.yaml from master/tag

d5e32a2

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>

kou approved these changes Dec 7, 2022

View reviewed changes

martin-g and others added 2 commits December 7, 2022 22:55

Remove empty line

2a65b5d

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

Fix wording

8b92e71

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

Change the tested branch back to master

e0692fa

kou merged commit a110049 into apache:master Dec 7, 2022

kou mentioned this pull request Dec 7, 2022

Add DataFusion Python docs apache/arrow-site#279

Closed

martin-g deleted the docs-workflow branch December 8, 2022 09:38

kou mentioned this pull request Dec 13, 2022

chore: set up release process apache/arrow-adbc#174

Merged

Conversation

martin-g commented Dec 5, 2022 • edited by kou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Uh oh!

martin-g commented Dec 5, 2022

Uh oh!

martin-g commented Dec 5, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

martin-g Dec 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

martin-g commented Dec 7, 2022

Uh oh!

kou commented Dec 7, 2022

Uh oh!

martin-g commented Dec 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kou commented Dec 7, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

martin-g commented Dec 5, 2022 •

edited by kou

Loading

martin-g Dec 6, 2022 •

edited

Loading

martin-g commented Dec 7, 2022 •

edited

Loading