Functional tests are not working

[mlewand]

After I:

• have built a local version of extension,
• followed a guide to set up the test environment,
  • also checked the recently updated version by Mateusz: https://github.com/ckeditor/github-writer/blob/5d7cf6f033a2d6259c5c5f6c34d59e23dcb7f4a0/tests/README.md

The tests are not passing despite the OS. I checked it on win10 and mac - both are failing.

The browser is opened, I can see that the user is signed in, but it hangs from there.

Here's the output from mac:

$ yarn run test-functional
yarn run v1.22.0
$ ./node_modules/.bin/mocha ./tests/functional/*.js --timeout 15000
 
 
  Issue
waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
    1) should create a new issue
    2) should create a new comment
    3) should edit the created comment
 
  Pull Request
waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
    4) should create a new pull request
    5) should create a new comment
    6) should edit the created comment
    7) should add a code line comment
    8) should add a review comment
 
  Wiki
waitFor is deprecated and will be removed in a future release. See https://github.com/puppeteer/puppeteer/issues/6214 for details and how to migrate your code.
    9) should create a new wiki
 
 
  0 passing (2m)
  9 failing
 
  1) Issue
       should create a new issue:
     TimeoutError: waiting for selector `meta[name="user-login"][content="user@user.test"]` failed: timeout 30000ms exceeded
      at new WaitTask (node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:509:34)
      at DOMWorld.waitForSelectorInPage (node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:420:26)
      at Object.internalHandler.waitFor (node_modules/puppeteer/lib/cjs/puppeteer/common/QueryHandler.js:31:77)
      at DOMWorld.waitForSelector (node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:313:29)
      at Frame.waitForSelector (node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:841:51)
      at Page.waitForSelector (node_modules/puppeteer/lib/cjs/puppeteer/common/Page.js:1335:33)
      at LoginPage.login (tests/_pom/pages/loginpage.js:35:9)
      at processTicksAndRejections (internal/process/task_queues.js:97:5)
      at async NewIssuePage.goto (tests/_pom/githubpage.js:55:5)
      at async Function.getPage (tests/_pom/githubpage.js:217:3)
      at async Context.<anonymous> (tests/functional/issue.js:21:10)
 
  2) Issue
       should create a new comment:
     AssertionError: expected undefined to be an instance of IssuePage
      at Context.<anonymous> (tests/functional/issue.js:39:27)
      at processImmediate (internal/timers.js:456:21)
 
  3) Issue
       should edit the created comment:
     AssertionError: expected undefined to be an instance of IssuePage
      at Context.<anonymous> (tests/functional/issue.js:57:27)
      at processImmediate (internal/timers.js:456:21)
 
  4) Pull Request
       should create a new pull request:
     TimeoutError: waiting for selector `[name="login"]` failed: timeout 30000ms exceeded
      at new WaitTask (node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:509:34)
      at DOMWorld.waitForSelectorInPage (node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:420:26)
      at Object.internalHandler.waitFor (node_modules/puppeteer/lib/cjs/puppeteer/common/QueryHandler.js:31:77)
      at DOMWorld.waitForSelector (node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:313:29)
      at Frame.waitForSelector (node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:841:51)
      at Frame.waitFor (node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:774:25)
      at Page.waitFor (node_modules/puppeteer/lib/cjs/puppeteer/common/Page.js:1310:33)
      at LoginPage.login (tests/_pom/pages/loginpage.js:30:14)
      at FileEditPage.goto (tests/_pom/githubpage.js:55:21)
      at processTicksAndRejections (internal/process/task_queues.js:97:5)
      at async Function.getPage (tests/_pom/githubpage.js:217:3)
      at async Context.<anonymous> (tests/functional/pullrequest.js:23:10)
 
  5) Pull Request
       should create a new comment:
     AssertionError: expected undefined to be an instance of PullRequestPage
      at Context.<anonymous> (tests/functional/pullrequest.js:50:27)
      at processImmediate (internal/timers.js:456:21)
 
  6) Pull Request
       should edit the created comment:
     AssertionError: expected undefined to be an instance of PullRequestPage
      at Context.<anonymous> (tests/functional/pullrequest.js:69:27)
      at processImmediate (internal/timers.js:456:21)
 
  7) Pull Request
       should add a code line comment:
     AssertionError: expected undefined to be an instance of PullRequestPage
      at Context.<anonymous> (tests/functional/pullrequest.js:88:27)
      at processImmediate (internal/timers.js:456:21)
 
  8) Pull Request
       should add a review comment:
     AssertionError: expected undefined to be an instance of PullRequestPage
      at Context.<anonymous> (tests/functional/pullrequest.js:107:27)
      at processImmediate (internal/timers.js:456:21)
 
  9) Wiki
       should create a new wiki:
     TimeoutError: waiting for selector `[name="login"]` failed: timeout 30000ms exceeded
      at new WaitTask (node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:509:34)
      at DOMWorld.waitForSelectorInPage (node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:420:26)
      at Object.internalHandler.waitFor (node_modules/puppeteer/lib/cjs/puppeteer/common/QueryHandler.js:31:77)
      at DOMWorld.waitForSelector (node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:313:29)
      at Frame.waitForSelector (node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:841:51)
      at Frame.waitFor (node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:774:25)
      at Page.waitFor (node_modules/puppeteer/lib/cjs/puppeteer/common/Page.js:1310:33)
      at LoginPage.login (tests/_pom/pages/loginpage.js:30:14)
      at NewWikiPage.goto (tests/_pom/githubpage.js:55:21)
      at processTicksAndRejections (internal/process/task_queues.js:97:5)
      at async Function.getPage (tests/_pom/githubpage.js:217:3)
      at async Context.<anonymous> (tests/functional/wiki.js:22:10)
 
 
 
error Command failed with exit code 9.

[mlewand]

Guess the information about waitFor function being deprecated is related to this.

[mlewand]

I made a fake account solely for this (it was flagged by GitHub - but works normally otherwise). The canary browser opened by the script signs in correctly, but does nothing after signing in.

[mateuszzagorski]

I've made a fake account to try this out as well. 

After running the yarn run test-functional command, similarly to the latest comment, the browser is opened, the script signs in correctly but it goes to the next steps correctly for me.
I can see it going through to the correct repository (the one set up in config.json file), filling out the title, and comment field but failing to submit.

Screenshot showing the result of the script just before failing:

The account is not flagged by GitHub and the config.json file was prepared as described in the tests readme.

First error message:

1. Issue
         should create a new issue:
       Error: Node is either not visible or not an HTMLElement

It seems like the page = await editor.submit(); does not result in creating a new issue.

[Dumluregn]

I get a bit different results. The first test is passing. The rest doesn't. The log is below:

  1 passing (42s)
  9 failing

  1) Issue
       should create a new comment:

      AssertionError: expected '<p dir="auto">Commenting with GitHub …' to equal '<p>Commenting with <strong>GitHub Wri…'
      + expected - actual

      -<p dir="auto">Commenting with GitHub Writer.</p>
      -<p dir="auto">Time stamp: 2022-05-30T10:28:30.813Z.</p>
      +<p>Commenting with <strong>GitHub Writer</strong>.</p>
      +<p>Time stamp: 2022-05-30T10:28:30.813Z.</p>
      
      at Context.<anonymous> (tests/functional/issue.js:51:47)
      at processTicksAndRejections (internal/process/task_queues.js:95:5)

  2) Issue
       should edit the created comment:
     Error: Evaluation failed: TypeError: Cannot read property 'closest' of null
    at __puppeteer_evaluation_script__:2:15
      at ExecutionContext._evaluateInternal (node_modules/puppeteer/lib/cjs/puppeteer/common/ExecutionContext.js:221:19)
      at processTicksAndRejections (internal/process/task_queues.js:95:5)
      at async JSHandle.evaluateHandle (node_modules/puppeteer/lib/cjs/puppeteer/common/JSHandle.js:120:16)
      at async IssuePage.editComment (tests/_pom/pages/commentstimelinepage.js:32:24)
      at async Context.<anonymous> (tests/functional/issue.js:61:18)

  3) Issue
       "after all" hook: should delete the issue for "should edit the created comment":
     Error: No node found for selector: .discussion-sidebar-item svg.octicon-trashcan
      at Object.assert (node_modules/puppeteer/lib/cjs/puppeteer/common/assert.js:26:15)
      at DOMWorld.click (node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:277:21)
      at processTicksAndRejections (internal/process/task_queues.js:95:5)
      at async IssuePage.deleteIssue (tests/_pom/pages/issuepage.js:28:3)
      at async Context.<anonymous> (tests/functional/issue.js:16:4)

  4) Pull Request
       should create a new pull request:
     Error: No root element found for the selector `form.js-blob-form`.
      at FileEditPage.getEditorByRoot (tests/_pom/githubpage.js:84:10)
      at processTicksAndRejections (internal/process/task_queues.js:95:5)
      at async FileEditPage.getMainEditor (tests/_pom/pages/fileeditpage.js:29:10)
      at async FileEditPage.appendText (tests/_pom/pages/fileeditpage.js:39:18)
      at async Context.<anonymous> (tests/functional/pullrequest.js:27:3)

  5) Pull Request
       should create a new comment:
     AssertionError: expected FileEditPage{ …(3) } to be an instance of PullRequestPage
      at Context.<anonymous> (tests/functional/pullrequest.js:50:27)
      at processImmediate (internal/timers.js:464:21)

  6) Pull Request
       should edit the created comment:
     AssertionError: expected FileEditPage{ …(3) } to be an instance of PullRequestPage
      at Context.<anonymous> (tests/functional/pullrequest.js:69:27)
      at processImmediate (internal/timers.js:464:21)

  7) Pull Request
       should add a code line comment:
     AssertionError: expected FileEditPage{ …(3) } to be an instance of PullRequestPage
      at Context.<anonymous> (tests/functional/pullrequest.js:88:27)
      at processImmediate (internal/timers.js:464:21)

  8) Pull Request
       should add a review comment:
     AssertionError: expected FileEditPage{ …(3) } to be an instance of PullRequestPage
      at Context.<anonymous> (tests/functional/pullrequest.js:107:27)
      at processImmediate (internal/timers.js:464:21)

  9) Wiki
       should create a new wiki:
     TimeoutError: waiting for selector `form[name="gollum-editor"][data-github-writer-id]` failed: timeout 30000ms exceeded
      at new WaitTask (node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:509:34)
      at DOMWorld.waitForSelectorInPage (node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:420:26)
      at Object.internalHandler.waitFor (node_modules/puppeteer/lib/cjs/puppeteer/common/QueryHandler.js:31:77)
      at DOMWorld.waitForSelector (node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:313:29)
      at Frame.waitForSelector (node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:841:51)
      at Frame.waitFor (node_modules/puppeteer/lib/cjs/puppeteer/common/FrameManager.js:774:25)
      at Page.waitFor (node_modules/puppeteer/lib/cjs/puppeteer/common/Page.js:1310:33)
      at NewWikiPage.getMainEditor (tests/_pom/pages/newwikipage.js:40:26)
      at Context.<anonymous> (tests/functional/wiki.js:29:29)
      at processTicksAndRejections (internal/process/task_queues.js:95:5)