AI-powered test failure analysis and fix as a GitHub Action

[CopilotMe]

Why are you starting this discussion?

Product Feedback

What GitHub Actions topic or product is this about?

General

Discussion Details

We built a GitHub Action that analyzes test failures with AI. It parses JUnit XML, explains root cause, classifies app bug vs test bug, and scores flakiness. Results posted as PR comments.

Works with any JUnit XML framework (Playwright, Cypress, Jest, pytest, WebdriverIO, etc).

https://github.com/marketplace/actions/fixsense-ai-test-failure-analysis

Would love feedback from the Actions community!

[vladlen-codes]

I have a few question:

1. When test results are analyzed, is the JUnit XML content sent to an external AI service? If so, what data handling/privacy guarantees are in place?
2. Which AI model powers the analysis? Is it configurable or bring-your-own-key?
3. How does it handle edge cases like environment-related failures (e.g., network timeouts, Docker issues) that are neither strictly an app bug nor a test bug?
4. Is this free to use or is there a usage tier?

[CopilotMe]

Thanks for the great questions!

1. Privacy & data handling
   Only the test name, error message, and stack trace from JUnit XML are sent to our API for analysis. We don't send your source code, environment variables, or any other repo content. We never store raw test output beyond the analysis window. All API communication is encrypted over HTTPS, and analysis results are strictly bound to your account - no other user or team can access them. We are fully GDPR-compliant and will never share, sell, or expose your test data to third parties. Your data is never used to train AI models. Data retention depends on your plan (7 days on Free, 60 days on Pro). You can delete your data anytime from the dashboard or upon request. Let me know if everything is clear now.
   Please also check our Privacy Policy for full details.

2. AI model
   The analysis uses a fast AI model optimized for test failure classification. For the auto-fix feature (where an AI agent creates PRs with actual code fixes), it's BYOK (bring your own Anthropic or OpenAI API key) - so you control the cost and have full transparency.

3. Environment-related failures
   Good edge case - we classify these as infrastructure/environment issues. The root cause explanation will say something like "CI runner network timeout - not related to application or test code." The flakiness score also helps here: environment-related failures typically score high (70-100) since they're intermittent by nature.

4. Pricing
   Free plan: 350 analyses/month + 3 auto-fixes. Pro ($49/mo) is now providing 1-month free trial too for early adopters: 3,500 analyses + 20 auto-fixes. More than enough for our targeted small teams to start.