Ask HN: How do you manage automated test workflow and view test analytics?

Unfortunately there is no silver bullet to make your tests run faster. I have used both Jenkins and CircleCI and they are great for most environments.

To combat this speed issue I have added a timer to all of my individual test runs. In Ruby with rspec, for example, this is relatively simple by just adding --profile to your .rspec file. Every time your test suite is run, this will show the top 10 slowest running tests. Each testing framework has a similar mechanism or add-on which will give you a report. Next start refactoring those long running tests. At first it will seem like a slow processes, but working on it a bit each day will quickly show significant improvement in your testing suite speed.

Next, watch out for tests, particularly integration tests and Selenium tests, which either test the framework and not your code, or which are redundant. A number of times I have walked into clients offices and have been told that their test suite is as "tight as it can be" and yet still runs slow. I look through it and find many areas where engineers in their desire to test thoroughly were unfortunately testing areas they don't need to. A bit of refactoring again showed massive dividends.

Finally, don't be afraid to segment your tests to run only portions that are effected by the code that was changed. It's true that running the entire suite is important, but you're correct in only running the whole thing at certain times. My current project has segments for API, various Front End resources, and service integrations. There are a number of sub segments as well. Again I think of this as a test refactoring exercise, but it will pay off if designed well.

So, unfortunately I don't have a software solution that will solve your problems. But I can say that a few refactoring activities can make all of the difference.

Selenium tests are a pain, and we come across a lot of customers who have unstable selenium tests. We're building a lot in the next few months to help with it.

Firstly, one feature that we've just put into beta is to show you failing tests. If you're using RSpec or Cucumber or another framework that can generate junit.xml files, we should be able to show you which tests failed, which is shown on the build page, in chat integrations, etc. (Also available via API if you want to consume it via hubot or at the command line or similar).

We're planning to use this to track flaky tests. Someone on our team has the objective to build it this year, so if things go as we optimistically guess that they might, we might see it this year. Otherwise, it should be easy to use the API to build that yourselves.

It seems like that might help you - let me know!

The best I've been able to accomplish is a set of parametrized Jenkins jobs that default to running latest tests on latest code, and report results to a custom external application. The test monitor has a status console and can automatically schedule more Jenkins jobs using build parameters to run the tests with specific versions of application and tests to bisect failures. It's ugly, but is tolerable for what we need.